Review ask

As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.

We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.

Summary

Lifecycle: draftEditUnder review
What this benchmark means
README.md

Public entries: Python and R. Passing coverage requires an installed inventory.

Goal: emit long-format adverse event incidence results, retaining the analysis definition with every value in an Analysis Results Standard (ARS) style dataset.

Input: subject analysis records with planned treatment, treatment code and safety flag, and adverse event analysis records with emergence flag.

Variables:

  • The subject-level file holds one record per safety subject: TRT01P and TRT01PN are the planned treatment name and code, and EVENTN is 1 when the subject has a treatment-emergent adverse event and 0 otherwise.
  • ANALYSISID, ANALYSISSETID, DATASET, VARIABLE and METHODID are AEINC, SAFETY, ADAE, USUBJID and AEINCIDENCE, respectively.
  • ANALYSISSET is SAFFL = 'Y', limiting the numerator and denominator to safety subjects. DATASUBSET is TRTEMFL = 'Y', limiting qualifying events to treatment-emergent events.
  • GROUPINGID is TRT01P. GROUPORDER and GROUPLEVEL are the planned treatment code and name from subject analysis records.
  • OPERATIONID is COUNT or PERCENT. RESULT is the number of safety subjects with at least one treatment-emergent adverse event, or that number as a percentage of all safety subjects in the treatment group. Neither result is rounded.

Note: repeated qualifying events count once per subject. A safety treatment group without qualifying events retains both result rows at 0; a group without safety subjects has no result rows. This flat results dataset illustrates ARS analysis-to-result linkage, not an ARS document.

Inputs

Under review
3 source files

input/adae.csv

Edit
9 rows
input/adae.csv
STUDYIDUSUBJIDASEQTRTEMFL
YAMAA-011Y
YAMAA-012Y
YAMAA-011N
YAMAA-011Y
YAMAA-011Y
YAMAA-011N
YAMAA-011Y
YAMAA-011Y
YAMAA-011Y

input/adsl.csv

Edit
9 rows
input/adsl.csv
STUDYIDUSUBJIDTRT01PTRT01PNSAFFL
YAMAA-01Drug1Y
YAMAA-01Drug1Y
YAMAA-01Drug1Y
YAMAA-01Drug1N
YAMAA-01Placebo2Y
YAMAA-01Placebo2Y
YAMAA-01Control3Y
YAMAA-01Control3N
YAMAA-01Excluded4N

input/subjects.schema.yaml

Edit
yamaa: "1.0"
type: spec
model: ADaM
domain: SUBJECT
keys:
- STUDYID
- USUBJID
input:
  ADSL: adsl.csv
  ADAE:
    path: adae.csv
    types:
      ASEQ: int
base: ADSL
filter: ADSL.SAFFL = 'Y'

output:
  path: subjects.csv
  columns:
  - STUDYID
  - USUBJID
  - TRT01P
  - TRT01PN
  - EVENTN

intermediates:
- id: FIRST
  dataset: ADAE
  key:
  - STUDYID
  - USUBJID
  filter: ADAE.TRTEMFL = 'Y'
  order_by:
  - ADAE.ASEQ
  keep: first

columns:
- name: STUDYID
  type: str
  label: Study Identifier
  derivation: ADSL.STUDYID

- name: USUBJID
  type: str
  label: Unique Subject Identifier
  derivation: ADSL.USUBJID

- name: TRT01P
  type: str
  label: Planned Treatment for Period 01
  derivation: ADSL.TRT01P

- name: TRT01PN
  type: int
  label: Planned Treatment for Period 01 Numeric
  derivation: ADSL.TRT01PN

- name: EVENTN
  type: int
  label: Subject Has a Treatment Emergent Event
  derivation:
    case:
    - when: FIRST.ASEQ IS NOT NULL
      then: 1
    - otherwise: 0

Expected output

Under review
12 expected rows

expected/results.csv

Edit
6 rows
expected/results.csv
STUDYIDANALYSISIDANALYSISSETIDDATASETVARIABLEANALYSISSETDATASUBSETGROUPINGIDGROUPORDERGROUPLEVELMETHODIDOPERATIONIDRESULT
YAMAA-01AEINCSAFETYADAEUSUBJIDSAFFL = 'Y'TRTEMFL = 'Y'TRT01P1DrugAEINCIDENCECOUNT2
YAMAA-01AEINCSAFETYADAEUSUBJIDSAFFL = 'Y'TRTEMFL = 'Y'TRT01P1DrugAEINCIDENCEPERCENT66.66666666666667
YAMAA-01AEINCSAFETYADAEUSUBJIDSAFFL = 'Y'TRTEMFL = 'Y'TRT01P2PlaceboAEINCIDENCECOUNT0
YAMAA-01AEINCSAFETYADAEUSUBJIDSAFFL = 'Y'TRTEMFL = 'Y'TRT01P2PlaceboAEINCIDENCEPERCENT0
YAMAA-01AEINCSAFETYADAEUSUBJIDSAFFL = 'Y'TRTEMFL = 'Y'TRT01P3ControlAEINCIDENCECOUNT1
YAMAA-01AEINCSAFETYADAEUSUBJIDSAFFL = 'Y'TRTEMFL = 'Y'TRT01P3ControlAEINCIDENCEPERCENT100

expected/subjects.csv

Edit
6 rows
expected/subjects.csv
STUDYIDUSUBJIDTRT01PTRT01PNEVENTN
YAMAA-01Drug11
YAMAA-01Drug10
YAMAA-01Drug11
YAMAA-01Placebo20
YAMAA-01Placebo20
YAMAA-01Control31
Shaded columns are derived— = missing value

Human review and assessment

Your review

3 questions and your comments · View benchmark discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Variable NameVariable LabelTypeLengthControlled Terms or FormatOriginCoreConversion DefinitionVariable TypeVariable OrderComments for Define
STUDYIDStudy IdentifierCharCollectedCopy value from SUBJECT.STUDYID.SDTM1
ANALYSISIDAnalysis IdentifierCharAssignedSet constant value "AEINC".SDTM2
ANALYSISSETIDAnalysis Set IdentifierCharAssignedSet constant value "SAFETY".SDTM3
DATASETAnalysis DatasetCharAssignedSet constant value "ADAE".SDTM4
VARIABLEAnalysis VariableCharAssignedSet constant value "USUBJID".SDTM5
ANALYSISSETAnalysis Set ConditionCharAssignedSet constant value "SAFFL = 'Y'".SDTM6
DATASUBSETData Subset ConditionCharAssignedSet constant value "TRTEMFL = 'Y'".SDTM7
GROUPINGIDGrouping Factor IdentifierCharAssignedSet constant value "TRT01P".SDTM8
METHODIDAnalysis Method IdentifierCharAssignedSet constant value "AEINCIDENCE".SDTM9
GROUPORDERGrouping Level OrderNumCollectedCopy value from SUBJECT.TRT01PN.SDTM10
GROUPLEVELGrouping LevelCharCollectedCopy value from SUBJECT.TRT01P.SDTM11
OPERATIONIDAnalysis Operation IdentifierCharDerivedSet per constructed row; see the Row construction sheet.SDTM12
RESULTAnalysis Operation ResultNumDerivedIf OPERATIONID = 'COUNT' then "NSUBJ"; otherwise compute 100 * NSUBJ / DENOM.SDTM13
TemplateInclude whenOPERATIONIDNSUBJDENOM
countConstant "COUNT"Aggregate SUM(SUBJECT.EVENTN) over records matching on the shared output keysAggregate COUNT(SUBJECT.USUBJID) over records matching on the shared output keys
percentConstant "PERCENT"Aggregate SUM(SUBJECT.EVENTN) over records matching on the shared output keysAggregate COUNT(SUBJECT.USUBJID) over records matching on the shared output keys
VersionDateAuthorDescriptionReviewerSign-off
1.72026-10-20yamaa docs buildRead a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default
1.62026-10-17yamaa docs buildDescribe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight
1.52026-10-17yamaa docs buildName collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part
1.42026-10-15yamaa docs buildState datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/adam-results-ae-incidence/full.md

Edit
28 lines

Following CDISC ADaM standards, use the provided ADAE and ADSL datasets to create a RESULTS dataset with one record per planned treatment group and operation.

The output dataset should contain the following columns in this order: STUDYID, ANALYSISID, ANALYSISSETID, DATASET, VARIABLE, ANALYSISSET, DATASUBSET, GROUPINGID, GROUPORDER, GROUPLEVEL, METHODID, OPERATIONID, RESULT

ANALYSISID is "AEINC", ANALYSISSETID is "SAFETY", DATASET is "ADAE", VARIABLE is "USUBJID" and METHODID is "AEINCIDENCE". ANALYSISSET is "SAFFL = 'Y'" and DATASUBSET is "TRTEMFL = 'Y'". GROUPINGID is "TRT01P". GROUPORDER is the planned treatment code and GROUPLEVEL the planned treatment name of the record's planned treatment group, taken from the subject records in ADSL. The treatment groups and both results cover only safety subjects, the subjects with SAFFL "Y" in ADSL. A qualifying event is an adverse event with TRTEMFL "Y", and a subject with at least one qualifying event counts once, however many qualifying events the subject has. Each planned treatment group with at least one safety subject has one record with OPERATIONID "COUNT" and one with OPERATIONID "PERCENT", and a planned treatment group with no safety subjects has no records. For "COUNT", RESULT is the number of safety subjects in the group with at least one qualifying event. For "PERCENT", RESULT is that number as a percentage of all safety subjects in the group. Neither result is rounded. A planned treatment group whose safety subjects have no qualifying events keeps both records, with RESULT 0 in each.

Read the source datasets from /app/input and save the completed dataset as /app/output/results.csv.

prompts/adam-results-ae-incidence/conventions.md

Edit
15 lines

Following CDISC ADaM standards, use the provided ADAE and ADSL datasets to create a RESULTS dataset with one record per planned treatment group and operation.

The output dataset should contain the following columns in this order: STUDYID, ANALYSISID, ANALYSISSETID, DATASET, VARIABLE, ANALYSISSET, DATASUBSET, GROUPINGID, GROUPORDER, GROUPLEVEL, METHODID, OPERATIONID, RESULT

ANALYSISID is "AEINC", ANALYSISSETID is "SAFETY", DATASET is "ADAE", VARIABLE is "USUBJID" and METHODID is "AEINCIDENCE". ANALYSISSET is "SAFFL = 'Y'" and DATASUBSET is "TRTEMFL = 'Y'". GROUPINGID is "TRT01P". OPERATIONID is "COUNT" or "PERCENT". Neither result is rounded.

Read the source datasets from /app/input and save the completed dataset as /app/output/results.csv.

prompts/adam-results-ae-incidence/brief.md

Edit
10 lines

Following CDISC ADaM standards, use the provided ADAE and ADSL datasets to create a RESULTS dataset with one record per planned treatment group and operation.

The output dataset should contain the following columns in this order: STUDYID, ANALYSISID, ANALYSISSETID, DATASET, VARIABLE, ANALYSISSET, DATASUBSET, GROUPINGID, GROUPORDER, GROUPLEVEL, METHODID, OPERATIONID, RESULT

Read the source datasets from /app/input and save the completed dataset as /app/output/results.csv.

yamaa code

Public calls generated from benchmark.yaml; results must match the expected output

R

Edit
4 lines
library(yamaa)yamaa_spec("input/subjects.schema.yaml")$run()$save()yamaa_spec("spec.yaml")$run()$output

Python

Edit
4 lines
import yamaayamaa.spec("input/subjects.schema.yaml").run().save()yamaa.spec("spec.yaml").run().output

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/adam-results-ae-incidence/result.R

Edit
22 lines
subjects <- read.csv("/app/input/adsl.csv", stringsAsFactors = FALSE)events <- read.csv("/app/input/adae.csv", stringsAsFactors = FALSE)subjects <- subjects[subjects$SAFFL == "Y", ]events <- events[events$TRTEMFL == "Y", ]subject_key <- paste(subjects$STUDYID, subjects$USUBJID, sep = "\r")event_key <- paste(events$STUDYID, events$USUBJID, sep = "\r")subjects$EVENTN <- as.integer(subject_key %in% event_key)groups <- split(subjects, interaction(subjects$STUDYID, subjects$TRT01PN,                                      subjects$TRT01P, drop = TRUE))result <- do.call(rbind, lapply(groups, function(group) {  count <- sum(group$EVENTN)  data.frame(STUDYID = group$STUDYID[1], ANALYSISID = "AEINC",    ANALYSISSETID = "SAFETY", DATASET = "ADAE", VARIABLE = "USUBJID",    ANALYSISSET = "SAFFL = 'Y'", DATASUBSET = "TRTEMFL = 'Y'",    GROUPINGID = "TRT01P", GROUPORDER = group$TRT01PN[1],    GROUPLEVEL = group$TRT01P[1], METHODID = "AEINCIDENCE",    OPERATIONID = c("COUNT", "PERCENT"),    RESULT = c(count, 100 * count / nrow(group)))}))result <- result[order(result$STUDYID, result$GROUPORDER,                       result$OPERATIONID), ]write.csv(result, "/app/output/results.csv", row.names = FALSE, na = "")

solutions/adam-results-ae-incidence/result.py

Edit
89 lines
# Reference solution for the yamaa benchmark adam-results-ae-incidence (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1."""Create the RESULTS dataset (AE incidence) from ADAE and ADSL.One record per planned treatment group and operation (COUNT, PERCENT),covering only safety subjects (SAFFL = 'Y' in ADSL).  A qualifying eventis an adverse event with TRTEMFL = 'Y'; a subject with at least onequalifying event counts once."""import csvfrom collections import OrderedDictINPUT_DIR = "/app/input"OUTPUT_DIR = "/app/output"def read_csv(path):    with open(path, newline="", encoding="utf-8") as fh:        return list(csv.DictReader(fh))def main():    adsl = read_csv(f"{INPUT_DIR}/adsl.csv")    adae = read_csv(f"{INPUT_DIR}/adae.csv")    studyid = adsl[0]["STUDYID"] if adsl else ""    # Planned treatment groups, ordered by the planned treatment code    # (TRT01PN); GROUPLEVEL is the planned treatment name (TRT01P).    groups = OrderedDict()  # TRT01PN -> {"name": ..., "subjects": set}    safety_subjects = set()    for row in adsl:        if row["SAFFL"] != "Y":            continue        safety_subjects.add(row["USUBJID"])        code = row["TRT01PN"]        group = groups.setdefault(code, {"name": row["TRT01P"], "subjects": set()})        group["subjects"].add(row["USUBJID"])    # Safety subjects with at least one qualifying (TRTEMFL = 'Y') event.    subjects_with_event = set()    for row in adae:        if row["TRTEMFL"] == "Y" and row["USUBJID"] in safety_subjects:            subjects_with_event.add(row["USUBJID"])    columns = [        "STUDYID", "ANALYSISID", "ANALYSISSETID", "DATASET", "VARIABLE",        "ANALYSISSET", "DATASUBSET", "GROUPINGID", "GROUPORDER",        "GROUPLEVEL", "METHODID", "OPERATIONID", "RESULT",    ]    out_rows = []    for code in sorted(groups, key=lambda c: (int(c) if c.lstrip("-").isdigit() else c, c)):        group = groups[code]        total = len(group["subjects"])        count = len(group["subjects"] & subjects_with_event)        percent = count / total * 100 if total else 0.0        base = {            "STUDYID": studyid,            "ANALYSISID": "AEINC",            "ANALYSISSETID": "SAFETY",            "DATASET": "ADAE",            "VARIABLE": "USUBJID",            "ANALYSISSET": "SAFFL = 'Y'",            "DATASUBSET": "TRTEMFL = 'Y'",            "GROUPINGID": "TRT01P",            "GROUPORDER": code,            "GROUPLEVEL": group["name"],            "METHODID": "AEINCIDENCE",        }        count_row = dict(base, OPERATIONID="COUNT", RESULT=str(count))        percent_row = dict(base, OPERATIONID="PERCENT", RESULT=repr(percent))        out_rows.extend([count_row, percent_row])    with open(f"{OUTPUT_DIR}/results.csv", "w", newline="", encoding="utf-8") as fh:        writer = csv.DictWriter(fh, fieldnames=columns)        writer.writeheader()        writer.writerows(out_rows)    print(f"Wrote {len(out_rows)} records to {OUTPUT_DIR}/results.csv")if __name__ == "__main__":    main()

solutions/adam-results-ae-incidence/result.sas

Edit
43 lines
/* opensas reference: long-format operations from the safety population. */data subjects;  length STUDYID USUBJID $80 TRT01P $20 SAFFL $1;  infile "/app/input/adsl.csv" dsd dlm="," firstobs=2 truncover;  input STUDYID : $80. USUBJID : $80. TRT01P : $20. TRT01PN SAFFL : $1.;  if SAFFL='Y';run;data events;  length STUDYID USUBJID $80 TRTEMFL $1;  infile "/app/input/adae.csv" dsd dlm="," firstobs=2 truncover;  input STUDYID : $80. USUBJID : $80. ASEQ TRTEMFL : $1.;  if TRTEMFL='Y';run;proc sql;  create table qualifying as select distinct STUDYID, USUBJID from events;  create table counted as select s.STUDYID, s.USUBJID, s.TRT01P, s.TRT01PN,    case when e.USUBJID is not null then 1 else 0 end as EVENTN  from subjects s left join qualifying e    on s.STUDYID=e.STUDYID and s.USUBJID=e.USUBJID;  create table totals as select STUDYID, TRT01P, TRT01PN,    sum(EVENTN) as NSUBJ, count(USUBJID) as DENOM  from counted group by STUDYID, TRT01P, TRT01PN;quit;data results;  length ANALYSISID $8 ANALYSISSETID $8 DATASET $8 VARIABLE $8    ANALYSISSET DATASUBSET $24 GROUPINGID $8 GROUPLEVEL $20    METHODID $16 OPERATIONID $8;  set totals;  ANALYSISID='AEINC'; ANALYSISSETID='SAFETY'; DATASET='ADAE';  VARIABLE='USUBJID'; ANALYSISSET="SAFFL = 'Y'";  DATASUBSET="TRTEMFL = 'Y'"; GROUPINGID='TRT01P';  GROUPORDER=TRT01PN; GROUPLEVEL=TRT01P; METHODID='AEINCIDENCE';  OPERATIONID='COUNT'; RESULT=NSUBJ; output;  OPERATIONID='PERCENT'; RESULT=100*NSUBJ/DENOM; output;run;proc sql;  create table final as select STUDYID, ANALYSISID, ANALYSISSETID,    DATASET, VARIABLE, ANALYSISSET, DATASUBSET, GROUPINGID, GROUPORDER,    GROUPLEVEL, METHODID, OPERATIONID, RESULT from results  order by STUDYID, GROUPORDER, OPERATIONID;quit;proc export data=final outfile="/app/output/results.csv" dbms=csv replace;run;