As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.
We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.
Summary
README.mdPublic entries: Python and R. Passing coverage requires an installed inventory.
Goal: emit long-format adverse event incidence results, retaining the analysis definition with every value in an Analysis Results Standard (ARS) style dataset.
Input: subject analysis records with planned treatment, treatment code and safety flag, and adverse event analysis records with emergence flag.
Variables:
- The subject-level file holds one record per safety subject:
TRT01PandTRT01PNare the planned treatment name and code, andEVENTNis 1 when the subject has a treatment-emergent adverse event and 0 otherwise. ANALYSISID,ANALYSISSETID,DATASET,VARIABLEandMETHODIDare AEINC, SAFETY, ADAE, USUBJID and AEINCIDENCE, respectively.ANALYSISSETis SAFFL = 'Y', limiting the numerator and denominator to safety subjects.DATASUBSETis TRTEMFL = 'Y', limiting qualifying events to treatment-emergent events.GROUPINGIDis TRT01P.GROUPORDERandGROUPLEVELare the planned treatment code and name from subject analysis records.OPERATIONIDis COUNT or PERCENT.RESULTis the number of safety subjects with at least one treatment-emergent adverse event, or that number as a percentage of all safety subjects in the treatment group. Neither result is rounded.
Note: repeated qualifying events count once per subject. A safety treatment group without qualifying events retains both result rows at 0; a group without safety subjects has no result rows. This flat results dataset illustrates ARS analysis-to-result linkage, not an ARS document.
Inputs
Under review3 source filesinput/adae.csv
Edit9 rows| STUDYID | USUBJID | ASEQ | TRTEMFL |
|---|---|---|---|
| YAMAA-01 | 1 | Y | |
| YAMAA-01 | 2 | Y | |
| YAMAA-01 | 1 | N | |
| YAMAA-01 | 1 | Y | |
| YAMAA-01 | 1 | Y | |
| YAMAA-01 | 1 | N | |
| YAMAA-01 | 1 | Y | |
| YAMAA-01 | 1 | Y | |
| YAMAA-01 | 1 | Y |
input/adsl.csv
Edit9 rows| STUDYID | USUBJID | TRT01P | TRT01PN | SAFFL |
|---|---|---|---|---|
| YAMAA-01 | Drug | 1 | Y | |
| YAMAA-01 | Drug | 1 | Y | |
| YAMAA-01 | Drug | 1 | Y | |
| YAMAA-01 | Drug | 1 | N | |
| YAMAA-01 | Placebo | 2 | Y | |
| YAMAA-01 | Placebo | 2 | Y | |
| YAMAA-01 | Control | 3 | Y | |
| YAMAA-01 | Control | 3 | N | |
| YAMAA-01 | Excluded | 4 | N |
input/subjects.schema.yaml
Edityamaa: "1.0"
type: spec
model: ADaM
domain: SUBJECT
keys:
- STUDYID
- USUBJID
input:
ADSL: adsl.csv
ADAE:
path: adae.csv
types:
ASEQ: int
base: ADSL
filter: ADSL.SAFFL = 'Y'
output:
path: subjects.csv
columns:
- STUDYID
- USUBJID
- TRT01P
- TRT01PN
- EVENTN
intermediates:
- id: FIRST
dataset: ADAE
key:
- STUDYID
- USUBJID
filter: ADAE.TRTEMFL = 'Y'
order_by:
- ADAE.ASEQ
keep: first
columns:
- name: STUDYID
type: str
label: Study Identifier
derivation: ADSL.STUDYID
- name: USUBJID
type: str
label: Unique Subject Identifier
derivation: ADSL.USUBJID
- name: TRT01P
type: str
label: Planned Treatment for Period 01
derivation: ADSL.TRT01P
- name: TRT01PN
type: int
label: Planned Treatment for Period 01 Numeric
derivation: ADSL.TRT01PN
- name: EVENTN
type: int
label: Subject Has a Treatment Emergent Event
derivation:
case:
- when: FIRST.ASEQ IS NOT NULL
then: 1
- otherwise: 0
Expected output
Under review12 expected rowsexpected/results.csv
Edit6 rows| STUDYID | ANALYSISID | ANALYSISSETID | DATASET | VARIABLE | ANALYSISSET | DATASUBSET | GROUPINGID | GROUPORDER | GROUPLEVEL | METHODID | OPERATIONID | RESULT |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| YAMAA-01 | AEINC | SAFETY | ADAE | USUBJID | SAFFL = 'Y' | TRTEMFL = 'Y' | TRT01P | 1 | Drug | AEINCIDENCE | COUNT | 2 |
| YAMAA-01 | AEINC | SAFETY | ADAE | USUBJID | SAFFL = 'Y' | TRTEMFL = 'Y' | TRT01P | 1 | Drug | AEINCIDENCE | PERCENT | 66.66666666666667 |
| YAMAA-01 | AEINC | SAFETY | ADAE | USUBJID | SAFFL = 'Y' | TRTEMFL = 'Y' | TRT01P | 2 | Placebo | AEINCIDENCE | COUNT | 0 |
| YAMAA-01 | AEINC | SAFETY | ADAE | USUBJID | SAFFL = 'Y' | TRTEMFL = 'Y' | TRT01P | 2 | Placebo | AEINCIDENCE | PERCENT | 0 |
| YAMAA-01 | AEINC | SAFETY | ADAE | USUBJID | SAFFL = 'Y' | TRTEMFL = 'Y' | TRT01P | 3 | Control | AEINCIDENCE | COUNT | 1 |
| YAMAA-01 | AEINC | SAFETY | ADAE | USUBJID | SAFFL = 'Y' | TRTEMFL = 'Y' | TRT01P | 3 | Control | AEINCIDENCE | PERCENT | 100 |
expected/subjects.csv
Edit6 rows| STUDYID | USUBJID | TRT01P | TRT01PN | EVENTN |
|---|---|---|---|---|
| YAMAA-01 | Drug | 1 | 1 | |
| YAMAA-01 | Drug | 1 | 0 | |
| YAMAA-01 | Drug | 1 | 1 | |
| YAMAA-01 | Placebo | 2 | 0 | |
| YAMAA-01 | Placebo | 2 | 0 | |
| YAMAA-01 | Control | 3 | 1 |
Human review and assessment
| Variable Name | Variable Label | Type | Length | Controlled Terms or Format | Origin | Core | Conversion Definition | Variable Type | Variable Order | Comments for Define |
|---|---|---|---|---|---|---|---|---|---|---|
| STUDYID | Study Identifier | Char | Collected | Copy value from SUBJECT.STUDYID. | SDTM | 1 | ||||
| ANALYSISID | Analysis Identifier | Char | Assigned | Set constant value "AEINC". | SDTM | 2 | ||||
| ANALYSISSETID | Analysis Set Identifier | Char | Assigned | Set constant value "SAFETY". | SDTM | 3 | ||||
| DATASET | Analysis Dataset | Char | Assigned | Set constant value "ADAE". | SDTM | 4 | ||||
| VARIABLE | Analysis Variable | Char | Assigned | Set constant value "USUBJID". | SDTM | 5 | ||||
| ANALYSISSET | Analysis Set Condition | Char | Assigned | Set constant value "SAFFL = 'Y'". | SDTM | 6 | ||||
| DATASUBSET | Data Subset Condition | Char | Assigned | Set constant value "TRTEMFL = 'Y'". | SDTM | 7 | ||||
| GROUPINGID | Grouping Factor Identifier | Char | Assigned | Set constant value "TRT01P". | SDTM | 8 | ||||
| METHODID | Analysis Method Identifier | Char | Assigned | Set constant value "AEINCIDENCE". | SDTM | 9 | ||||
| GROUPORDER | Grouping Level Order | Num | Collected | Copy value from SUBJECT.TRT01PN. | SDTM | 10 | ||||
| GROUPLEVEL | Grouping Level | Char | Collected | Copy value from SUBJECT.TRT01P. | SDTM | 11 | ||||
| OPERATIONID | Analysis Operation Identifier | Char | Derived | Set per constructed row; see the Row construction sheet. | SDTM | 12 | ||||
| RESULT | Analysis Operation Result | Num | Derived | If OPERATIONID = 'COUNT' then "NSUBJ"; otherwise compute 100 * NSUBJ / DENOM. | SDTM | 13 |
| Template | Include when | OPERATIONID | NSUBJ | DENOM |
|---|---|---|---|---|
| count | Constant "COUNT" | Aggregate SUM(SUBJECT.EVENTN) over records matching on the shared output keys | Aggregate COUNT(SUBJECT.USUBJID) over records matching on the shared output keys | |
| percent | Constant "PERCENT" | Aggregate SUM(SUBJECT.EVENTN) over records matching on the shared output keys | Aggregate COUNT(SUBJECT.USUBJID) over records matching on the shared output keys |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.7 | 2026-10-20 | yamaa docs build | Read a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default | ||
| 1.6 | 2026-10-17 | yamaa docs build | Describe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight | ||
| 1.5 | 2026-10-17 | yamaa docs build | Name collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part | ||
| 1.4 | 2026-10-15 | yamaa docs build | State datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks | ||
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/adam-results-ae-incidence/full.md
Edit28 linesFollowing CDISC ADaM standards, use the provided ADAE and ADSL datasets to create a RESULTS dataset with one record per planned treatment group and operation.
The output dataset should contain the following columns in this order: STUDYID, ANALYSISID, ANALYSISSETID, DATASET, VARIABLE, ANALYSISSET, DATASUBSET, GROUPINGID, GROUPORDER, GROUPLEVEL, METHODID, OPERATIONID, RESULT
ANALYSISID is "AEINC", ANALYSISSETID is "SAFETY", DATASET is "ADAE", VARIABLE is "USUBJID" and METHODID is "AEINCIDENCE". ANALYSISSET is "SAFFL = 'Y'" and DATASUBSET is "TRTEMFL = 'Y'". GROUPINGID is "TRT01P". GROUPORDER is the planned treatment code and GROUPLEVEL the planned treatment name of the record's planned treatment group, taken from the subject records in ADSL. The treatment groups and both results cover only safety subjects, the subjects with SAFFL "Y" in ADSL. A qualifying event is an adverse event with TRTEMFL "Y", and a subject with at least one qualifying event counts once, however many qualifying events the subject has. Each planned treatment group with at least one safety subject has one record with OPERATIONID "COUNT" and one with OPERATIONID "PERCENT", and a planned treatment group with no safety subjects has no records. For "COUNT", RESULT is the number of safety subjects in the group with at least one qualifying event. For "PERCENT", RESULT is that number as a percentage of all safety subjects in the group. Neither result is rounded. A planned treatment group whose safety subjects have no qualifying events keeps both records, with RESULT 0 in each.
Read the source datasets from /app/input and save the completed dataset as /app/output/results.csv.
prompts/adam-results-ae-incidence/conventions.md
Edit15 linesFollowing CDISC ADaM standards, use the provided ADAE and ADSL datasets to create a RESULTS dataset with one record per planned treatment group and operation.
The output dataset should contain the following columns in this order: STUDYID, ANALYSISID, ANALYSISSETID, DATASET, VARIABLE, ANALYSISSET, DATASUBSET, GROUPINGID, GROUPORDER, GROUPLEVEL, METHODID, OPERATIONID, RESULT
ANALYSISID is "AEINC", ANALYSISSETID is "SAFETY", DATASET is "ADAE", VARIABLE is "USUBJID" and METHODID is "AEINCIDENCE". ANALYSISSET is "SAFFL = 'Y'" and DATASUBSET is "TRTEMFL = 'Y'". GROUPINGID is "TRT01P". OPERATIONID is "COUNT" or "PERCENT". Neither result is rounded.
Read the source datasets from /app/input and save the completed dataset as /app/output/results.csv.
prompts/adam-results-ae-incidence/brief.md
Edit10 linesFollowing CDISC ADaM standards, use the provided ADAE and ADSL datasets to create a RESULTS dataset with one record per planned treatment group and operation.
The output dataset should contain the following columns in this order: STUDYID, ANALYSISID, ANALYSISSETID, DATASET, VARIABLE, ANALYSISSET, DATASUBSET, GROUPINGID, GROUPORDER, GROUPLEVEL, METHODID, OPERATIONID, RESULT
Read the source datasets from /app/input and save the completed dataset as /app/output/results.csv.
R
Edit4 lineslibrary(yamaa)yamaa_spec("input/subjects.schema.yaml")$run()$save()yamaa_spec("spec.yaml")$run()$outputPython
Edit4 linesimport yamaayamaa.spec("input/subjects.schema.yaml").run().save()yamaa.spec("spec.yaml").run().outputsolutions/adam-results-ae-incidence/result.R
Edit22 linessubjects <- read.csv("/app/input/adsl.csv", stringsAsFactors = FALSE)events <- read.csv("/app/input/adae.csv", stringsAsFactors = FALSE)subjects <- subjects[subjects$SAFFL == "Y", ]events <- events[events$TRTEMFL == "Y", ]subject_key <- paste(subjects$STUDYID, subjects$USUBJID, sep = "\r")event_key <- paste(events$STUDYID, events$USUBJID, sep = "\r")subjects$EVENTN <- as.integer(subject_key %in% event_key)groups <- split(subjects, interaction(subjects$STUDYID, subjects$TRT01PN, subjects$TRT01P, drop = TRUE))result <- do.call(rbind, lapply(groups, function(group) { count <- sum(group$EVENTN) data.frame(STUDYID = group$STUDYID[1], ANALYSISID = "AEINC", ANALYSISSETID = "SAFETY", DATASET = "ADAE", VARIABLE = "USUBJID", ANALYSISSET = "SAFFL = 'Y'", DATASUBSET = "TRTEMFL = 'Y'", GROUPINGID = "TRT01P", GROUPORDER = group$TRT01PN[1], GROUPLEVEL = group$TRT01P[1], METHODID = "AEINCIDENCE", OPERATIONID = c("COUNT", "PERCENT"), RESULT = c(count, 100 * count / nrow(group)))}))result <- result[order(result$STUDYID, result$GROUPORDER, result$OPERATIONID), ]write.csv(result, "/app/output/results.csv", row.names = FALSE, na = "")solutions/adam-results-ae-incidence/result.py
Edit89 lines# Reference solution for the yamaa benchmark adam-results-ae-incidence (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1."""Create the RESULTS dataset (AE incidence) from ADAE and ADSL.One record per planned treatment group and operation (COUNT, PERCENT),covering only safety subjects (SAFFL = 'Y' in ADSL). A qualifying eventis an adverse event with TRTEMFL = 'Y'; a subject with at least onequalifying event counts once."""import csvfrom collections import OrderedDictINPUT_DIR = "/app/input"OUTPUT_DIR = "/app/output"def read_csv(path): with open(path, newline="", encoding="utf-8") as fh: return list(csv.DictReader(fh))def main(): adsl = read_csv(f"{INPUT_DIR}/adsl.csv") adae = read_csv(f"{INPUT_DIR}/adae.csv") studyid = adsl[0]["STUDYID"] if adsl else "" # Planned treatment groups, ordered by the planned treatment code # (TRT01PN); GROUPLEVEL is the planned treatment name (TRT01P). groups = OrderedDict() # TRT01PN -> {"name": ..., "subjects": set} safety_subjects = set() for row in adsl: if row["SAFFL"] != "Y": continue safety_subjects.add(row["USUBJID"]) code = row["TRT01PN"] group = groups.setdefault(code, {"name": row["TRT01P"], "subjects": set()}) group["subjects"].add(row["USUBJID"]) # Safety subjects with at least one qualifying (TRTEMFL = 'Y') event. subjects_with_event = set() for row in adae: if row["TRTEMFL"] == "Y" and row["USUBJID"] in safety_subjects: subjects_with_event.add(row["USUBJID"]) columns = [ "STUDYID", "ANALYSISID", "ANALYSISSETID", "DATASET", "VARIABLE", "ANALYSISSET", "DATASUBSET", "GROUPINGID", "GROUPORDER", "GROUPLEVEL", "METHODID", "OPERATIONID", "RESULT", ] out_rows = [] for code in sorted(groups, key=lambda c: (int(c) if c.lstrip("-").isdigit() else c, c)): group = groups[code] total = len(group["subjects"]) count = len(group["subjects"] & subjects_with_event) percent = count / total * 100 if total else 0.0 base = { "STUDYID": studyid, "ANALYSISID": "AEINC", "ANALYSISSETID": "SAFETY", "DATASET": "ADAE", "VARIABLE": "USUBJID", "ANALYSISSET": "SAFFL = 'Y'", "DATASUBSET": "TRTEMFL = 'Y'", "GROUPINGID": "TRT01P", "GROUPORDER": code, "GROUPLEVEL": group["name"], "METHODID": "AEINCIDENCE", } count_row = dict(base, OPERATIONID="COUNT", RESULT=str(count)) percent_row = dict(base, OPERATIONID="PERCENT", RESULT=repr(percent)) out_rows.extend([count_row, percent_row]) with open(f"{OUTPUT_DIR}/results.csv", "w", newline="", encoding="utf-8") as fh: writer = csv.DictWriter(fh, fieldnames=columns) writer.writeheader() writer.writerows(out_rows) print(f"Wrote {len(out_rows)} records to {OUTPUT_DIR}/results.csv")if __name__ == "__main__": main()solutions/adam-results-ae-incidence/result.sas
Edit43 lines/* opensas reference: long-format operations from the safety population. */data subjects; length STUDYID USUBJID $80 TRT01P $20 SAFFL $1; infile "/app/input/adsl.csv" dsd dlm="," firstobs=2 truncover; input STUDYID : $80. USUBJID : $80. TRT01P : $20. TRT01PN SAFFL : $1.; if SAFFL='Y';run;data events; length STUDYID USUBJID $80 TRTEMFL $1; infile "/app/input/adae.csv" dsd dlm="," firstobs=2 truncover; input STUDYID : $80. USUBJID : $80. ASEQ TRTEMFL : $1.; if TRTEMFL='Y';run;proc sql; create table qualifying as select distinct STUDYID, USUBJID from events; create table counted as select s.STUDYID, s.USUBJID, s.TRT01P, s.TRT01PN, case when e.USUBJID is not null then 1 else 0 end as EVENTN from subjects s left join qualifying e on s.STUDYID=e.STUDYID and s.USUBJID=e.USUBJID; create table totals as select STUDYID, TRT01P, TRT01PN, sum(EVENTN) as NSUBJ, count(USUBJID) as DENOM from counted group by STUDYID, TRT01P, TRT01PN;quit;data results; length ANALYSISID $8 ANALYSISSETID $8 DATASET $8 VARIABLE $8 ANALYSISSET DATASUBSET $24 GROUPINGID $8 GROUPLEVEL $20 METHODID $16 OPERATIONID $8; set totals; ANALYSISID='AEINC'; ANALYSISSETID='SAFETY'; DATASET='ADAE'; VARIABLE='USUBJID'; ANALYSISSET="SAFFL = 'Y'"; DATASUBSET="TRTEMFL = 'Y'"; GROUPINGID='TRT01P'; GROUPORDER=TRT01PN; GROUPLEVEL=TRT01P; METHODID='AEINCIDENCE'; OPERATIONID='COUNT'; RESULT=NSUBJ; output; OPERATIONID='PERCENT'; RESULT=100*NSUBJ/DENOM; output;run;proc sql; create table final as select STUDYID, ANALYSISID, ANALYSISSETID, DATASET, VARIABLE, ANALYSISSET, DATASUBSET, GROUPINGID, GROUPORDER, GROUPLEVEL, METHODID, OPERATIONID, RESULT from results order by STUDYID, GROUPORDER, OPERATIONID;quit;proc export data=final outfile="/app/output/results.csv" dbms=csv replace;run;
yamaa