As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.
We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.
Summary
README.mdPublic entries: Python and R. Passing coverage requires an installed inventory.
Goal: report event-specific time at risk and exposure adjusted incidence per actual treatment, with subject-level risk attribution alongside it.
Input: treated subjects with first and last dose dates, and adverse event analysis records with term and complete onset date.
Variables:
TRT01AandAEDECODidentify actual treatment and the prespecified terms DIARRHOEA and HEADACHE, including terms a subject never experiences.- In ADRISK,
FIRSTDTis the first onset on or after first dose and on or before 30 days after last dose; empty if none qualifies.EVENTNis 1 when such an onset exists and 0 otherwise.RISKDAYScounts both first dose and first eligible onset, or both first dose and collection-window end when no event qualifies. - In ADINC,
NSUBJcounts subjects with an eligible event.PERSONDAYStotals their term-specific risk days, including event-free subjects.PERSONYRSis that total divided by 365.25;EAIR100isNSUBJper 100 person-years. Neither numeric result is rounded.
Note: repeated events contribute only their first eligible onset. An onset before first dose or after the collection window contributes no event and does not shorten time at risk. Different terms have separate denominators.
Inputs
Under review2 source filesinput/adae.csv
Edit10 rows| STUDYID | USUBJID | ASEQ | AEDECOD | ASTDT |
|---|---|---|---|---|
| YAMAA-01 | 1 | DIARRHOEA | 2025-01-02 | |
| YAMAA-01 | 2 | DIARRHOEA | 2025-01-08 | |
| YAMAA-01 | 3 | HEADACHE | 2025-02-02 | |
| YAMAA-01 | 1 | DIARRHOEA | 2025-02-10 | |
| YAMAA-01 | 2 | HEADACHE | 2025-02-09 | |
| YAMAA-01 | 1 | DIARRHOEA | 2025-01-04 | |
| YAMAA-01 | 2 | HEADACHE | 2025-01-01 | |
| YAMAA-01 | 1 | DIARRHOEA | 2024-12-31 | |
| YAMAA-01 | 2 | DIARRHOEA | 2025-01-07 | |
| YAMAA-01 | 3 | HEADACHE | 2025-01-22 |
input/adsl.csv
Edit6 rows| STUDYID | USUBJID | TRT01A | TRTSDT | TRTEDT |
|---|---|---|---|---|
| YAMAA-01 | Drug | 2025-01-01 | 2025-01-10 | |
| YAMAA-01 | Drug | 2025-01-01 | 2025-01-10 | |
| YAMAA-01 | Drug | 2025-01-01 | 2025-01-10 | |
| YAMAA-01 | Placebo | 2025-01-01 | 2025-01-05 | |
| YAMAA-01 | Placebo | 2025-01-01 | 2025-01-20 | |
| YAMAA-01 | Placebo | 2025-01-01 | 2025-01-10 |
Expected output
Under review16 expected rowsexpected/adinc.csv
Edit4 rows| STUDYID | TRT01A | AEDECOD | NSUBJ | PERSONDAYS | PERSONYRS | EAIR100 |
|---|---|---|---|---|---|---|
| YAMAA-01 | Drug | DIARRHOEA | 1 | 82 | 0.2245037645448323 | 445.4268292682927 |
| YAMAA-01 | Drug | HEADACHE | 2 | 113 | 0.30937713894592744 | 646.4601769911504 |
| YAMAA-01 | Placebo | DIARRHOEA | 2 | 61 | 0.16700889801505817 | 1197.5409836065576 |
| YAMAA-01 | Placebo | HEADACHE | 2 | 73 | 0.1998631074606434 | 1000.6849315068494 |
expected/adrisk.csv
Edit12 rows| STUDYID | USUBJID | TRT01A | AEDECOD | FIRSTDT | EVENTN | RISKDAYS |
|---|---|---|---|---|---|---|
| YAMAA-01 | Drug | DIARRHOEA | — | 0 | 40 | |
| YAMAA-01 | Drug | HEADACHE | — | 0 | 40 | |
| YAMAA-01 | Drug | DIARRHOEA | 2025-01-02 | 1 | 2 | |
| YAMAA-01 | Drug | HEADACHE | 2025-02-02 | 1 | 33 | |
| YAMAA-01 | Drug | DIARRHOEA | — | 0 | 40 | |
| YAMAA-01 | Drug | HEADACHE | 2025-02-09 | 1 | 40 | |
| YAMAA-01 | Placebo | DIARRHOEA | 2025-01-04 | 1 | 4 | |
| YAMAA-01 | Placebo | HEADACHE | 2025-01-01 | 1 | 1 | |
| YAMAA-01 | Placebo | DIARRHOEA | — | 0 | 50 | |
| YAMAA-01 | Placebo | HEADACHE | — | 0 | 50 | |
| YAMAA-01 | Placebo | DIARRHOEA | 2025-01-07 | 1 | 7 | |
| YAMAA-01 | Placebo | HEADACHE | 2025-01-22 | 1 | 22 |
Human review and assessment
| Dataset Name | Variable Name | Variable Label | Type | Codelist/Controlled Terms | Core | Computational Method | Origin |
|---|---|---|---|---|---|---|---|
| ADINC | STUDYID | Study Identifier | character | Copy value from ADRISK.STUDYID. | Predecessor | ||
| ADINC | TRT01A | Actual Treatment for Period 01 | character | Copy value from ADRISK.TRT01A. | Predecessor | ||
| ADINC | AEDECOD | Dictionary Derived Event Term | character | Copy value from ADRISK.AEDECOD. | Predecessor | ||
| ADINC | NSUBJ | Subjects with At Least One Eligible Event | numeric | Set per constructed row; see the Row construction sheet. | Derived | ||
| ADINC | PERSONDAYS | Total Time at Risk in Days | numeric | Set per constructed row; see the Row construction sheet. | Derived | ||
| ADINC | PERSONYRS | Total Time at Risk in Person Years | numeric | Compute PERSONDAYS / 365.25. | Derived | ||
| ADINC | EAIR100 | Incidence per 100 Person Years | numeric | Compute 100 * NSUBJ / PERSONYRS. | Derived |
| Template | Include when | NSUBJ | PERSONDAYS |
|---|---|---|---|
| incidence | Aggregate SUM(ADRISK.EVENTN) over records matching on the shared output keys | Aggregate SUM(ADRISK.RISKDAYS) over records matching on the shared output keys |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.7 | 2026-10-20 | yamaa docs build | Read a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default | ||
| 1.6 | 2026-10-17 | yamaa docs build | Describe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight | ||
| 1.5 | 2026-10-17 | yamaa docs build | Name collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part | ||
| 1.4 | 2026-10-15 | yamaa docs build | State datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks | ||
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/adam-adae-exposure-adjusted-incidence/full.md
Edit38 linesFollowing CDISC ADaM standards, use the provided ADAE and ADSL datasets to create an ADRISK dataset with one record per study and subject per event term and an ADINC dataset with one record per study and actual treatment per event term.
The ADRISK dataset should contain the following columns in this order: STUDYID, USUBJID, TRT01A, AEDECOD, FIRSTDT, EVENTN, RISKDAYS The ADINC dataset should contain the following columns in this order: STUDYID, TRT01A, AEDECOD, NSUBJ, PERSONDAYS, PERSONYRS, EAIR100
- ADRISK includes both prespecified AEDECOD terms, "DIARRHOEA" and "HEADACHE", once for each subject, including a term the subject never experienced.
- The collection window for each subject runs from the subject's first dose date through 30 days after the subject's last dose date.
- For each subject and term, FIRSTDT is the earliest onset date of that term in the collection window, and FIRSTDT has no value when no onset of that term falls in the window.
- EVENTN is 1 when FIRSTDT has a value and 0 when FIRSTDT has no value.
- RISKDAYS is the number of days from the first dose date through FIRSTDT, counting both dates, when EVENTN is 1.
- RISKDAYS is the number of days from the first dose date through the collection-window end, counting both dates, when EVENTN is 0.
- An onset before the first dose date or after the collection-window end is not eligible and does not shorten RISKDAYS.
- In ADINC, NSUBJ is the number of subjects with an EVENTN of 1 for the same study, actual treatment, and term.
- PERSONDAYS is the sum of RISKDAYS over all subjects in the same study, actual treatment, and term, including subjects with an EVENTN of 0.
- PERSONYRS is PERSONDAYS divided by 365.25, and PERSONYRS is not rounded.
- EAIR100 is 100 times NSUBJ divided by PERSONYRS, and EAIR100 is not rounded.
Read the source datasets from /app/input and save the completed ADRISK dataset as /app/output/adrisk.csv and the completed ADINC dataset as /app/output/adinc.csv.
prompts/adam-adae-exposure-adjusted-incidence/conventions.md
Edit17 linesFollowing CDISC ADaM standards, use the provided ADAE and ADSL datasets to create an ADRISK dataset with one record per study and subject per event term and an ADINC dataset with one record per study and actual treatment per event term.
The ADRISK dataset should contain the following columns in this order: STUDYID, USUBJID, TRT01A, AEDECOD, FIRSTDT, EVENTN, RISKDAYS The ADINC dataset should contain the following columns in this order: STUDYID, TRT01A, AEDECOD, NSUBJ, PERSONDAYS, PERSONYRS, EAIR100
AEDECOD is "DIARRHOEA" or "HEADACHE". RISKDAYS is the number of days, counting both dates. PERSONYRS is not rounded, and EAIR100 is not rounded.
Read the source datasets from /app/input and save the completed ADRISK dataset as /app/output/adrisk.csv and the completed ADINC dataset as /app/output/adinc.csv.
prompts/adam-adae-exposure-adjusted-incidence/brief.md
Edit13 linesFollowing CDISC ADaM standards, use the provided ADAE and ADSL datasets to create an ADRISK dataset with one record per study and subject per event term and an ADINC dataset with one record per study and actual treatment per event term.
The ADRISK dataset should contain the following columns in this order: STUDYID, USUBJID, TRT01A, AEDECOD, FIRSTDT, EVENTN, RISKDAYS The ADINC dataset should contain the following columns in this order: STUDYID, TRT01A, AEDECOD, NSUBJ, PERSONDAYS, PERSONYRS, EAIR100
Read the source datasets from /app/input and save the completed ADRISK dataset as /app/output/adrisk.csv and the completed ADINC dataset as /app/output/adinc.csv.
R
Edit4 lineslibrary(yamaa)yamaa_spec("spec_risk.yaml")$run()$save()yamaa_spec("spec_rate.yaml")$run()$outputPython
Edit4 linesimport yamaayamaa.spec("spec_risk.yaml").run().save()yamaa.spec("spec_rate.yaml").run().outputsolutions/adam-adae-exposure-adjusted-incidence/result.R
Edit34 linessubjects <- read.csv("/app/input/adsl.csv", stringsAsFactors = FALSE)events <- read.csv("/app/input/adae.csv", stringsAsFactors = FALSE)events$ASTDT <- as.Date(events$ASTDT)risk <- do.call(rbind, lapply(seq_len(nrow(subjects)), function(i) { subject <- subjects[i, ] start <- as.Date(subject$TRTSDT) end <- as.Date(subject$TRTEDT) + 30 do.call(rbind, lapply(c("DIARRHOEA", "HEADACHE"), function(term) { selected <- events$STUDYID == subject$STUDYID & events$USUBJID == subject$USUBJID & events$AEDECOD == term & !is.na(events$ASTDT) & events$ASTDT >= start & events$ASTDT <= end onsets <- events$ASTDT[selected] first <- if (length(onsets)) min(onsets) else as.Date(NA) endpoint <- if (length(onsets)) first else end data.frame(STUDYID = subject$STUDYID, USUBJID = subject$USUBJID, TRT01A = subject$TRT01A, AEDECOD = term, FIRSTDT = first, EVENTN = as.integer(length(onsets) > 0), RISKDAYS = as.integer(endpoint - start) + 1L) }))}))groups <- split(risk, interaction(risk$STUDYID, risk$TRT01A, risk$AEDECOD, drop = TRUE))rates <- do.call(rbind, lapply(groups, function(group) { count <- sum(group$EVENTN) days <- sum(group$RISKDAYS) years <- days / 365.25 data.frame(STUDYID = group$STUDYID[1], TRT01A = group$TRT01A[1], AEDECOD = group$AEDECOD[1], NSUBJ = count, PERSONDAYS = days, PERSONYRS = years, EAIR100 = 100 * count / years)}))risk <- risk[order(risk$STUDYID, risk$USUBJID, risk$AEDECOD), ]rates <- rates[order(rates$STUDYID, rates$TRT01A, rates$AEDECOD), ]write.csv(risk, "/app/output/adrisk.csv", row.names = FALSE, na = "")write.csv(rates, "/app/output/adinc.csv", row.names = FALSE, na = "")solutions/adam-adae-exposure-adjusted-incidence/result.py
Edit156 lines# Reference solution for the yamaa benchmark adam-adae-exposure-adjusted-incidence (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.import csvfrom datetime import date, timedeltafrom pathlib import PathINPUT_DIR = Path("/app/input")OUTPUT_DIR = Path("/app/output")TERMS = ["DIARRHOEA", "HEADACHE"]def parse_date(value: str) -> date: return date.fromisoformat(value)def main() -> None: with open(INPUT_DIR / "adsl.csv", newline="", encoding="utf-8") as handle: adsl_rows = list(csv.DictReader(handle)) with open(INPUT_DIR / "adae.csv", newline="", encoding="utf-8") as handle: adae_rows = list(csv.DictReader(handle)) # One subject record, retaining the order in which subjects appear. subjects = {} subject_order = [] for row in adsl_rows: key = (row["STUDYID"], row["USUBJID"]) first_dose = parse_date(row["TRTSDT"]) last_dose = parse_date(row["TRTEDT"]) if key not in subjects: subjects[key] = { "STUDYID": row["STUDYID"], "USUBJID": row["USUBJID"], "TRT01A": row["TRT01A"], "first_dose": first_dose, "last_dose": last_dose, } subject_order.append(key) else: subjects[key]["first_dose"] = min( subjects[key]["first_dose"], first_dose ) subjects[key]["last_dose"] = max(subjects[key]["last_dose"], last_dose) # Index event onset dates by subject and prespecified term. onsets = {} for row in adae_rows: if row["AEDECOD"] not in TERMS or not row["ASTDT"]: continue key = (row["STUDYID"], row["USUBJID"], row["AEDECOD"]) onsets.setdefault(key, []).append(parse_date(row["ASTDT"])) risk_rows = [] for key in subject_order: subject = subjects[key] first_dose = subject["first_dose"] window_end = subject["last_dose"] + timedelta(days=30) for term in TERMS: in_window = [ onset for onset in onsets.get( (subject["STUDYID"], subject["USUBJID"], term), [] ) if first_dose <= onset <= window_end ] first_date = min(in_window) if in_window else None event_n = 1 if in_window else 0 risk_end = first_date if event_n else window_end risk_rows.append( { "STUDYID": subject["STUDYID"], "USUBJID": subject["USUBJID"], "TRT01A": subject["TRT01A"], "AEDECOD": term, "FIRSTDT": first_date.isoformat() if first_date else "", "EVENTN": event_n, "RISKDAYS": (risk_end - first_dose).days + 1, } ) # Aggregate in the order treatments and terms first appear in ADRISK. incidence = {} incidence_order = [] for row in risk_rows: key = (row["STUDYID"], row["TRT01A"], row["AEDECOD"]) if key not in incidence: incidence[key] = { "STUDYID": row["STUDYID"], "TRT01A": row["TRT01A"], "AEDECOD": row["AEDECOD"], "NSUBJ": 0, "PERSONDAYS": 0, } incidence_order.append(key) incidence[key]["NSUBJ"] += row["EVENTN"] incidence[key]["PERSONDAYS"] += row["RISKDAYS"] incidence_rows = [] for key in incidence_order: row = incidence[key] person_years = row["PERSONDAYS"] / 365.25 incidence_rows.append( { "STUDYID": row["STUDYID"], "TRT01A": row["TRT01A"], "AEDECOD": row["AEDECOD"], "NSUBJ": row["NSUBJ"], "PERSONDAYS": row["PERSONDAYS"], "PERSONYRS": person_years, "EAIR100": 100 * row["NSUBJ"] / person_years, } ) OUTPUT_DIR.mkdir(parents=True, exist_ok=True) risk_fields = [ "STUDYID", "USUBJID", "TRT01A", "AEDECOD", "FIRSTDT", "EVENTN", "RISKDAYS", ] with open( OUTPUT_DIR / "adrisk.csv", "w", newline="", encoding="utf-8" ) as handle: writer = csv.DictWriter(handle, fieldnames=risk_fields) writer.writeheader() writer.writerows(risk_rows) incidence_fields = [ "STUDYID", "TRT01A", "AEDECOD", "NSUBJ", "PERSONDAYS", "PERSONYRS", "EAIR100", ] with open( OUTPUT_DIR / "adinc.csv", "w", newline="", encoding="utf-8" ) as handle: writer = csv.DictWriter(handle, fieldnames=incidence_fields) writer.writeheader() writer.writerows(incidence_rows)if __name__ == "__main__": main()solutions/adam-adae-exposure-adjusted-incidence/result.sas
Edit58 lines/* opensas reference: first qualifying onset defines each term's risk. */data subjects; length STUDYID USUBJID $80 TRT01A $20 start_text end_text $10; infile "/app/input/adsl.csv" dsd dlm="," firstobs=2 truncover; input STUDYID : $80. USUBJID : $80. TRT01A : $20. start_text : $10. end_text : $10.; TRTSDT=input(start_text,yymmdd10.); TRTEDT=input(end_text,yymmdd10.);run;data events; length STUDYID USUBJID $80 AEDECOD $20 date_text $10; infile "/app/input/adae.csv" dsd dlm="," firstobs=2 truncover; input STUDYID : $80. USUBJID : $80. ASEQ AEDECOD : $20. date_text : $10.; ASTDT=input(date_text,yymmdd10.);run;data population; length AEDECOD $20; set subjects; AEDECOD='DIARRHOEA'; output; AEDECOD='HEADACHE'; output;run;proc sql; create table selected as select p.STUDYID, p.USUBJID, p.TRT01A, p.AEDECOD, p.TRTSDT, p.TRTEDT, min(e.ASTDT) as FIRSTDT from population p left join events e on p.STUDYID=e.STUDYID and p.USUBJID=e.USUBJID and p.AEDECOD=e.AEDECOD and e.ASTDT >= p.TRTSDT and e.ASTDT <= p.TRTEDT+30 group by p.STUDYID, p.USUBJID, p.TRT01A, p.AEDECOD, p.TRTSDT, p.TRTEDT;quit;data risk; set selected; EVENTN=not missing(FIRSTDT); if EVENTN then RISKDAYS=FIRSTDT-TRTSDT+1; else RISKDAYS=TRTEDT+30-TRTSDT+1; format FIRSTDT yymmdd10.;run;proc sql; create table risk_output as select STUDYID, USUBJID, TRT01A, AEDECOD, put(FIRSTDT,yymmdd10.) as FIRSTDT, EVENTN, RISKDAYS from risk order by STUDYID, USUBJID, AEDECOD; create table totals as select STUDYID, TRT01A, AEDECOD, sum(EVENTN) as NSUBJ, sum(RISKDAYS) as PERSONDAYS from risk group by STUDYID, TRT01A, AEDECOD;quit;data rates; set totals; PERSONYRS=PERSONDAYS/365.25; EAIR100=100*NSUBJ/PERSONYRS;run;proc sql; create table rate_output as select STUDYID, TRT01A, AEDECOD, NSUBJ, PERSONDAYS, PERSONYRS, EAIR100 from rates order by STUDYID, TRT01A, AEDECOD;quit;proc export data=risk_output outfile="/app/output/adrisk.csv" dbms=csv replace; run;proc export data=rate_output outfile="/app/output/adinc.csv" dbms=csv replace; run;
yamaa