Review ask

As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.

We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.

Summary

Lifecycle: draftEditUnder review
What this benchmark means
README.md

Public entries: Python and R. Passing coverage requires an installed inventory.

Goal: report event-specific time at risk and exposure adjusted incidence per actual treatment, with subject-level risk attribution alongside it.

Input: treated subjects with first and last dose dates, and adverse event analysis records with term and complete onset date.

Variables:

  • TRT01A and AEDECOD identify actual treatment and the prespecified terms DIARRHOEA and HEADACHE, including terms a subject never experiences.
  • In ADRISK, FIRSTDT is the first onset on or after first dose and on or before 30 days after last dose; empty if none qualifies. EVENTN is 1 when such an onset exists and 0 otherwise. RISKDAYS counts both first dose and first eligible onset, or both first dose and collection-window end when no event qualifies.
  • In ADINC, NSUBJ counts subjects with an eligible event. PERSONDAYS totals their term-specific risk days, including event-free subjects. PERSONYRS is that total divided by 365.25; EAIR100 is NSUBJ per 100 person-years. Neither numeric result is rounded.

Note: repeated events contribute only their first eligible onset. An onset before first dose or after the collection window contributes no event and does not shorten time at risk. Different terms have separate denominators.

Inputs

Under review
2 source files

input/adae.csv

Edit
10 rows
input/adae.csv
STUDYIDUSUBJIDASEQAEDECODASTDT
YAMAA-011DIARRHOEA2025-01-02
YAMAA-012DIARRHOEA2025-01-08
YAMAA-013HEADACHE2025-02-02
YAMAA-011DIARRHOEA2025-02-10
YAMAA-012HEADACHE2025-02-09
YAMAA-011DIARRHOEA2025-01-04
YAMAA-012HEADACHE2025-01-01
YAMAA-011DIARRHOEA2024-12-31
YAMAA-012DIARRHOEA2025-01-07
YAMAA-013HEADACHE2025-01-22

input/adsl.csv

Edit
6 rows
input/adsl.csv
STUDYIDUSUBJIDTRT01ATRTSDTTRTEDT
YAMAA-01Drug2025-01-012025-01-10
YAMAA-01Drug2025-01-012025-01-10
YAMAA-01Drug2025-01-012025-01-10
YAMAA-01Placebo2025-01-012025-01-05
YAMAA-01Placebo2025-01-012025-01-20
YAMAA-01Placebo2025-01-012025-01-10

Expected output

Under review
16 expected rows

expected/adinc.csv

Edit
4 rows
expected/adinc.csv
STUDYIDTRT01AAEDECODNSUBJPERSONDAYSPERSONYRSEAIR100
YAMAA-01DrugDIARRHOEA1820.2245037645448323445.4268292682927
YAMAA-01DrugHEADACHE21130.30937713894592744646.4601769911504
YAMAA-01PlaceboDIARRHOEA2610.167008898015058171197.5409836065576
YAMAA-01PlaceboHEADACHE2730.19986310746064341000.6849315068494

expected/adrisk.csv

Edit
12 rows
expected/adrisk.csv
STUDYIDUSUBJIDTRT01AAEDECODFIRSTDTEVENTNRISKDAYS
YAMAA-01DrugDIARRHOEA—040
YAMAA-01DrugHEADACHE—040
YAMAA-01DrugDIARRHOEA2025-01-0212
YAMAA-01DrugHEADACHE2025-02-02133
YAMAA-01DrugDIARRHOEA—040
YAMAA-01DrugHEADACHE2025-02-09140
YAMAA-01PlaceboDIARRHOEA2025-01-0414
YAMAA-01PlaceboHEADACHE2025-01-0111
YAMAA-01PlaceboDIARRHOEA—050
YAMAA-01PlaceboHEADACHE—050
YAMAA-01PlaceboDIARRHOEA2025-01-0717
YAMAA-01PlaceboHEADACHE2025-01-22122
Shaded columns are derived— = missing value

Human review and assessment

Your review

3 questions and your comments · View benchmark discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Dataset NameVariable NameVariable LabelTypeCodelist/Controlled TermsCoreComputational MethodOrigin
ADINCSTUDYIDStudy IdentifiercharacterCopy value from ADRISK.STUDYID.Predecessor
ADINCTRT01AActual Treatment for Period 01characterCopy value from ADRISK.TRT01A.Predecessor
ADINCAEDECODDictionary Derived Event TermcharacterCopy value from ADRISK.AEDECOD.Predecessor
ADINCNSUBJSubjects with At Least One Eligible EventnumericSet per constructed row; see the Row construction sheet.Derived
ADINCPERSONDAYSTotal Time at Risk in DaysnumericSet per constructed row; see the Row construction sheet.Derived
ADINCPERSONYRSTotal Time at Risk in Person YearsnumericCompute PERSONDAYS / 365.25.Derived
ADINCEAIR100Incidence per 100 Person YearsnumericCompute 100 * NSUBJ / PERSONYRS.Derived
TemplateInclude whenNSUBJPERSONDAYS
incidenceAggregate SUM(ADRISK.EVENTN) over records matching on the shared output keysAggregate SUM(ADRISK.RISKDAYS) over records matching on the shared output keys
VersionDateAuthorDescriptionReviewerSign-off
1.72026-10-20yamaa docs buildRead a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default
1.62026-10-17yamaa docs buildDescribe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight
1.52026-10-17yamaa docs buildName collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part
1.42026-10-15yamaa docs buildState datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/adam-adae-exposure-adjusted-incidence/full.md

Edit
38 lines

Following CDISC ADaM standards, use the provided ADAE and ADSL datasets to create an ADRISK dataset with one record per study and subject per event term and an ADINC dataset with one record per study and actual treatment per event term.

The ADRISK dataset should contain the following columns in this order: STUDYID, USUBJID, TRT01A, AEDECOD, FIRSTDT, EVENTN, RISKDAYS The ADINC dataset should contain the following columns in this order: STUDYID, TRT01A, AEDECOD, NSUBJ, PERSONDAYS, PERSONYRS, EAIR100

  • ADRISK includes both prespecified AEDECOD terms, "DIARRHOEA" and "HEADACHE", once for each subject, including a term the subject never experienced.
  • The collection window for each subject runs from the subject's first dose date through 30 days after the subject's last dose date.
  • For each subject and term, FIRSTDT is the earliest onset date of that term in the collection window, and FIRSTDT has no value when no onset of that term falls in the window.
  • EVENTN is 1 when FIRSTDT has a value and 0 when FIRSTDT has no value.
  • RISKDAYS is the number of days from the first dose date through FIRSTDT, counting both dates, when EVENTN is 1.
  • RISKDAYS is the number of days from the first dose date through the collection-window end, counting both dates, when EVENTN is 0.
  • An onset before the first dose date or after the collection-window end is not eligible and does not shorten RISKDAYS.
  • In ADINC, NSUBJ is the number of subjects with an EVENTN of 1 for the same study, actual treatment, and term.
  • PERSONDAYS is the sum of RISKDAYS over all subjects in the same study, actual treatment, and term, including subjects with an EVENTN of 0.
  • PERSONYRS is PERSONDAYS divided by 365.25, and PERSONYRS is not rounded.
  • EAIR100 is 100 times NSUBJ divided by PERSONYRS, and EAIR100 is not rounded.

Read the source datasets from /app/input and save the completed ADRISK dataset as /app/output/adrisk.csv and the completed ADINC dataset as /app/output/adinc.csv.

prompts/adam-adae-exposure-adjusted-incidence/conventions.md

Edit
17 lines

Following CDISC ADaM standards, use the provided ADAE and ADSL datasets to create an ADRISK dataset with one record per study and subject per event term and an ADINC dataset with one record per study and actual treatment per event term.

The ADRISK dataset should contain the following columns in this order: STUDYID, USUBJID, TRT01A, AEDECOD, FIRSTDT, EVENTN, RISKDAYS The ADINC dataset should contain the following columns in this order: STUDYID, TRT01A, AEDECOD, NSUBJ, PERSONDAYS, PERSONYRS, EAIR100

AEDECOD is "DIARRHOEA" or "HEADACHE". RISKDAYS is the number of days, counting both dates. PERSONYRS is not rounded, and EAIR100 is not rounded.

Read the source datasets from /app/input and save the completed ADRISK dataset as /app/output/adrisk.csv and the completed ADINC dataset as /app/output/adinc.csv.

prompts/adam-adae-exposure-adjusted-incidence/brief.md

Edit
13 lines

Following CDISC ADaM standards, use the provided ADAE and ADSL datasets to create an ADRISK dataset with one record per study and subject per event term and an ADINC dataset with one record per study and actual treatment per event term.

The ADRISK dataset should contain the following columns in this order: STUDYID, USUBJID, TRT01A, AEDECOD, FIRSTDT, EVENTN, RISKDAYS The ADINC dataset should contain the following columns in this order: STUDYID, TRT01A, AEDECOD, NSUBJ, PERSONDAYS, PERSONYRS, EAIR100

Read the source datasets from /app/input and save the completed ADRISK dataset as /app/output/adrisk.csv and the completed ADINC dataset as /app/output/adinc.csv.

yamaa code

Public calls generated from benchmark.yaml; results must match the expected output

R

Edit
4 lines
library(yamaa)yamaa_spec("spec_risk.yaml")$run()$save()yamaa_spec("spec_rate.yaml")$run()$output

Python

Edit
4 lines
import yamaayamaa.spec("spec_risk.yaml").run().save()yamaa.spec("spec_rate.yaml").run().output

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/adam-adae-exposure-adjusted-incidence/result.R

Edit
34 lines
subjects <- read.csv("/app/input/adsl.csv", stringsAsFactors = FALSE)events <- read.csv("/app/input/adae.csv", stringsAsFactors = FALSE)events$ASTDT <- as.Date(events$ASTDT)risk <- do.call(rbind, lapply(seq_len(nrow(subjects)), function(i) {  subject <- subjects[i, ]  start <- as.Date(subject$TRTSDT)  end <- as.Date(subject$TRTEDT) + 30  do.call(rbind, lapply(c("DIARRHOEA", "HEADACHE"), function(term) {    selected <- events$STUDYID == subject$STUDYID &      events$USUBJID == subject$USUBJID & events$AEDECOD == term &      !is.na(events$ASTDT) & events$ASTDT >= start & events$ASTDT <= end    onsets <- events$ASTDT[selected]    first <- if (length(onsets)) min(onsets) else as.Date(NA)    endpoint <- if (length(onsets)) first else end    data.frame(STUDYID = subject$STUDYID, USUBJID = subject$USUBJID,      TRT01A = subject$TRT01A, AEDECOD = term, FIRSTDT = first,      EVENTN = as.integer(length(onsets) > 0),      RISKDAYS = as.integer(endpoint - start) + 1L)  }))}))groups <- split(risk, interaction(risk$STUDYID, risk$TRT01A, risk$AEDECOD,                                  drop = TRUE))rates <- do.call(rbind, lapply(groups, function(group) {  count <- sum(group$EVENTN)  days <- sum(group$RISKDAYS)  years <- days / 365.25  data.frame(STUDYID = group$STUDYID[1], TRT01A = group$TRT01A[1],    AEDECOD = group$AEDECOD[1], NSUBJ = count, PERSONDAYS = days,    PERSONYRS = years, EAIR100 = 100 * count / years)}))risk <- risk[order(risk$STUDYID, risk$USUBJID, risk$AEDECOD), ]rates <- rates[order(rates$STUDYID, rates$TRT01A, rates$AEDECOD), ]write.csv(risk, "/app/output/adrisk.csv", row.names = FALSE, na = "")write.csv(rates, "/app/output/adinc.csv", row.names = FALSE, na = "")

solutions/adam-adae-exposure-adjusted-incidence/result.py

Edit
156 lines
# Reference solution for the yamaa benchmark adam-adae-exposure-adjusted-incidence (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.import csvfrom datetime import date, timedeltafrom pathlib import PathINPUT_DIR = Path("/app/input")OUTPUT_DIR = Path("/app/output")TERMS = ["DIARRHOEA", "HEADACHE"]def parse_date(value: str) -> date:    return date.fromisoformat(value)def main() -> None:    with open(INPUT_DIR / "adsl.csv", newline="", encoding="utf-8") as handle:        adsl_rows = list(csv.DictReader(handle))    with open(INPUT_DIR / "adae.csv", newline="", encoding="utf-8") as handle:        adae_rows = list(csv.DictReader(handle))    # One subject record, retaining the order in which subjects appear.    subjects = {}    subject_order = []    for row in adsl_rows:        key = (row["STUDYID"], row["USUBJID"])        first_dose = parse_date(row["TRTSDT"])        last_dose = parse_date(row["TRTEDT"])        if key not in subjects:            subjects[key] = {                "STUDYID": row["STUDYID"],                "USUBJID": row["USUBJID"],                "TRT01A": row["TRT01A"],                "first_dose": first_dose,                "last_dose": last_dose,            }            subject_order.append(key)        else:            subjects[key]["first_dose"] = min(                subjects[key]["first_dose"], first_dose            )            subjects[key]["last_dose"] = max(subjects[key]["last_dose"], last_dose)    # Index event onset dates by subject and prespecified term.    onsets = {}    for row in adae_rows:        if row["AEDECOD"] not in TERMS or not row["ASTDT"]:            continue        key = (row["STUDYID"], row["USUBJID"], row["AEDECOD"])        onsets.setdefault(key, []).append(parse_date(row["ASTDT"]))    risk_rows = []    for key in subject_order:        subject = subjects[key]        first_dose = subject["first_dose"]        window_end = subject["last_dose"] + timedelta(days=30)        for term in TERMS:            in_window = [                onset                for onset in onsets.get(                    (subject["STUDYID"], subject["USUBJID"], term), []                )                if first_dose <= onset <= window_end            ]            first_date = min(in_window) if in_window else None            event_n = 1 if in_window else 0            risk_end = first_date if event_n else window_end            risk_rows.append(                {                    "STUDYID": subject["STUDYID"],                    "USUBJID": subject["USUBJID"],                    "TRT01A": subject["TRT01A"],                    "AEDECOD": term,                    "FIRSTDT": first_date.isoformat() if first_date else "",                    "EVENTN": event_n,                    "RISKDAYS": (risk_end - first_dose).days + 1,                }            )    # Aggregate in the order treatments and terms first appear in ADRISK.    incidence = {}    incidence_order = []    for row in risk_rows:        key = (row["STUDYID"], row["TRT01A"], row["AEDECOD"])        if key not in incidence:            incidence[key] = {                "STUDYID": row["STUDYID"],                "TRT01A": row["TRT01A"],                "AEDECOD": row["AEDECOD"],                "NSUBJ": 0,                "PERSONDAYS": 0,            }            incidence_order.append(key)        incidence[key]["NSUBJ"] += row["EVENTN"]        incidence[key]["PERSONDAYS"] += row["RISKDAYS"]    incidence_rows = []    for key in incidence_order:        row = incidence[key]        person_years = row["PERSONDAYS"] / 365.25        incidence_rows.append(            {                "STUDYID": row["STUDYID"],                "TRT01A": row["TRT01A"],                "AEDECOD": row["AEDECOD"],                "NSUBJ": row["NSUBJ"],                "PERSONDAYS": row["PERSONDAYS"],                "PERSONYRS": person_years,                "EAIR100": 100 * row["NSUBJ"] / person_years,            }        )    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)    risk_fields = [        "STUDYID",        "USUBJID",        "TRT01A",        "AEDECOD",        "FIRSTDT",        "EVENTN",        "RISKDAYS",    ]    with open(        OUTPUT_DIR / "adrisk.csv", "w", newline="", encoding="utf-8"    ) as handle:        writer = csv.DictWriter(handle, fieldnames=risk_fields)        writer.writeheader()        writer.writerows(risk_rows)    incidence_fields = [        "STUDYID",        "TRT01A",        "AEDECOD",        "NSUBJ",        "PERSONDAYS",        "PERSONYRS",        "EAIR100",    ]    with open(        OUTPUT_DIR / "adinc.csv", "w", newline="", encoding="utf-8"    ) as handle:        writer = csv.DictWriter(handle, fieldnames=incidence_fields)        writer.writeheader()        writer.writerows(incidence_rows)if __name__ == "__main__":    main()

solutions/adam-adae-exposure-adjusted-incidence/result.sas

Edit
58 lines
/* opensas reference: first qualifying onset defines each term's risk. */data subjects;  length STUDYID USUBJID $80 TRT01A $20 start_text end_text $10;  infile "/app/input/adsl.csv" dsd dlm="," firstobs=2 truncover;  input STUDYID : $80. USUBJID : $80. TRT01A : $20.        start_text : $10. end_text : $10.;  TRTSDT=input(start_text,yymmdd10.); TRTEDT=input(end_text,yymmdd10.);run;data events;  length STUDYID USUBJID $80 AEDECOD $20 date_text $10;  infile "/app/input/adae.csv" dsd dlm="," firstobs=2 truncover;  input STUDYID : $80. USUBJID : $80. ASEQ AEDECOD : $20.        date_text : $10.;  ASTDT=input(date_text,yymmdd10.);run;data population;  length AEDECOD $20;  set subjects;  AEDECOD='DIARRHOEA'; output;  AEDECOD='HEADACHE'; output;run;proc sql;  create table selected as select p.STUDYID, p.USUBJID, p.TRT01A,    p.AEDECOD, p.TRTSDT, p.TRTEDT, min(e.ASTDT) as FIRSTDT  from population p left join events e    on p.STUDYID=e.STUDYID and p.USUBJID=e.USUBJID    and p.AEDECOD=e.AEDECOD and e.ASTDT >= p.TRTSDT    and e.ASTDT <= p.TRTEDT+30  group by p.STUDYID, p.USUBJID, p.TRT01A, p.AEDECOD, p.TRTSDT, p.TRTEDT;quit;data risk;  set selected;  EVENTN=not missing(FIRSTDT);  if EVENTN then RISKDAYS=FIRSTDT-TRTSDT+1;  else RISKDAYS=TRTEDT+30-TRTSDT+1;  format FIRSTDT yymmdd10.;run;proc sql;  create table risk_output as select STUDYID, USUBJID, TRT01A, AEDECOD,    put(FIRSTDT,yymmdd10.) as FIRSTDT, EVENTN, RISKDAYS from risk  order by STUDYID, USUBJID, AEDECOD;  create table totals as select STUDYID, TRT01A, AEDECOD,    sum(EVENTN) as NSUBJ, sum(RISKDAYS) as PERSONDAYS  from risk group by STUDYID, TRT01A, AEDECOD;quit;data rates;  set totals;  PERSONYRS=PERSONDAYS/365.25; EAIR100=100*NSUBJ/PERSONYRS;run;proc sql;  create table rate_output as select STUDYID, TRT01A, AEDECOD, NSUBJ,    PERSONDAYS, PERSONYRS, EAIR100 from rates  order by STUDYID, TRT01A, AEDECOD;quit;proc export data=risk_output outfile="/app/output/adrisk.csv"  dbms=csv replace; run;proc export data=rate_output outfile="/app/output/adinc.csv"  dbms=csv replace; run;