yamaa As a clinical trial statistician or programmer, challenge anything in this benchmark: describe a problem in Comments, or propose a fix with the Edit button on any source file.
- Task: Would a real study need this derivation, specified this way? Does it follow CDISC ADaM: variable names, controlled terms, record structure? Flag any rule that is ambiguous or that a sponsor would handle differently.
- Inputs: Do they look like data collected in a real trial, and do they include the situations you would expect to meet in one?
- Expected output: Would independent QC programming reproduce every value from these inputs and rules?
The other tabs beside Comments are background for assessment, not under review.
Summary
EditUnder reviewWhat this benchmark meansREADME.mdGoal: derive one overall-survival record per subject with
parameter code OS (Overall Survival), carrying STARTDT,
ADT, AVAL, CNSR, EVNTDESC, CNSDTDSC, SRCDOM,
SRCVAR, and SRCSEQ.
Input: subject-level randomization and last-known-alive dates
(RANDDT, LSTALVDT) alongside response records holding death
analysis dates (ADT). The death selection keeps only dated
response records where PARAMCD is DEATH, AVALC is Y, and
ANL01FL is Y; it takes the one with the earliest ADT (on a
tied date, the lower sequence number wins) and treats a subject
with no such record as having no death.
Variables:
STARTDTis the randomization date (RANDDT), the origin from which survival time is counted.ADTis the analysis date: the death date when a qualifying death falls on or afterSTARTDT, andSTARTDTitself when the death falls before it. With no death,ADTis the last-alive date when that date falls after randomization, else the randomization date (including a tie).AVALis the number of days fromSTARTDTthroughADT, counting both endpoints.CNSRis0when the subject has a qualifying death and1otherwise.EVNTDESCisDeathwhen the subject has a qualifying death,Alivewhen censoring at a last-alive date after randomization, andRandomizationotherwise.CNSDTDSCis blank for a death,Alive During Studywhen censoring at the last-alive date, andRandomizationwhen censoring at randomization.SRCDOMisADRSfor a death andADSLfor censoring.SRCVARisADTfor a death,LSTALVDTwhen censoring at the last-alive date, andRANDDTotherwise.SRCSEQis the sequence number from the selected death record, and blank when the record is censored.
Note: a death dated before randomization still counts as an
event: ADT is set to randomization while CNSR, EVNTDESC,
and the source columns still reflect the death.
Inputs
Under review2 source filesinput/adrs.csv
Edit10 rows| STUDYID | USUBJID | PARAMCD | AVALC | ANL01FL | ADT | ASEQ |
|---|---|---|---|---|---|---|
| YAMAA-01 | DEATH | Y | Y | 2026-06-01 | 1 | |
| YAMAA-01 | DEATH | N | Y | 2026-06-01 | 1 | |
| YAMAA-01 | DEATH | Y | Y | 2026-04-01 | 2 | |
| YAMAA-01 | DEATH | Y | Y | 2026-02-01 | 1 | |
| YAMAA-01 | DEATH | Y | Y | 2026-07-01 | 3 | |
| YAMAA-01 | DEATH | Y | Y | 2026-07-01 | 2 | |
| YAMAA-01 | BOR | Y | Y | 2026-02-01 | 3 | |
| YAMAA-01 | DEATH | Y | N | 2026-03-01 | 1 | |
| YAMAA-01 | DEATH | Y | Y | 2026-05-01 | 2 | |
| YAMAA-01 | DEATH | Y | Y | — | 1 |
input/adsl.csv
Edit10 rows| STUDYID | USUBJID | RANDDT | LSTALVDT |
|---|---|---|---|
| YAMAA-01 | 2026-01-01 | — | |
| YAMAA-01 | 2026-01-01 | 2026-09-01 | |
| YAMAA-01 | 2026-01-01 | — | |
| YAMAA-01 | 2026-03-01 | 2026-02-01 | |
| YAMAA-01 | 2026-04-01 | 2026-04-01 | |
| YAMAA-01 | 2026-05-01 | — | |
| YAMAA-01 | 2026-02-01 | — | |
| YAMAA-01 | 2026-01-01 | 2026-12-01 | |
| YAMAA-01 | 2026-01-01 | — | |
| YAMAA-01 | 2026-01-01 | 2026-12-01 |
Expected output
Under review10 expected rowsexpected/adtte.csv
Edit10 rows| STUDYID | USUBJID | PARAMCD | PARAM | STARTDT | ADT | AVAL | CNSR | EVNTDESC | CNSDTDSC | SRCDOM | SRCVAR | SRCSEQ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| YAMAA-01 | OS | Overall Survival | 2026-01-01 | 2026-06-01 | 152 | 0 | Death | — | ADRS | ADT | 1 | |
| YAMAA-01 | OS | Overall Survival | 2026-01-01 | 2026-09-01 | 244 | 1 | Alive | Alive During Study | ADSL | LSTALVDT | — | |
| YAMAA-01 | OS | Overall Survival | 2026-01-01 | 2026-01-01 | 1 | 1 | Randomization | Randomization | ADSL | RANDDT | — | |
| YAMAA-01 | OS | Overall Survival | 2026-03-01 | 2026-03-01 | 1 | 1 | Randomization | Randomization | ADSL | RANDDT | — | |
| YAMAA-01 | OS | Overall Survival | 2026-04-01 | 2026-04-01 | 1 | 1 | Randomization | Randomization | ADSL | RANDDT | — | |
| YAMAA-01 | OS | Overall Survival | 2026-05-01 | 2026-05-01 | 1 | 0 | Death | — | ADRS | ADT | 2 | |
| YAMAA-01 | OS | Overall Survival | 2026-02-01 | 2026-02-01 | 1 | 0 | Death | — | ADRS | ADT | 1 | |
| YAMAA-01 | OS | Overall Survival | 2026-01-01 | 2026-07-01 | 182 | 0 | Death | — | ADRS | ADT | 2 | |
| YAMAA-01 | OS | Overall Survival | 2026-01-01 | 2026-05-01 | 121 | 0 | Death | — | ADRS | ADT | 2 | |
| YAMAA-01 | OS | Overall Survival | 2026-01-01 | 2026-12-01 | 335 | 1 | Alive | Alive During Study | ADSL | LSTALVDT | — |
Comments and assessment materials
| Dataset Name | Variable Name | Variable Label | Type | Codelist/Controlled Terms | Core | Computational Method | Origin |
|---|---|---|---|---|---|---|---|
| ADTTE | STUDYID | Study Identifier | character | Copy value from ADSL.STUDYID. | Predecessor | ||
| ADTTE | USUBJID | Unique Subject Identifier | character | Copy value from ADSL.USUBJID. | Predecessor | ||
| ADTTE | PARAMCD | Parameter Code | character | Set constant value "OS". | Assigned | ||
| ADTTE | PARAM | Parameter | character | Set constant value "Overall Survival". | Assigned | ||
| ADTTE | STARTDT | Time-to-Event Origin Date | numeric | ISO 8601 | Copy value from ADSL.RANDDT. | Predecessor | |
| ADTTE | ADT | Analysis Date | numeric | ISO 8601 | If DEATHDT >= STARTDT then "DEATHDT"; If DEATHDT IS NOT NULL then "STARTDT"; If CENSORDT1 > STARTDT then "CENSORDT1"; otherwise "STARTDT". | Derived | |
| ADTTE | AVAL | Analysis Value | numeric | Whole days from STARTDT to ADT, counting both endpoints. | Derived | ||
| ADTTE | CNSR | Censor | numeric | If DEATHDT IS NOT NULL then "0"; otherwise "1". | Derived | ||
| ADTTE | EVNTDESC | Event or Censoring Description | character | If DEATHDT IS NOT NULL then "Death"; If CENSORDT1 > STARTDT then "Alive"; otherwise "Randomization". | Derived | ||
| ADTTE | CNSDTDSC | Censoring Date Description | character | If DEATHDT IS NULL AND CENSORDT1 > STARTDT then "Alive During Study"; If DEATHDT IS NULL then "Randomization"; otherwise blank. | Derived | ||
| ADTTE | SRCDOM | Source Data | character | If DEATHDT IS NOT NULL then "ADRS"; otherwise "ADSL". | Derived | ||
| ADTTE | SRCVAR | Source Variable | character | If DEATHDT IS NOT NULL then "ADT"; If CENSORDT1 > STARTDT then "LSTALVDT"; otherwise "RANDDT". | Derived | ||
| ADTTE | SRCSEQ | Source Sequence Number | numeric | If DEATHDT IS NOT NULL then "DEATH_REC.ASEQ"; otherwise blank. | Derived |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/adam-adtte-os/full.md
Edit34 linesFollowing CDISC ADaM standards, use the provided ADSL and ADRS datasets to create an ADTTE dataset for overall survival, with one record per subject.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, STARTDT, ADT, AVAL, CNSR, EVNTDESC, CNSDTDSC, SRCDOM, SRCVAR, SRCSEQ
Use PARAMCD "OS" and PARAM "Overall Survival". STARTDT is the randomization date, the origin from which survival time is counted.
A qualifying death is a dated response record with code DEATH, result Y, and analysis flag Y; the earliest such date counts, with the lower sequence number breaking a tie on the same date.
ADT is the death date when it falls on or after STARTDT, and STARTDT itself when the death falls before it. With no death, ADT is the last-alive date when that date falls after randomization, else the randomization date. A death dated before randomization still counts as an event.
AVAL is the number of days from STARTDT to ADT, counting both days. CNSR is 0 for a death and 1 otherwise. EVNTDESC is Death for a death, Alive when censoring at the last-alive date, and Randomization when censoring at randomization. CNSDTDSC has no value for a death, and holds Alive During Study or Randomization for a censored record.
Trace ADT with SRCDOM ADRS, SRCVAR ADT, and the death record sequence for a death; with SRCDOM ADSL, SRCVAR LSTALVDT, and no sequence when censoring at the last-alive date; and with SRCDOM ADSL and SRCVAR RANDDT when censoring at randomization.
Read the source datasets from /app/input and save the completed dataset as /app/output/adtte.csv.
prompts/adam-adtte-os/conventions.md
Edit20 linesFollowing CDISC ADaM standards, use the provided ADSL and ADRS datasets to create an ADTTE dataset for overall survival, with one record per subject.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, STARTDT, ADT, AVAL, CNSR, EVNTDESC, CNSDTDSC, SRCDOM, SRCVAR, SRCSEQ
Use PARAMCD "OS" and PARAM "Overall Survival".
AVAL counts both STARTDT and ADT. CNSR is 0 for a death and 1 otherwise. EVNTDESC is Death, Alive, or Randomization. CNSDTDSC holds Alive During Study or Randomization, or has no value.
Trace ADT with SRCDOM, SRCVAR, and SRCSEQ, as one of: ADRS, ADT, and the death record sequence; ADSL, LSTALVDT, and no sequence; or ADSL and RANDDT.
Read the source datasets from /app/input and save the completed dataset as /app/output/adtte.csv.
prompts/adam-adtte-os/brief.md
Edit10 linesFollowing CDISC ADaM standards, use the provided ADSL and ADRS datasets to create an ADTTE dataset for overall survival, with one record per subject.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, STARTDT, ADT, AVAL, CNSR, EVNTDESC, CNSDTDSC, SRCDOM, SRCVAR, SRCSEQ
Read the source datasets from /app/input and save the completed dataset as /app/output/adtte.csv.
run.R
Edit4 lineslibrary(yamaa)adtte <- yamaa_domain("spec.yaml")$outputadtterun.py
Edit4 linesimport yamaaadtte = yamaa.yamaa_domain("spec.yaml").outputadttesolutions/adam-adtte-os/result.R
Edit70 lines# Reference solution for the yamaa benchmark adam-adtte-os (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)adsl <- read_csv( "/app/input/adsl.csv", col_types = cols(RANDDT = col_date(), LSTALVDT = col_date(), .default = col_character()))adrs <- read_csv( "/app/input/adrs.csv", col_types = cols(ASEQ = col_integer(), ADT = col_date(), .default = col_character()))# The earliest dated qualifying death: code DEATH, result Y, flag Y.# On a tied date the lower sequence wins.death <- adrs |> filter(PARAMCD == "DEATH", AVALC == "Y", ANL01FL == "Y", !is.na(ADT)) |> arrange(STUDYID, USUBJID, ADT, ASEQ) |> distinct(STUDYID, USUBJID, .keep_all = TRUE) |> select(STUDYID, USUBJID, DEATHDT = ADT, DEATHSEQ = ASEQ)adtte <- adsl |> left_join(death, by = c("STUDYID", "USUBJID")) |> mutate( STARTDT = RANDDT, CENSORDT1 = LSTALVDT, # A death before randomization still counts, moved up to it; with # no death, the last-alive date when after randomization, else # randomization itself. ADT = case_when( !is.na(DEATHDT) & !is.na(STARTDT) & DEATHDT >= STARTDT ~ DEATHDT, !is.na(DEATHDT) ~ STARTDT, !is.na(CENSORDT1) & !is.na(STARTDT) & CENSORDT1 > STARTDT ~ CENSORDT1, .default = STARTDT ), AVAL = as.integer(ADT - STARTDT) + 1L, CNSR = if_else(!is.na(DEATHDT), 0L, 1L), EVNTDESC = case_when( !is.na(DEATHDT) ~ "Death", !is.na(CENSORDT1) & !is.na(STARTDT) & CENSORDT1 > STARTDT ~ "Alive", .default = "Randomization" ), CNSDTDSC = case_when( !is.na(DEATHDT) ~ NA_character_, !is.na(CENSORDT1) & !is.na(STARTDT) & CENSORDT1 > STARTDT ~ "Alive During Study", .default = "Randomization" ), SRCDOM = if_else(!is.na(DEATHDT), "ADRS", "ADSL"), SRCVAR = case_when( !is.na(DEATHDT) ~ "ADT", !is.na(CENSORDT1) & !is.na(STARTDT) & CENSORDT1 > STARTDT ~ "LSTALVDT", .default = "RANDDT" ), SRCSEQ = if_else(!is.na(DEATHDT), DEATHSEQ, NA_integer_), PARAMCD = "OS", PARAM = "Overall Survival" ) |> select( STUDYID, USUBJID, PARAMCD, PARAM, STARTDT, ADT, AVAL, CNSR, EVNTDESC, CNSDTDSC, SRCDOM, SRCVAR, SRCSEQ ) |> arrange(USUBJID)dir.create("/app/output", showWarnings = FALSE)write_csv(adtte, "/app/output/adtte.csv", na = "")solutions/adam-adtte-os/result.py
Edit98 lines# Reference solution for the yamaa benchmark adam-adtte-os (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.from pathlib import Pathimport polars as pladsl = pl.read_csv("/app/input/adsl.csv", infer_schema=False, null_values="").with_columns( pl.col("RANDDT").str.to_date(strict=False), pl.col("LSTALVDT").str.to_date(strict=False),)adrs = pl.read_csv("/app/input/adrs.csv", infer_schema=False, null_values="").with_columns( pl.col("ASEQ").cast(pl.Int64, strict=False), pl.col("ADT").str.to_date(strict=False),)# The earliest dated qualifying death: code DEATH, result Y, flag Y.# On a tied date the lower sequence wins.death = ( adrs.filter( (pl.col("PARAMCD") == "DEATH") & (pl.col("AVALC") == "Y") & (pl.col("ANL01FL") == "Y") & pl.col("ADT").is_not_null() ) .sort(["STUDYID", "USUBJID", "ADT", "ASEQ"]) .unique(["STUDYID", "USUBJID"], keep="first", maintain_order=True) .select("STUDYID", "USUBJID", DEATHDT="ADT", DEATHSEQ="ASEQ"))adtte = ( adsl.join(death, on=["STUDYID", "USUBJID"], how="left", maintain_order="left") .with_columns( STARTDT=pl.col("RANDDT"), CENSORDT1=pl.col("LSTALVDT"), ) .with_columns( # A death before randomization still counts, moved up to it; with # no death, the last-alive date when after randomization, else # randomization itself. ADT=pl.when(pl.col("DEATHDT") >= pl.col("STARTDT")) .then(pl.col("DEATHDT")) .when(pl.col("DEATHDT").is_not_null()) .then(pl.col("STARTDT")) .when(pl.col("CENSORDT1") > pl.col("STARTDT")) .then(pl.col("CENSORDT1")) .otherwise(pl.col("STARTDT")), ) .with_columns( AVAL=(pl.col("ADT") - pl.col("STARTDT")).dt.total_days() + 1, CNSR=pl.when(pl.col("DEATHDT").is_not_null()).then(0).otherwise(1), EVNTDESC=pl.when(pl.col("DEATHDT").is_not_null()) .then(pl.lit("Death")) .when(pl.col("CENSORDT1") > pl.col("STARTDT")) .then(pl.lit("Alive")) .otherwise(pl.lit("Randomization")), CNSDTDSC=pl.when(pl.col("DEATHDT").is_not_null()) .then(None) .when(pl.col("CENSORDT1") > pl.col("STARTDT")) .then(pl.lit("Alive During Study")) .otherwise(pl.lit("Randomization")), SRCDOM=pl.when(pl.col("DEATHDT").is_not_null()) .then(pl.lit("ADRS")) .otherwise(pl.lit("ADSL")), SRCVAR=pl.when(pl.col("DEATHDT").is_not_null()) .then(pl.lit("ADT")) .when(pl.col("CENSORDT1") > pl.col("STARTDT")) .then(pl.lit("LSTALVDT")) .otherwise(pl.lit("RANDDT")), SRCSEQ=pl.when(pl.col("DEATHDT").is_not_null()) .then(pl.col("DEATHSEQ")) .otherwise(None), PARAMCD=pl.lit("OS"), PARAM=pl.lit("Overall Survival"), ) .select( "STUDYID", "USUBJID", "PARAMCD", "PARAM", "STARTDT", "ADT", "AVAL", "CNSR", "EVNTDESC", "CNSDTDSC", "SRCDOM", "SRCVAR", "SRCSEQ", ) .sort("USUBJID"))Path("/app/output").mkdir(parents=True, exist_ok=True)adtte.write_csv("/app/output/adtte.csv", null_value="")solutions/adam-adtte-os/result.sas
Edit35 lines/* opensas reference solution for the yamaa benchmark adam-adtte-os. Reads the benchmark inputs and derives the requested output. Execute with the existing opensas runtime: opensas /app/output/result.sas. */data adrs; length STUDYID USUBJID PARAMCD AVALC ANL01FL ADT $1024; infile "/app/input/adrs.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. PARAMCD : $char1024. AVALC : $char1024. ANL01FL : $char1024. ADT : $char1024. ASEQ;run;data adsl; length STUDYID USUBJID RANDDT LSTALVDT $1024; infile "/app/input/adsl.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. RANDDT : $char1024. LSTALVDT : $char1024.;run;proc sort data=adrs(where=(PARAMCD='DEATH' and AVALC='Y' and ANL01FL='Y' and ADT ne '')) out=events; by STUDYID USUBJID ADT ASEQ; run;data firstevent; set events; by STUDYID USUBJID; if first.USUBJID; rename ADT=DEATHDT ASEQ=DEATHSEQ; keep STUDYID USUBJID ADT ASEQ; run;proc sql; create table joined as select a.*,b.DEATHDT,b.DEATHSEQ from adsl as a left join firstevent as b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID; quit;data result; length PARAMCD $8 PARAM $40 STARTDT ADT $10 EVNTDESC CNSDTDSC $40 SRCDOM $4 SRCVAR $16; set joined; PARAMCD='OS'; PARAM='Overall Survival'; STARTDT=RANDDT; if not missing(DEATHDT) then do; ADT=DEATHDT; if ADT<STARTDT then ADT=STARTDT; CNSR=0; EVNTDESC='Death'; SRCDOM='ADRS'; SRCVAR='ADT'; SRCSEQ=DEATHSEQ; end; else if not missing(LSTALVDT) and LSTALVDT>RANDDT then do; ADT=LSTALVDT; CNSR=1; EVNTDESC='Alive'; CNSDTDSC='Alive During Study'; SRCDOM='ADSL'; SRCVAR='LSTALVDT'; end; else do; ADT=RANDDT; CNSR=1; EVNTDESC='Randomization'; CNSDTDSC='Randomization'; SRCDOM='ADSL'; SRCVAR='RANDDT'; end; if not missing(STARTDT) and not missing(ADT) then AVAL=input(ADT,yymmdd10.)-input(STARTDT,yymmdd10.)+1;run;proc sql; create table final as select STUDYID, USUBJID, PARAMCD, PARAM, STARTDT, ADT, AVAL, CNSR, EVNTDESC, CNSDTDSC, SRCDOM, SRCVAR, SRCSEQ from result;quit;proc export data=final outfile="/app/output/adtte.csv" dbms=csv replace; run;
Comments
Signed in with GitHub; saved in repository Discussions