yamaa Review ask
As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.
We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.
Summary
EditUnder reviewWhat this benchmark meansREADME.mdPublic entries: Python and R. Passing coverage requires an installed inventory.
Goal: derive one overall-survival record per subject with
parameter code OS (Overall Survival), carrying STARTDT,
ADT, AVAL, CNSR, EVNTDESC, CNSDTDSC, SRCDOM,
SRCVAR, and SRCSEQ.
Input: subject-level randomization and last-known-alive dates
(RANDDT, LSTALVDT) alongside response records holding death
analysis dates (ADT). The death selection keeps only dated
response records where PARAMCD is DEATH, AVALC is Y, and
ANL01FL is Y; it takes the one with the earliest ADT (on a
tied date, the lower sequence number wins) and treats a subject
with no such record as having no death.
Variables:
STARTDTis the randomization date (RANDDT), the origin from which survival time is counted.ADTis the analysis date: the death date when a qualifying death falls on or afterSTARTDT, andSTARTDTitself when the death falls before it. With no death,ADTis the last-alive date when that date falls after randomization, else the randomization date (including a tie).AVALis the number of days fromSTARTDTthroughADT, counting both endpoints.CNSRis0when the subject has a qualifying death and1otherwise.EVNTDESCisDeathwhen the subject has a qualifying death,Alivewhen censoring at a last-alive date after randomization, andRandomizationotherwise.CNSDTDSCis blank for a death,Alive During Studywhen censoring at the last-alive date, andRandomizationwhen censoring at randomization.SRCDOMisADRSfor a death andADSLfor censoring.SRCVARisADTfor a death,LSTALVDTwhen censoring at the last-alive date, andRANDDTotherwise.SRCSEQis the sequence number from the selected death record, and blank when the record is censored.
Note: a death dated before randomization still counts as an
event: ADT is set to randomization while CNSR, EVNTDESC,
and the source columns still reflect the death.
Inputs
Under review2 source filesinput/adrs.csv
Edit10 rows| STUDYID | USUBJID | PARAMCD | AVALC | ANL01FL | ADT | ASEQ |
|---|---|---|---|---|---|---|
| YAMAA-01 | DEATH | Y | Y | 2026-06-01 | 1 | |
| YAMAA-01 | DEATH | N | Y | 2026-06-01 | 1 | |
| YAMAA-01 | DEATH | Y | Y | 2026-04-01 | 2 | |
| YAMAA-01 | DEATH | Y | Y | 2026-02-01 | 1 | |
| YAMAA-01 | DEATH | Y | Y | 2026-07-01 | 3 | |
| YAMAA-01 | DEATH | Y | Y | 2026-07-01 | 2 | |
| YAMAA-01 | BOR | Y | Y | 2026-02-01 | 3 | |
| YAMAA-01 | DEATH | Y | N | 2026-03-01 | 1 | |
| YAMAA-01 | DEATH | Y | Y | 2026-05-01 | 2 | |
| YAMAA-01 | DEATH | Y | Y | — | 1 |
input/adsl.csv
Edit10 rows| STUDYID | USUBJID | RANDDT | LSTALVDT |
|---|---|---|---|
| YAMAA-01 | 2026-01-01 | — | |
| YAMAA-01 | 2026-01-01 | 2026-09-01 | |
| YAMAA-01 | 2026-01-01 | — | |
| YAMAA-01 | 2026-03-01 | 2026-02-01 | |
| YAMAA-01 | 2026-04-01 | 2026-04-01 | |
| YAMAA-01 | 2026-05-01 | — | |
| YAMAA-01 | 2026-02-01 | — | |
| YAMAA-01 | 2026-01-01 | 2026-12-01 | |
| YAMAA-01 | 2026-01-01 | — | |
| YAMAA-01 | 2026-01-01 | 2026-12-01 |
Expected output
Under review10 expected rowsexpected/adtte.csv
Edit10 rows| STUDYID | USUBJID | PARAMCD | PARAM | STARTDT | ADT | AVAL | CNSR | EVNTDESC | CNSDTDSC | SRCDOM | SRCVAR | SRCSEQ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| YAMAA-01 | OS | Overall Survival | 2026-01-01 | 2026-06-01 | 152 | 0 | Death | — | ADRS | ADT | 1 | |
| YAMAA-01 | OS | Overall Survival | 2026-01-01 | 2026-09-01 | 244 | 1 | Alive | Alive During Study | ADSL | LSTALVDT | — | |
| YAMAA-01 | OS | Overall Survival | 2026-01-01 | 2026-01-01 | 1 | 1 | Randomization | Randomization | ADSL | RANDDT | — | |
| YAMAA-01 | OS | Overall Survival | 2026-03-01 | 2026-03-01 | 1 | 1 | Randomization | Randomization | ADSL | RANDDT | — | |
| YAMAA-01 | OS | Overall Survival | 2026-04-01 | 2026-04-01 | 1 | 1 | Randomization | Randomization | ADSL | RANDDT | — | |
| YAMAA-01 | OS | Overall Survival | 2026-05-01 | 2026-05-01 | 1 | 0 | Death | — | ADRS | ADT | 2 | |
| YAMAA-01 | OS | Overall Survival | 2026-02-01 | 2026-02-01 | 1 | 0 | Death | — | ADRS | ADT | 1 | |
| YAMAA-01 | OS | Overall Survival | 2026-01-01 | 2026-07-01 | 182 | 0 | Death | — | ADRS | ADT | 2 | |
| YAMAA-01 | OS | Overall Survival | 2026-01-01 | 2026-05-01 | 121 | 0 | Death | — | ADRS | ADT | 2 | |
| YAMAA-01 | OS | Overall Survival | 2026-01-01 | 2026-12-01 | 335 | 1 | Alive | Alive During Study | ADSL | LSTALVDT | — |
Shaded columns are derived— = missing value
Human review and assessment
Not under review
| Dataset Name | Variable Name | Variable Label | Type | Codelist/Controlled Terms | Core | Computational Method | Origin |
|---|---|---|---|---|---|---|---|
| ADTTE | STUDYID | Study Identifier | character | Copy value from ADSL.STUDYID. | Predecessor | ||
| ADTTE | USUBJID | Unique Subject Identifier | character | Copy value from ADSL.USUBJID. | Predecessor | ||
| ADTTE | PARAMCD | Parameter Code | character | Set constant value "OS". | Assigned | ||
| ADTTE | PARAM | Parameter | character | Set constant value "Overall Survival". | Assigned | ||
| ADTTE | STARTDT | Time-to-Event Origin Date | numeric | ISO 8601 | Copy value from ADSL.RANDDT. | Predecessor | |
| ADTTE | ADT | Analysis Date | numeric | ISO 8601 | If DEATHDT >= STARTDT then "DEATHDT"; If DEATHDT IS NOT NULL then "STARTDT"; If CENSORDT1 > STARTDT then "CENSORDT1"; otherwise "STARTDT". | Derived | |
| ADTTE | AVAL | Analysis Value | numeric | Whole days from STARTDT to ADT, counting both endpoints. | Derived | ||
| ADTTE | CNSR | Censor | numeric | If DEATHDT IS NOT NULL then "0"; otherwise "1". | Derived | ||
| ADTTE | EVNTDESC | Event or Censoring Description | character | If DEATHDT IS NOT NULL then "Death"; If CENSORDT1 > STARTDT then "Alive"; otherwise "Randomization". | Derived | ||
| ADTTE | CNSDTDSC | Censoring Date Description | character | If DEATHDT IS NULL AND CENSORDT1 > STARTDT then "Alive During Study"; If DEATHDT IS NULL then "Randomization"; otherwise blank. | Derived | ||
| ADTTE | SRCDOM | Source Data | character | If DEATHDT IS NOT NULL then "ADRS"; otherwise "ADSL". | Derived | ||
| ADTTE | SRCVAR | Source Variable | character | If DEATHDT IS NOT NULL then "ADT"; If CENSORDT1 > STARTDT then "LSTALVDT"; otherwise "RANDDT". | Derived | ||
| ADTTE | SRCSEQ | Source Sequence Number | numeric | If DEATHDT IS NOT NULL then "DEATH_REC.ASEQ"; otherwise blank. | Derived |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.7 | 2026-10-20 | yamaa docs build | Read a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default | ||
| 1.6 | 2026-10-17 | yamaa docs build | Describe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight | ||
| 1.5 | 2026-10-17 | yamaa docs build | Name collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part | ||
| 1.4 | 2026-10-15 | yamaa docs build | State datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks | ||
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/adam-adtte-os/full.md
Edit38 linesFollowing CDISC ADaM standards, use the provided ADRS and ADSL datasets to create an ADTTE dataset with one record per subject.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, STARTDT, ADT, AVAL, CNSR, EVNTDESC, CNSDTDSC, SRCDOM, SRCVAR, SRCSEQ
- PARAMCD is "OS" and PARAM is "Overall Survival" on every record.
- STARTDT is the randomization date (RANDDT), the origin from which survival time is counted.
- A qualifying death record is an ADRS record with PARAMCD "DEATH", AVALC "Y", ANL01FL "Y", and a recorded ADT. The subject's death is the qualifying record with the earliest ADT, and on a tied date the one with the lower ASEQ. A subject with no qualifying record has no death.
- ADT is the death date when a qualifying death falls on or after STARTDT, and STARTDT itself when the death falls before it; a death dated before randomization still counts as the event. With no death, ADT is the last-alive date (LSTALVDT) when that date falls after STARTDT, and STARTDT itself otherwise, including on a tie and when no last-alive date is recorded.
- AVAL is the number of days from STARTDT through ADT, counting both endpoints.
- CNSR is 0 when the subject has a qualifying death and 1 otherwise.
- EVNTDESC is "Death" when the subject has a qualifying death, "Alive" when the subject is censored at a last-alive date after randomization, and "Randomization" otherwise.
- CNSDTDSC has no value for a death, is "Alive During Study" when the subject is censored at the last-alive date, and "Randomization" when the subject is censored at randomization.
- SRCDOM is "ADRS" for a death and "ADSL" for censoring.
- SRCVAR is "ADT" for a death, "LSTALVDT" when the subject is censored at the last-alive date, and "RANDDT" otherwise.
- SRCSEQ is the ASEQ from the selected death record, and has no value when the record is censored.
Read the source datasets from /app/input and save the completed dataset as /app/output/adtte.csv.
prompts/adam-adtte-os/conventions.md
Edit16 linesFollowing CDISC ADaM standards, use the provided ADRS and ADSL datasets to create an ADTTE dataset with one record per subject.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, STARTDT, ADT, AVAL, CNSR, EVNTDESC, CNSDTDSC, SRCDOM, SRCVAR, SRCSEQ
PARAMCD is "OS" and PARAM is "Overall Survival" on every record. AVAL is the number of days from STARTDT through ADT, counting both endpoints. CNSR is 0 for a qualifying death and 1 otherwise. EVNTDESC is "Death", "Alive", or "Randomization". CNSDTDSC is "Alive During Study" or "Randomization", or has no value. SRCDOM is "ADRS" or "ADSL". SRCVAR is "ADT", "LSTALVDT", or "RANDDT".
Read the source datasets from /app/input and save the completed dataset as /app/output/adtte.csv.
prompts/adam-adtte-os/brief.md
Edit9 linesFollowing CDISC ADaM standards, use the provided ADRS and ADSL datasets to create an ADTTE dataset with one record per subject.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, STARTDT, ADT, AVAL, CNSR, EVNTDESC, CNSDTDSC, SRCDOM, SRCVAR, SRCSEQ
Read the source datasets from /app/input and save the completed dataset as /app/output/adtte.csv.
R
Edit3 lineslibrary(yamaa)yamaa_spec("spec.yaml")$run()$outputPython
Edit3 linesimport yamaayamaa.spec("spec.yaml").run().outputsolutions/adam-adtte-os/result.R
Edit70 lines# Reference solution for the yamaa benchmark adam-adtte-os (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)adsl <- read_csv( "/app/input/adsl.csv", col_types = cols(RANDDT = col_date(), LSTALVDT = col_date(), .default = col_character()))adrs <- read_csv( "/app/input/adrs.csv", col_types = cols(ASEQ = col_integer(), ADT = col_date(), .default = col_character()))# The earliest dated qualifying death: code DEATH, result Y, flag Y.# On a tied date the lower sequence wins.death <- adrs |> filter(PARAMCD == "DEATH", AVALC == "Y", ANL01FL == "Y", !is.na(ADT)) |> arrange(STUDYID, USUBJID, ADT, ASEQ) |> distinct(STUDYID, USUBJID, .keep_all = TRUE) |> select(STUDYID, USUBJID, DEATHDT = ADT, DEATHSEQ = ASEQ)adtte <- adsl |> left_join(death, by = c("STUDYID", "USUBJID")) |> mutate( STARTDT = RANDDT, CENSORDT1 = LSTALVDT, # A death before randomization still counts, moved up to it; with # no death, the last-alive date when after randomization, else # randomization itself. ADT = case_when( !is.na(DEATHDT) & !is.na(STARTDT) & DEATHDT >= STARTDT ~ DEATHDT, !is.na(DEATHDT) ~ STARTDT, !is.na(CENSORDT1) & !is.na(STARTDT) & CENSORDT1 > STARTDT ~ CENSORDT1, .default = STARTDT ), AVAL = as.integer(ADT - STARTDT) + 1L, CNSR = if_else(!is.na(DEATHDT), 0L, 1L), EVNTDESC = case_when( !is.na(DEATHDT) ~ "Death", !is.na(CENSORDT1) & !is.na(STARTDT) & CENSORDT1 > STARTDT ~ "Alive", .default = "Randomization" ), CNSDTDSC = case_when( !is.na(DEATHDT) ~ NA_character_, !is.na(CENSORDT1) & !is.na(STARTDT) & CENSORDT1 > STARTDT ~ "Alive During Study", .default = "Randomization" ), SRCDOM = if_else(!is.na(DEATHDT), "ADRS", "ADSL"), SRCVAR = case_when( !is.na(DEATHDT) ~ "ADT", !is.na(CENSORDT1) & !is.na(STARTDT) & CENSORDT1 > STARTDT ~ "LSTALVDT", .default = "RANDDT" ), SRCSEQ = if_else(!is.na(DEATHDT), DEATHSEQ, NA_integer_), PARAMCD = "OS", PARAM = "Overall Survival" ) |> select( STUDYID, USUBJID, PARAMCD, PARAM, STARTDT, ADT, AVAL, CNSR, EVNTDESC, CNSDTDSC, SRCDOM, SRCVAR, SRCSEQ ) |> arrange(USUBJID)dir.create("/app/output", showWarnings = FALSE)write_csv(adtte, "/app/output/adtte.csv", na = "")solutions/adam-adtte-os/result.py
Edit128 lines# Reference solution for the yamaa benchmark adam-adtte-os (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1."""Create the ADTTE overall-survival dataset from ADRS and ADSL.One record per subject. A qualifying death is an ADRS record withPARAMCD "DEATH", AVALC "Y", ANL01FL "Y" and a recorded ADT; theearliest such date (lowest ASEQ on a tie) is the event. Subjectswithout a qualifying death are censored at their last-alive date whenit falls after randomization, and at randomization otherwise."""import csvfrom datetime import dateINPUT_DIR = "/app/input"OUTPUT_DIR = "/app/output"def parse_date(text): text = (text or "").strip() if not text: return None return date.fromisoformat(text)def read_rows(path): with open(path, newline="") as handle: return list(csv.DictReader(handle))def main(): adsl = read_rows(f"{INPUT_DIR}/adsl.csv") adrs = read_rows(f"{INPUT_DIR}/adrs.csv") # Select each subject's qualifying death record: earliest ADT, # lowest ASEQ on a tied date. deaths = {} for row in adrs: if row["PARAMCD"] != "DEATH": continue if row["AVALC"] != "Y" or row["ANL01FL"] != "Y": continue death_date = parse_date(row["ADT"]) if death_date is None: continue key = (death_date, int(row["ASEQ"])) subject = row["USUBJID"] if subject not in deaths or key < deaths[subject][0]: deaths[subject] = (key, row) records = [] for subject_row in adsl: studyid = subject_row["STUDYID"] usubjid = subject_row["USUBJID"] startdt = parse_date(subject_row["RANDDT"]) lstalvdt = parse_date(subject_row["LSTALVDT"]) if usubjid in deaths: _, death_row = deaths[usubjid] death_date = parse_date(death_row["ADT"]) adt = death_date if death_date >= startdt else startdt cnsr = "0" evntdesc = "Death" cnsdtdsc = "" srcdom = "ADRS" srcvar = "ADT" srcseq = death_row["ASEQ"] else: at_last_alive = lstalvdt is not None and lstalvdt > startdt adt = lstalvdt if at_last_alive else startdt cnsr = "1" if at_last_alive: evntdesc = "Alive" cnsdtdsc = "Alive During Study" srcvar = "LSTALVDT" else: evntdesc = "Randomization" cnsdtdsc = "Randomization" srcvar = "RANDDT" srcdom = "ADSL" srcseq = "" aval = (adt - startdt).days + 1 records.append( { "STUDYID": studyid, "USUBJID": usubjid, "PARAMCD": "OS", "PARAM": "Overall Survival", "STARTDT": startdt.isoformat(), "ADT": adt.isoformat(), "AVAL": str(aval), "CNSR": cnsr, "EVNTDESC": evntdesc, "CNSDTDSC": cnsdtdsc, "SRCDOM": srcdom, "SRCVAR": srcvar, "SRCSEQ": srcseq, } ) columns = [ "STUDYID", "USUBJID", "PARAMCD", "PARAM", "STARTDT", "ADT", "AVAL", "CNSR", "EVNTDESC", "CNSDTDSC", "SRCDOM", "SRCVAR", "SRCSEQ", ] with open(f"{OUTPUT_DIR}/adtte.csv", "w", newline="") as handle: writer = csv.DictWriter(handle, fieldnames=columns) writer.writeheader() writer.writerows(records)if __name__ == "__main__": main()solutions/adam-adtte-os/result.sas
Edit35 lines/* opensas reference solution for the yamaa benchmark adam-adtte-os. Reads the benchmark inputs and derives the requested output. Execute with the existing opensas runtime: opensas /app/output/result.sas. */data adrs; length STUDYID USUBJID PARAMCD AVALC ANL01FL ADT $1024; infile "/app/input/adrs.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. PARAMCD : $char1024. AVALC : $char1024. ANL01FL : $char1024. ADT : $char1024. ASEQ;run;data adsl; length STUDYID USUBJID RANDDT LSTALVDT $1024; infile "/app/input/adsl.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. RANDDT : $char1024. LSTALVDT : $char1024.;run;proc sort data=adrs(where=(PARAMCD='DEATH' and AVALC='Y' and ANL01FL='Y' and ADT ne '')) out=events; by STUDYID USUBJID ADT ASEQ; run;data firstevent; set events; by STUDYID USUBJID; if first.USUBJID; rename ADT=DEATHDT ASEQ=DEATHSEQ; keep STUDYID USUBJID ADT ASEQ; run;proc sql; create table joined as select a.*,b.DEATHDT,b.DEATHSEQ from adsl as a left join firstevent as b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID; quit;data result; length PARAMCD $8 PARAM $40 STARTDT ADT $10 EVNTDESC CNSDTDSC $40 SRCDOM $4 SRCVAR $16; set joined; PARAMCD='OS'; PARAM='Overall Survival'; STARTDT=RANDDT; if not missing(DEATHDT) then do; ADT=DEATHDT; if ADT<STARTDT then ADT=STARTDT; CNSR=0; EVNTDESC='Death'; SRCDOM='ADRS'; SRCVAR='ADT'; SRCSEQ=DEATHSEQ; end; else if not missing(LSTALVDT) and LSTALVDT>RANDDT then do; ADT=LSTALVDT; CNSR=1; EVNTDESC='Alive'; CNSDTDSC='Alive During Study'; SRCDOM='ADSL'; SRCVAR='LSTALVDT'; end; else do; ADT=RANDDT; CNSR=1; EVNTDESC='Randomization'; CNSDTDSC='Randomization'; SRCDOM='ADSL'; SRCVAR='RANDDT'; end; if not missing(STARTDT) and not missing(ADT) then AVAL=input(ADT,yymmdd10.)-input(STARTDT,yymmdd10.)+1;run;proc sql; create table final as select STUDYID, USUBJID, PARAMCD, PARAM, STARTDT, ADT, AVAL, CNSR, EVNTDESC, CNSDTDSC, SRCDOM, SRCVAR, SRCSEQ from result;quit;proc export data=final outfile="/app/output/adtte.csv" dbms=csv replace; run;