As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.
We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.
Summary
README.mdPublic entries: Python and R. Passing coverage requires an installed inventory.
Goal: carry each prepared target-lesion assessment forward and
add NADIR, the running lowest sum for the same subject. All
records share the fixed parameter SDIAM (Sum of Target Lesion Diameters (mm)).
Input: one prepared assessment record per subject and visit,
carrying the visit number (AVISITN), assessment date (ADT),
measured sum (AVAL), measured-lesion count (NMEAS),
entry-selected count (NTARGET), and completeness flag
(ANL01FL).
Variables:
AVISITN: numeric order of the assessment within the subject, carried from the input.ADT: date of the assessment, carried from the input; blank when the date is unknown.AVAL: sum of the measured target-lesion diameters in millimeters, carried from the input; blank when the assessment measured no target lesion.NMEAS: number of target lesions the assessment measured, carried from the input; blank when the assessment has no lesion record.NTARGET: number of target lesions selected at study entry, carried from the input.ANL01FL: completeness of the assessment, carried from the input;Ywhen every target lesion was measured, blank otherwise.NADIR: lowestAVALamong complete assessments (ANL01FLisY) for the same subject dated on or before the currentADT, including the current assessment itself. Blank when the currentADTis blank, or when no complete assessment falls in the window.
Note: an incomplete current assessment keeps the nadir set by an earlier complete one, while a complete assessment with a blank date never counts toward any nadir, its own included.
Inputs
Under review1 source fileinput/adtr_raw.csv
Edit14 rows| STUDYID | USUBJID | AVISIT | AVISITN | ADT | PARAMCD | PARAM | AVAL | NMEAS | NTARGET | ANL01FL |
|---|---|---|---|---|---|---|---|---|---|---|
| YAMAA-01 | BASELINE | 0 | 2025-01-05 | SDIAM | Sum of Target Lesion Diameters (mm) | 80 | 3 | 3 | Y | |
| YAMAA-01 | WEEK 8 | 8 | 2025-03-02 | SDIAM | Sum of Target Lesion Diameters (mm) | 50 | 2 | 3 | — | |
| YAMAA-01 | WEEK 16 | 16 | 2025-04-27 | SDIAM | Sum of Target Lesion Diameters (mm) | — | — | 3 | — | |
| YAMAA-01 | BASELINE | 0 | 2025-01-12 | SDIAM | Sum of Target Lesion Diameters (mm) | 40 | 2 | 2 | Y | |
| YAMAA-01 | WEEK 8 | 8 | 2025-03-09 | SDIAM | Sum of Target Lesion Diameters (mm) | 22 | 2 | 2 | Y | |
| YAMAA-01 | BASELINE | 0 | 2025-01-19 | SDIAM | Sum of Target Lesion Diameters (mm) | 50 | 1 | 1 | Y | |
| YAMAA-01 | WEEK 8 | 8 | — | SDIAM | Sum of Target Lesion Diameters (mm) | 45 | 1 | 1 | Y | |
| YAMAA-01 | BASELINE | 0 | 2025-02-03 | SDIAM | Sum of Target Lesion Diameters (mm) | 64 | 2 | 2 | Y | |
| YAMAA-01 | BASELINE | 0 | 2025-01-08 | SDIAM | Sum of Target Lesion Diameters (mm) | 70 | 3 | 3 | Y | |
| YAMAA-01 | WEEK 8 | 8 | 2025-03-05 | SDIAM | Sum of Target Lesion Diameters (mm) | 45 | 3 | 3 | Y | |
| YAMAA-01 | WEEK 16 | 16 | 2025-04-30 | SDIAM | Sum of Target Lesion Diameters (mm) | 45 | 2 | 3 | Y | |
| YAMAA-01 | WEEK 24 | 24 | 2025-06-25 | SDIAM | Sum of Target Lesion Diameters (mm) | 58 | 2 | 3 | Y | |
| YAMAA-01 | BASELINE | 0 | 2025-02-10 | SDIAM | Sum of Target Lesion Diameters (mm) | 30 | 2 | 2 | Y | |
| YAMAA-01 | WEEK 8 | 8 | 2025-04-06 | SDIAM | Sum of Target Lesion Diameters (mm) | 0 | 1 | 2 | Y |
Expected output
Under review14 expected rowsexpected/adtr.csv
Edit14 rows| STUDYID | USUBJID | AVISIT | AVISITN | ADT | PARAMCD | PARAM | AVAL | NMEAS | NTARGET | ANL01FL | NADIR |
|---|---|---|---|---|---|---|---|---|---|---|---|
| YAMAA-01 | BASELINE | 0 | 2025-01-05 | SDIAM | Sum of Target Lesion Diameters (mm) | 80 | 3 | 3 | Y | 80 | |
| YAMAA-01 | WEEK 8 | 8 | 2025-03-02 | SDIAM | Sum of Target Lesion Diameters (mm) | 50 | 2 | 3 | — | 80 | |
| YAMAA-01 | WEEK 16 | 16 | 2025-04-27 | SDIAM | Sum of Target Lesion Diameters (mm) | — | — | 3 | — | 80 | |
| YAMAA-01 | BASELINE | 0 | 2025-01-12 | SDIAM | Sum of Target Lesion Diameters (mm) | 40 | 2 | 2 | Y | 40 | |
| YAMAA-01 | WEEK 8 | 8 | 2025-03-09 | SDIAM | Sum of Target Lesion Diameters (mm) | 22 | 2 | 2 | Y | 22 | |
| YAMAA-01 | BASELINE | 0 | 2025-01-19 | SDIAM | Sum of Target Lesion Diameters (mm) | 50 | 1 | 1 | Y | 50 | |
| YAMAA-01 | WEEK 8 | 8 | — | SDIAM | Sum of Target Lesion Diameters (mm) | 45 | 1 | 1 | Y | — | |
| YAMAA-01 | BASELINE | 0 | 2025-02-03 | SDIAM | Sum of Target Lesion Diameters (mm) | 64 | 2 | 2 | Y | 64 | |
| YAMAA-01 | BASELINE | 0 | 2025-01-08 | SDIAM | Sum of Target Lesion Diameters (mm) | 70 | 3 | 3 | Y | 70 | |
| YAMAA-01 | WEEK 8 | 8 | 2025-03-05 | SDIAM | Sum of Target Lesion Diameters (mm) | 45 | 3 | 3 | Y | 45 | |
| YAMAA-01 | WEEK 16 | 16 | 2025-04-30 | SDIAM | Sum of Target Lesion Diameters (mm) | 45 | 2 | 3 | Y | 45 | |
| YAMAA-01 | WEEK 24 | 24 | 2025-06-25 | SDIAM | Sum of Target Lesion Diameters (mm) | 58 | 2 | 3 | Y | 45 | |
| YAMAA-01 | BASELINE | 0 | 2025-02-10 | SDIAM | Sum of Target Lesion Diameters (mm) | 30 | 2 | 2 | Y | 30 | |
| YAMAA-01 | WEEK 8 | 8 | 2025-04-06 | SDIAM | Sum of Target Lesion Diameters (mm) | 0 | 1 | 2 | Y | 0 |
Human review and assessment
| Dataset Name | Variable Name | Variable Label | Type | Codelist/Controlled Terms | Core | Computational Method | Origin |
|---|---|---|---|---|---|---|---|
| ADTR | STUDYID | Study Identifier | character | Copy value from ADTR_RAW.STUDYID. | Predecessor | ||
| ADTR | USUBJID | Unique Subject Identifier | character | Copy value from ADTR_RAW.USUBJID. | Predecessor | ||
| ADTR | AVISIT | Analysis Visit | character | Copy value from ADTR_RAW.AVISIT. | Predecessor | ||
| ADTR | AVISITN | Analysis Visit (N) | numeric | Copy value from ADTR_RAW.AVISITN. | Predecessor | ||
| ADTR | ADT | Analysis Date | numeric | ISO 8601 | Copy value from ADTR_RAW.ADT. | Predecessor | |
| ADTR | PARAMCD | Parameter Code | character | Copy value from ADTR_RAW.PARAMCD. | Predecessor | ||
| ADTR | PARAM | Parameter | character | Copy value from ADTR_RAW.PARAM. | Predecessor | ||
| ADTR | AVAL | Analysis Value | numeric | Copy value from ADTR_RAW.AVAL. | Predecessor | ||
| ADTR | NMEAS | Number of Measured Target Lesions | numeric | Copy value from ADTR_RAW.NMEAS. | Predecessor | ||
| ADTR | NTARGET | Number of Target Lesions | numeric | Copy value from ADTR_RAW.NTARGET. | Predecessor | ||
| ADTR | ANL01FL | Analysis Record Flag 01 | character | Y | Copy value from ADTR_RAW.ANL01FL. | Predecessor | |
| ADTR | NADIR | Nadir Sum of Target Lesion Diameters | numeric | Aggregate MIN(ADTR_RAW.AVAL) over records grouped by ADTR_RAW.STUDYID, ADTR_RAW.USUBJID, matching on STUDYID and USUBJID, where ADT is ADT or more, where ADTR_RAW.ANL01FL = 'Y'. | Derived |
| Variable | Permitted values |
|---|---|
| ANL01FL | Y |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.7 | 2026-10-20 | yamaa docs build | Read a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default | ||
| 1.6 | 2026-10-17 | yamaa docs build | Describe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight | ||
| 1.5 | 2026-10-17 | yamaa docs build | Name collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part | ||
| 1.4 | 2026-10-15 | yamaa docs build | State datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks | ||
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/adam-adtr-nadir/full.md
Edit19 linesFollowing CDISC ADaM standards, use the provided ADTR_RAW dataset to create an ADTR dataset with one record per subject per visit.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, AVISIT, AVISITN, ADT, PARAMCD, PARAM, AVAL, NMEAS, NTARGET, ANL01FL, NADIR
Every record has PARAMCD "SDIAM" and PARAM "Sum of Target Lesion Diameters (mm)". NADIR is the lowest AVAL among the subject's complete assessments, those with ANL01FL "Y", dated on or before the record's ADT, including the record itself when it is complete. NADIR has no value when the record's ADT has no value, or when no complete assessment is dated on or before it. An incomplete record keeps the NADIR set by an earlier complete assessment, and its own AVAL does not count toward it. A complete assessment whose ADT has no value never counts toward any NADIR, its own included.
Read the source datasets from /app/input and save the completed dataset as /app/output/adtr.csv.
prompts/adam-adtr-nadir/conventions.md
Edit12 linesFollowing CDISC ADaM standards, use the provided ADTR_RAW dataset to create an ADTR dataset with one record per subject per visit.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, AVISIT, AVISITN, ADT, PARAMCD, PARAM, AVAL, NMEAS, NTARGET, ANL01FL, NADIR
Every record has PARAMCD "SDIAM" and PARAM "Sum of Target Lesion Diameters (mm)". ANL01FL is "Y".
Read the source datasets from /app/input and save the completed dataset as /app/output/adtr.csv.
prompts/adam-adtr-nadir/brief.md
Edit9 linesFollowing CDISC ADaM standards, use the provided ADTR_RAW dataset to create an ADTR dataset with one record per subject per visit.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, AVISIT, AVISITN, ADT, PARAMCD, PARAM, AVAL, NMEAS, NTARGET, ANL01FL, NADIR
Read the source datasets from /app/input and save the completed dataset as /app/output/adtr.csv.
R
Edit3 lineslibrary(yamaa)yamaa_spec("spec.yaml")$run()$outputPython
Edit3 linesimport yamaayamaa.spec("spec.yaml").run().outputsolutions/adam-adtr-nadir/result.R
Edit44 lines# Reference solution for the yamaa benchmark adam-adtr-nadir (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)raw <- read_csv( "/app/input/adtr_raw.csv", col_types = cols( AVISITN = col_integer(), ADT = col_date(), AVAL = col_double(), NMEAS = col_integer(), NTARGET = col_integer(), .default = col_character() ))# Complete dated assessments carry the nadir candidates; a complete# assessment with no date never counts toward any nadir, its own# included.complete <- raw |> filter(ANL01FL == "Y", !is.na(ADT), !is.na(AVAL)) |> select(USUBJID, ADT, AVAL)# An incomplete current assessment keeps the nadir set by an earlier# complete one, while a current record with no date has none.adtr <- raw |> rowwise() |> mutate( NADIR = { cands <- complete$AVAL[complete$USUBJID == USUBJID & complete$ADT <= ADT] if (is.na(ADT) || length(cands) == 0L) NA_real_ else min(cands) } ) |> ungroup() |> select( STUDYID, USUBJID, AVISIT, AVISITN, ADT, PARAMCD, PARAM, AVAL, NMEAS, NTARGET, ANL01FL, NADIR ) |> arrange(USUBJID, AVISITN)dir.create("/app/output", showWarnings = FALSE)write_csv(adtr, "/app/output/adtr.csv", na = "")solutions/adam-adtr-nadir/result.py
Edit60 lines# Reference solution for the yamaa benchmark adam-adtr-nadir (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.import csvfrom datetime import dateINPUT = "/app/input/adtr_raw.csv"OUTPUT = "/app/output/adtr.csv"COLUMNS = [ "STUDYID", "USUBJID", "AVISIT", "AVISITN", "ADT", "PARAMCD", "PARAM", "AVAL", "NMEAS", "NTARGET", "ANL01FL", "NADIR",]def parse_date(text): text = (text or "").strip() if not text: return None return date.fromisoformat(text)def fmt_num(value): if value is None: return "" if float(value).is_integer(): return str(int(value)) return str(value)with open(INPUT, newline="") as f: rows = list(csv.DictReader(f))# Complete assessments per subject: ANL01FL "Y" with a dated ADT.complete = {}for row in rows: if (row.get("ANL01FL") or "").strip() == "Y": adt = parse_date(row.get("ADT")) aval_text = (row.get("AVAL") or "").strip() if adt is not None and aval_text: complete.setdefault(row["USUBJID"], []).append((adt, float(aval_text)))for row in rows: adt = parse_date(row.get("ADT")) nadir = None if adt is not None: candidates = [aval for d, aval in complete.get(row["USUBJID"], []) if d <= adt] if candidates: nadir = min(candidates) row["NADIR"] = fmt_num(nadir)with open(OUTPUT, "w", newline="") as f: writer = csv.DictWriter(f, fieldnames=COLUMNS) writer.writeheader() for row in rows: writer.writerow({c: row.get(c, "") for c in COLUMNS})solutions/adam-adtr-nadir/result.sas
Edit18 lines/* opensas reference solution for the yamaa benchmark adam-adtr-nadir. Reads the benchmark inputs and derives the requested output. Execute with the existing opensas runtime: opensas /app/output/result.sas. */data adtr_raw; length STUDYID USUBJID AVISIT ADT PARAMCD PARAM ANL01FL $1024; infile "/app/input/adtr_raw.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. AVISIT : $char1024. AVISITN ADT : $char1024. PARAMCD : $char1024. PARAM : $char1024. AVAL NMEAS NTARGET ANL01FL : $char1024.;run;data history; set adtr_raw; PRIORDATE=ADT; PRIORVALUE=AVAL; PRIORFLAG=ANL01FL; keep STUDYID USUBJID PRIORDATE PRIORVALUE PRIORFLAG; run;proc sql;create table nadirs as select a.STUDYID,a.USUBJID,a.AVISIT,min(b.PRIORVALUE) as NADIR from adtr_raw a left join history b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID and not missing(a.ADT) and not missing(b.PRIORDATE) and b.PRIORDATE<=a.ADT and b.PRIORFLAG='Y' group by a.STUDYID,a.USUBJID,a.AVISIT;create table result as select a.*,b.NADIR from adtr_raw a left join nadirs b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID and a.AVISIT=b.AVISIT; quit;proc sql; create table final as select STUDYID, USUBJID, AVISIT, AVISITN, ADT, PARAMCD, PARAM, AVAL, NMEAS, NTARGET, ANL01FL, NADIR from result;quit;proc export data=final outfile="/app/output/adtr.csv" dbms=csv replace; run;
yamaa