Review ask

As a clinical trial statistician or programmer, challenge anything in this benchmark: describe a problem in Comments, or propose a fix with the Edit button on any source file.

  1. Task: Would a real study need this derivation, specified this way? Does it follow CDISC ADaM: variable names, controlled terms, record structure? Flag any rule that is ambiguous or that a sponsor would handle differently.
  2. Inputs: Do they look like data collected in a real trial, and do they include the situations you would expect to meet in one?
  3. Expected output: Would independent QC programming reproduce every value from these inputs and rules?

The other tabs beside Comments are background for assessment, not under review.

Summary

Lifecycle: reviewedEditUnder review
What this benchmark means
README.md

Goal: carry each prepared target-lesion assessment forward and add NADIR, the running lowest sum for the same subject. All records share the fixed parameter SDIAM (Sum of Target Lesion Diameters (mm)).

Input: one prepared assessment record per subject and visit, carrying the visit number (AVISITN), assessment date (ADT), measured sum (AVAL), measured-lesion count (NMEAS), entry-selected count (NTARGET), and completeness flag (ANL01FL).

Variables:

  • AVISITN: numeric order of the assessment within the subject, carried from the input.
  • ADT: date of the assessment, carried from the input; blank when the date is unknown.
  • AVAL: sum of the measured target-lesion diameters in millimeters, carried from the input; blank when the assessment measured no target lesion.
  • NMEAS: number of target lesions the assessment measured, carried from the input; blank when the assessment has no lesion record.
  • NTARGET: number of target lesions selected at study entry, carried from the input.
  • ANL01FL: completeness of the assessment, carried from the input; Y when every target lesion was measured, blank otherwise.
  • NADIR: lowest AVAL among complete assessments (ANL01FL is Y) for the same subject dated on or before the current ADT, including the current assessment itself. Blank when the current ADT is blank, or when no complete assessment falls in the window.

Note: an incomplete current assessment keeps the nadir set by an earlier complete one, while a complete assessment with a blank date never counts toward any nadir, its own included.

Inputs

Under review
1 source file

input/adtr_raw.csv

Edit
14 rows
input/adtr_raw.csv
STUDYIDUSUBJIDAVISITAVISITNADTPARAMCDPARAMAVALNMEASNTARGETANL01FL
YAMAA-01BASELINE02025-01-05SDIAMSum of Target Lesion Diameters (mm)8033Y
YAMAA-01WEEK 882025-03-02SDIAMSum of Target Lesion Diameters (mm)5023—
YAMAA-01WEEK 16162025-04-27SDIAMSum of Target Lesion Diameters (mm)——3—
YAMAA-01BASELINE02025-01-12SDIAMSum of Target Lesion Diameters (mm)4022Y
YAMAA-01WEEK 882025-03-09SDIAMSum of Target Lesion Diameters (mm)2222Y
YAMAA-01BASELINE02025-01-19SDIAMSum of Target Lesion Diameters (mm)5011Y
YAMAA-01WEEK 88—SDIAMSum of Target Lesion Diameters (mm)4511Y
YAMAA-01BASELINE02025-02-03SDIAMSum of Target Lesion Diameters (mm)6422Y
YAMAA-01BASELINE02025-01-08SDIAMSum of Target Lesion Diameters (mm)7033Y
YAMAA-01WEEK 882025-03-05SDIAMSum of Target Lesion Diameters (mm)4533Y
YAMAA-01WEEK 16162025-04-30SDIAMSum of Target Lesion Diameters (mm)4523Y
YAMAA-01WEEK 24242025-06-25SDIAMSum of Target Lesion Diameters (mm)5823Y
YAMAA-01BASELINE02025-02-10SDIAMSum of Target Lesion Diameters (mm)3022Y
YAMAA-01WEEK 882025-04-06SDIAMSum of Target Lesion Diameters (mm)012Y

Expected output

Under review
14 expected rows

expected/adtr.csv

Edit
14 rows
expected/adtr.csv
STUDYIDUSUBJIDAVISITAVISITNADTPARAMCDPARAMAVALNMEASNTARGETANL01FLNADIR
YAMAA-01BASELINE02025-01-05SDIAMSum of Target Lesion Diameters (mm)8033Y80
YAMAA-01WEEK 882025-03-02SDIAMSum of Target Lesion Diameters (mm)5023—80
YAMAA-01WEEK 16162025-04-27SDIAMSum of Target Lesion Diameters (mm)——3—80
YAMAA-01BASELINE02025-01-12SDIAMSum of Target Lesion Diameters (mm)4022Y40
YAMAA-01WEEK 882025-03-09SDIAMSum of Target Lesion Diameters (mm)2222Y22
YAMAA-01BASELINE02025-01-19SDIAMSum of Target Lesion Diameters (mm)5011Y50
YAMAA-01WEEK 88—SDIAMSum of Target Lesion Diameters (mm)4511Y—
YAMAA-01BASELINE02025-02-03SDIAMSum of Target Lesion Diameters (mm)6422Y64
YAMAA-01BASELINE02025-01-08SDIAMSum of Target Lesion Diameters (mm)7033Y70
YAMAA-01WEEK 882025-03-05SDIAMSum of Target Lesion Diameters (mm)4533Y45
YAMAA-01WEEK 16162025-04-30SDIAMSum of Target Lesion Diameters (mm)4523Y45
YAMAA-01WEEK 24242025-06-25SDIAMSum of Target Lesion Diameters (mm)5823Y45
YAMAA-01BASELINE02025-02-10SDIAMSum of Target Lesion Diameters (mm)3022Y30
YAMAA-01WEEK 882025-04-06SDIAMSum of Target Lesion Diameters (mm)012Y0
Shaded columns are derived— = missing value

Comments and assessment materials

Comments

Signed in with GitHub; saved in repository Discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Dataset NameVariable NameVariable LabelTypeCodelist/Controlled TermsCoreComputational MethodOrigin
ADTRSTUDYIDStudy IdentifiercharacterCopy value from ADTR_RAW.STUDYID.Predecessor
ADTRUSUBJIDUnique Subject IdentifiercharacterCopy value from ADTR_RAW.USUBJID.Predecessor
ADTRAVISITAnalysis VisitcharacterCopy value from ADTR_RAW.AVISIT.Predecessor
ADTRAVISITNAnalysis Visit (N)numericCopy value from ADTR_RAW.AVISITN.Predecessor
ADTRADTAnalysis DatenumericISO 8601Copy value from ADTR_RAW.ADT.Predecessor
ADTRPARAMCDParameter CodecharacterCopy value from ADTR_RAW.PARAMCD.Predecessor
ADTRPARAMParametercharacterCopy value from ADTR_RAW.PARAM.Predecessor
ADTRAVALAnalysis ValuenumericCopy value from ADTR_RAW.AVAL.Predecessor
ADTRNMEASNumber of Measured Target LesionsnumericCopy value from ADTR_RAW.NMEAS.Predecessor
ADTRNTARGETNumber of Target LesionsnumericCopy value from ADTR_RAW.NTARGET.Predecessor
ADTRANL01FLAnalysis Record Flag 01characterYCopy value from ADTR_RAW.ANL01FL.Predecessor
ADTRNADIRNadir Sum of Target Lesion DiametersnumericAggregate MIN(ADTR_RAW.AVAL) over records grouped by ADTR_RAW.STUDYID, ADTR_RAW.USUBJID, matching on STUDYID and USUBJID, where ADT is ADT or more, where ADTR_RAW.ANL01FL = 'Y'.Derived
VariablePermitted values
ANL01FLY
VersionDateAuthorDescriptionReviewerSign-off
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/adam-adtr-nadir/full.md

Edit
20 lines

Following CDISC ADaM standards, use the provided ADTR_RAW dataset to create an ADTR dataset with one record per subject per visit.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, AVISIT, AVISITN, ADT, PARAMCD, PARAM, AVAL, NMEAS, NTARGET, ANL01FL, NADIR

Use PARAMCD "SDIAM" and PARAM "Sum of Target Lesion Diameters (mm)" on every record.

NADIR is the lowest AVAL among complete assessments (ANL01FL is Y) for the same subject dated on or before the current ADT, including the current assessment itself. It has no value when the current ADT has none, or when no complete assessment falls in the window, so an incomplete current assessment keeps the nadir set by an earlier complete one, while a complete assessment with no date never counts toward any nadir, its own included.

Read the source datasets from /app/input and save the completed dataset as /app/output/adtr.csv.

prompts/adam-adtr-nadir/conventions.md

Edit
12 lines

Following CDISC ADaM standards, use the provided ADTR_RAW dataset to create an ADTR dataset with one record per subject per visit.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, AVISIT, AVISITN, ADT, PARAMCD, PARAM, AVAL, NMEAS, NTARGET, ANL01FL, NADIR

Use PARAMCD "SDIAM" and PARAM "Sum of Target Lesion Diameters (mm)" on every record.

Read the source datasets from /app/input and save the completed dataset as /app/output/adtr.csv.

prompts/adam-adtr-nadir/brief.md

Edit
9 lines

Following CDISC ADaM standards, use the provided ADTR_RAW dataset to create an ADTR dataset with one record per subject per visit.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, AVISIT, AVISITN, ADT, PARAMCD, PARAM, AVAL, NMEAS, NTARGET, ANL01FL, NADIR

Read the source datasets from /app/input and save the completed dataset as /app/output/adtr.csv.

yamaa code

Runs the yamaa specification; its result must match the expected output

run.R

Edit
4 lines
library(yamaa)adtr <- yamaa_domain("spec.yaml")$outputadtr

run.py

Edit
4 lines
import yamaaadtr = yamaa.yamaa_domain("spec.yaml").outputadtr

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/adam-adtr-nadir/result.R

Edit
44 lines
# Reference solution for the yamaa benchmark adam-adtr-nadir (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)raw <- read_csv(  "/app/input/adtr_raw.csv",  col_types = cols(    AVISITN = col_integer(), ADT = col_date(), AVAL = col_double(),    NMEAS = col_integer(), NTARGET = col_integer(),    .default = col_character()  ))# Complete dated assessments carry the nadir candidates; a complete# assessment with no date never counts toward any nadir, its own# included.complete <- raw |>  filter(ANL01FL == "Y", !is.na(ADT), !is.na(AVAL)) |>  select(USUBJID, ADT, AVAL)# An incomplete current assessment keeps the nadir set by an earlier# complete one, while a current record with no date has none.adtr <- raw |>  rowwise() |>  mutate(    NADIR = {      cands <- complete$AVAL[complete$USUBJID == USUBJID & complete$ADT <= ADT]      if (is.na(ADT) || length(cands) == 0L) NA_real_ else min(cands)    }  ) |>  ungroup() |>  select(    STUDYID, USUBJID, AVISIT, AVISITN, ADT, PARAMCD, PARAM,    AVAL, NMEAS, NTARGET, ANL01FL, NADIR  ) |>  arrange(USUBJID, AVISITN)dir.create("/app/output", showWarnings = FALSE)write_csv(adtr, "/app/output/adtr.csv", na = "")

solutions/adam-adtr-nadir/result.py

Edit
57 lines
# Reference solution for the yamaa benchmark adam-adtr-nadir (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.from pathlib import Pathimport polars as plraw = pl.read_csv("/app/input/adtr_raw.csv", infer_schema=False, null_values="").with_columns(    pl.col("AVISITN").cast(pl.Int64, strict=False),    pl.col("ADT").str.to_date(strict=False),    pl.col("AVAL").cast(pl.Float64, strict=False),    pl.col("NMEAS").cast(pl.Int64, strict=False),    pl.col("NTARGET").cast(pl.Int64, strict=False),)# Complete dated assessments carry the nadir candidates; a complete# assessment with no date never counts toward any nadir, its own# included. An incomplete current assessment keeps the nadir set by an# earlier complete one, while a current record with no date has none.complete = [    (r["USUBJID"], r["ADT"], r["AVAL"])    for r in raw.to_dicts()    if r["ANL01FL"] == "Y" and r["ADT"] is not None and r["AVAL"] is not None]nadirs = []for r in raw.to_dicts():    if r["ADT"] is None:        nadirs.append(None)        continue    cands = [v for (u, d, v) in complete if u == r["USUBJID"] and d <= r["ADT"]]    nadirs.append(min(cands) if cands else None)adtr = (    raw.with_columns(NADIR=pl.Series(nadirs, dtype=pl.Float64))    .select(        "STUDYID",        "USUBJID",        "AVISIT",        "AVISITN",        "ADT",        "PARAMCD",        "PARAM",        "AVAL",        "NMEAS",        "NTARGET",        "ANL01FL",        "NADIR",    )    .sort(["USUBJID", "AVISITN"]))Path("/app/output").mkdir(parents=True, exist_ok=True)adtr.write_csv("/app/output/adtr.csv", null_value="")

solutions/adam-adtr-nadir/result.sas

Edit
18 lines
/* opensas reference solution for the yamaa benchmark adam-adtr-nadir.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data adtr_raw;  length STUDYID USUBJID AVISIT ADT PARAMCD PARAM ANL01FL $1024;  infile "/app/input/adtr_raw.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. AVISIT : $char1024. AVISITN ADT : $char1024. PARAMCD : $char1024. PARAM : $char1024. AVAL NMEAS NTARGET ANL01FL : $char1024.;run;data history; set adtr_raw; PRIORDATE=ADT; PRIORVALUE=AVAL; PRIORFLAG=ANL01FL; keep STUDYID USUBJID PRIORDATE PRIORVALUE PRIORFLAG; run;proc sql;create table nadirs as select a.STUDYID,a.USUBJID,a.AVISIT,min(b.PRIORVALUE) as NADIR from adtr_raw a left join history b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID and not missing(a.ADT) and not missing(b.PRIORDATE) and b.PRIORDATE<=a.ADT and b.PRIORFLAG='Y' group by a.STUDYID,a.USUBJID,a.AVISIT;create table result as select a.*,b.NADIR from adtr_raw a left join nadirs b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID and a.AVISIT=b.AVISIT; quit;proc sql;  create table final as select STUDYID, USUBJID, AVISIT, AVISITN, ADT, PARAMCD, PARAM, AVAL, NMEAS, NTARGET, ANL01FL, NADIR from result;quit;proc export data=final outfile="/app/output/adtr.csv" dbms=csv replace; run;