Review ask

As a clinical trial statistician or programmer, challenge anything in this benchmark: describe a problem in Comments, or propose a fix with the Edit button on any source file.

  1. Task: Would a real study need this derivation, specified this way? Does it follow CDISC ADaM: variable names, controlled terms, record structure? Flag any rule that is ambiguous or that a sponsor would handle differently.
  2. Inputs: Do they look like data collected in a real trial, and do they include the situations you would expect to meet in one?
  3. Expected output: Would independent QC programming reproduce every value from these inputs and rules?

The other tabs beside Comments are background for assessment, not under review.

Summary

Lifecycle: reviewedEditUnder review
What this benchmark means
README.md

Goal: build one record for each subject and scheduled tumor assessment holding the visit order (AVISITN), the assessment date (ADT), the summed diameters in millimeters (AVAL), the counts behind the sum (NMEAS and NTARGET), and the completeness flag (ANL01FL).

Input: scheduled tumor assessments with visit order and date, tumor measurement records with lesion group, test code, numeric result, and completion status, plus the lesion inventory selected at study entry with lesion group and lesion identifier.

Variables:

  • AVISITN is the numeric order of the assessment, carried from the scheduled assessment; it orders the assessments of a subject.
  • ADT is the date of the assessment, carried from the scheduled assessment; missing when the assessment has no recorded date.
  • AVAL is the sum of TRSTRESN over the target longest-diameter records at the assessment, those with TRGRPID of TARGET and TRTESTCD of LDIAM (longest diameter). A record with a missing result contributes nothing, while a zero result (a lesion that disappeared) counts as measured; AVAL is missing when the assessment measured no target lesion.
  • NMEAS is how many target lesions the assessment measured: the count of non-missing TRSTRESN values among those records. It is missing when the assessment has no target longest-diameter records, and 0 when it has records but none carries a result; this separates an assessment that was never performed from one that measured nothing.
  • NTARGET is how many lesions were selected as target lesions at study entry: the count of inventory records with TUGRPID of TARGET, counting TULNKID; it is the same at every assessment of the subject.
  • ANL01FL is Y when every target lesion was measured, that is when NMEAS equals NTARGET; blank otherwise.

Note: lesions outside the target inventory never enter the sum, and a sum over an incomplete set of target lesions is still reported; ANL01FL marks the assessments where every target lesion was measured.

Inputs

Under review
3 source files

input/tr.csv

Edit
17 rows
input/tr.csv
STUDYIDUSUBJIDAVISITTRSEQTRLNKIDTRGRPIDTRTESTCDTRSTRESNTRSTAT
YAMAA-01BASELINE1L1TARGETLDIAM40—
YAMAA-01BASELINE2L2TARGETLDIAM25—
YAMAA-01BASELINE3L3TARGETLDIAM15—
YAMAA-01WEEK 84L1TARGETLDIAM30—
YAMAA-01WEEK 85L2TARGETLDIAM20—
YAMAA-01WEEK 86L3TARGETLDIAM—NOT DONE
YAMAA-01BASELINE1L1TARGETLDIAM22—
YAMAA-01BASELINE2L2TARGETLDIAM18—
YAMAA-01WEEK 83L1TARGETLDIAM12—
YAMAA-01WEEK 84L2TARGETLDIAM10—
YAMAA-01BASELINE1L1TARGETLDIAM50—
YAMAA-01BASELINE2L2NON-TARGETLDIAM14—
YAMAA-01WEEK 83L1TARGETLDIAM45—
YAMAA-01BASELINE1L1TARGETLDIAM—NOT DONE
YAMAA-01BASELINE2L2TARGETLDIAM—NOT DONE
YAMAA-01WEEK 83L1TARGETLDIAM0—
YAMAA-01WEEK 84L2TARGETLDIAM30—

input/trvisit.csv

Edit
9 rows
input/trvisit.csv
STUDYIDUSUBJIDAVISITAVISITNADT
YAMAA-01BASELINE02025-01-05
YAMAA-01WEEK 882025-03-02
YAMAA-01WEEK 16162025-04-27
YAMAA-01BASELINE02025-01-12
YAMAA-01WEEK 882025-03-09
YAMAA-01BASELINE02025-01-19
YAMAA-01WEEK 88—
YAMAA-01BASELINE02025-02-03
YAMAA-01WEEK 882025-03-30

input/tu.csv

Edit
9 rows
input/tu.csv
STUDYIDUSUBJIDTULNKIDTUGRPID
YAMAA-01L1TARGET
YAMAA-01L2TARGET
YAMAA-01L3TARGET
YAMAA-01L1TARGET
YAMAA-01L2TARGET
YAMAA-01L1TARGET
YAMAA-01L2NON-TARGET
YAMAA-01L1TARGET
YAMAA-01L2TARGET

Expected output

Under review
9 expected rows

expected/adtr.csv

Edit
9 rows
expected/adtr.csv
STUDYIDUSUBJIDAVISITAVISITNADTPARAMCDPARAMAVALNMEASNTARGETANL01FL
YAMAA-01BASELINE02025-01-05SDIAMSum of Target Lesion Diameters (mm)8033Y
YAMAA-01WEEK 882025-03-02SDIAMSum of Target Lesion Diameters (mm)5023—
YAMAA-01WEEK 16162025-04-27SDIAMSum of Target Lesion Diameters (mm)——3—
YAMAA-01BASELINE02025-01-12SDIAMSum of Target Lesion Diameters (mm)4022Y
YAMAA-01WEEK 882025-03-09SDIAMSum of Target Lesion Diameters (mm)2222Y
YAMAA-01BASELINE02025-01-19SDIAMSum of Target Lesion Diameters (mm)5011Y
YAMAA-01WEEK 88—SDIAMSum of Target Lesion Diameters (mm)4511Y
YAMAA-01BASELINE02025-02-03SDIAMSum of Target Lesion Diameters (mm)—02—
YAMAA-01WEEK 882025-03-30SDIAMSum of Target Lesion Diameters (mm)3022Y
Shaded columns are derived— = missing value

Comments and assessment materials

Comments

Signed in with GitHub; saved in repository Discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Dataset NameVariable NameVariable LabelTypeCodelist/Controlled TermsCoreComputational MethodOrigin
ADTRSTUDYIDStudy IdentifiercharacterCopy value from TRVISIT.STUDYID.Predecessor
ADTRUSUBJIDUnique Subject IdentifiercharacterCopy value from TRVISIT.USUBJID.Predecessor
ADTRAVISITAnalysis VisitcharacterCopy value from TRVISIT.AVISIT.Predecessor
ADTRAVISITNAnalysis Visit (N)numericCopy value from TRVISIT.AVISITN.Predecessor
ADTRADTAnalysis DatenumericISO 8601Copy value from TRVISIT.ADT.Predecessor
ADTRPARAMCDParameter CodecharacterSet constant value "SDIAM".Assigned
ADTRPARAMParametercharacterSet constant value "Sum of Target Lesion Diameters (mm)".Assigned
ADTRAVALAnalysis ValuenumericAggregate SUM(TR.TRSTRESN) over records matching on the shared output keys, where TR.TRGRPID = 'TARGET' AND TR.TRTESTCD = 'LDIAM'.Derived
ADTRNMEASNumber of Measured Target LesionsnumericAggregate COUNT(TR.TRSTRESN) over records matching on the shared output keys, where TR.TRGRPID = 'TARGET' AND TR.TRTESTCD = 'LDIAM'.Derived
ADTRNTARGETNumber of Target LesionsnumericAggregate COUNT(TU.TULNKID) over records matching on the shared output keys, where TU.TUGRPID = 'TARGET'.Derived
ADTRANL01FLAnalysis Record Flag 01characterY"Y" when NMEAS = NTARGET; blank otherwise.Derived
VariablePermitted values
ANL01FLY
VersionDateAuthorDescriptionReviewerSign-off
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/adam-adtr-sum/full.md

Edit
30 lines

Following CDISC ADaM standards, use the provided TRVISIT, TR, and TU datasets to create an ADTR dataset with one record per subject per scheduled tumor assessment.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, AVISIT, AVISITN, ADT, PARAMCD, PARAM, AVAL, NMEAS, NTARGET, ANL01FL

Use PARAMCD "SDIAM" and PARAM "Sum of Target Lesion Diameters (mm)" on every record.

AVAL is the sum of the target longest-diameter records at the assessment: records with lesion group TARGET and test code LDIAM. A record with no result contributes nothing, while a zero result counts as measured; AVAL has no value when the assessment measured no target lesion.

NMEAS is how many target lesions the assessment measured: the count of records with a result. It has no value when the assessment has no target longest-diameter records, and is 0 when it has records but none carries a result. NTARGET is how many lesions were selected as target lesions at study entry, and is the same at every assessment of the subject.

ANL01FL is Y when every target lesion was measured, that is when NMEAS equals NTARGET, and has no value otherwise. Lesions outside the target inventory never enter the sum.

Read the source datasets from /app/input and save the completed dataset as /app/output/adtr.csv.

prompts/adam-adtr-sum/conventions.md

Edit
15 lines

Following CDISC ADaM standards, use the provided TRVISIT, TR, and TU datasets to create an ADTR dataset with one record per subject per scheduled tumor assessment.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, AVISIT, AVISITN, ADT, PARAMCD, PARAM, AVAL, NMEAS, NTARGET, ANL01FL

Use PARAMCD "SDIAM" and PARAM "Sum of Target Lesion Diameters (mm)" on every record.

ANL01FL is Y or has no value.

Read the source datasets from /app/input and save the completed dataset as /app/output/adtr.csv.

prompts/adam-adtr-sum/brief.md

Edit
10 lines

Following CDISC ADaM standards, use the provided TRVISIT, TR, and TU datasets to create an ADTR dataset with one record per subject per scheduled tumor assessment.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, AVISIT, AVISITN, ADT, PARAMCD, PARAM, AVAL, NMEAS, NTARGET, ANL01FL

Read the source datasets from /app/input and save the completed dataset as /app/output/adtr.csv.

yamaa code

Runs the yamaa specification; its result must match the expected output

run.R

Edit
4 lines
library(yamaa)adtr <- yamaa_domain("spec.yaml")$outputadtr

run.py

Edit
4 lines
import yamaaadtr = yamaa.yamaa_domain("spec.yaml").outputadtr

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/adam-adtr-sum/result.R

Edit
57 lines
# Reference solution for the yamaa benchmark adam-adtr-sum (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)trvisit <- read_csv(  "/app/input/trvisit.csv",  col_types = cols(AVISITN = col_integer(), ADT = col_date(), .default = col_character()))tr <- read_csv(  "/app/input/tr.csv",  col_types = cols(TRSTRESN = col_double(), .default = col_character()))tu <- read_csv(  "/app/input/tu.csv",  col_types = cols(.default = col_character()))# Lesions selected as target at study entry: the same count at every# assessment of the subject.ntarget <- tu |>  filter(TUGRPID == "TARGET") |>  count(STUDYID, USUBJID, name = "NTARGET")# Target longest diameters at each assessment. A record with no result# contributes nothing, while a zero counts as measured.agg <- tr |>  filter(TRGRPID == "TARGET", TRTESTCD == "LDIAM") |>  group_by(STUDYID, USUBJID, AVISIT) |>  summarise(    AVAL_sum = sum(TRSTRESN, na.rm = TRUE),    NMEAS = sum(!is.na(TRSTRESN)),    .groups = "drop"  ) |>  mutate(AVAL_sum = if_else(NMEAS == 0L, NA_real_, AVAL_sum))adtr <- trvisit |>  left_join(agg, by = c("STUDYID", "USUBJID", "AVISIT")) |>  left_join(ntarget, by = c("STUDYID", "USUBJID")) |>  mutate(    AVAL = AVAL_sum,    PARAMCD = "SDIAM",    PARAM = "Sum of Target Lesion Diameters (mm)",    ANL01FL = if_else(!is.na(NMEAS) & !is.na(NTARGET) & NMEAS == NTARGET, "Y", NA_character_)  ) |>  select(    STUDYID, USUBJID, AVISIT, AVISITN, ADT, PARAMCD, PARAM,    AVAL, NMEAS, NTARGET, ANL01FL  ) |>  arrange(USUBJID, AVISITN)dir.create("/app/output", showWarnings = FALSE)write_csv(adtr, "/app/output/adtr.csv", na = "")

solutions/adam-adtr-sum/result.py

Edit
76 lines
# Reference solution for the yamaa benchmark adam-adtr-sum (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.from pathlib import Pathimport polars as pltrvisit = pl.read_csv("/app/input/trvisit.csv", infer_schema=False, null_values="").with_columns(    pl.col("AVISITN").cast(pl.Int64, strict=False),    pl.col("ADT").str.to_date(strict=False),)tr = pl.read_csv("/app/input/tr.csv", infer_schema=False, null_values="").with_columns(    pl.col("TRSTRESN").cast(pl.Float64, strict=False),)tu = pl.read_csv("/app/input/tu.csv", infer_schema=False, null_values="")# Lesions selected as target at study entry: the same count at every# assessment of the subject.ntarget = (    tu.filter(pl.col("TUGRPID") == "TARGET")    .group_by(["STUDYID", "USUBJID"])    .agg(NTARGET=pl.len()))# Target longest diameters at each assessment. A record with no result# contributes nothing, while a zero counts as measured.agg = (    tr.filter((pl.col("TRGRPID") == "TARGET") & (pl.col("TRTESTCD") == "LDIAM"))    .group_by(["STUDYID", "USUBJID", "AVISIT"])    .agg(        __sum=pl.col("TRSTRESN").sum(),        __cnt=pl.col("TRSTRESN").count(),    ))adtr = (    trvisit.join(agg, on=["STUDYID", "USUBJID", "AVISIT"], how="left", maintain_order="left")    .join(ntarget, on=["STUDYID", "USUBJID"], how="left")    .with_columns(        AVAL=pl.when(pl.col("__cnt") == 0)        .then(None)        .otherwise(pl.col("__sum")),        NMEAS=pl.col("__cnt"),        PARAMCD=pl.lit("SDIAM"),        PARAM=pl.lit("Sum of Target Lesion Diameters (mm)"),    )    .with_columns(        ANL01FL=pl.when(            pl.col("NMEAS").is_not_null()            & pl.col("NTARGET").is_not_null()            & (pl.col("NMEAS") == pl.col("NTARGET"))        )        .then(pl.lit("Y"))        .otherwise(None),    )    .select(        "STUDYID",        "USUBJID",        "AVISIT",        "AVISITN",        "ADT",        "PARAMCD",        "PARAM",        "AVAL",        "NMEAS",        "NTARGET",        "ANL01FL",    )    .sort(["USUBJID", "AVISITN"]))Path("/app/output").mkdir(parents=True, exist_ok=True)adtr.write_csv("/app/output/adtr.csv", null_value="")

solutions/adam-adtr-sum/result.sas

Edit
38 lines
/* opensas reference solution for the yamaa benchmark adam-adtr-sum.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data tr;  length STUDYID USUBJID AVISIT TRLNKID TRGRPID TRTESTCD TRSTAT $1024;  infile "/app/input/tr.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. AVISIT : $char1024. TRSEQ TRLNKID : $char1024. TRGRPID : $char1024. TRTESTCD : $char1024. TRSTRESN TRSTAT : $char1024.;run;data trvisit;  length STUDYID USUBJID AVISIT ADT $1024;  infile "/app/input/trvisit.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. AVISIT : $char1024. AVISITN ADT : $char1024.;run;data tu;  length STUDYID USUBJID TULNKID TUGRPID $1024;  infile "/app/input/tu.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. TULNKID : $char1024. TUGRPID : $char1024.;run;proc sql;  create table inventory as select STUDYID,USUBJID,count(*) as NTARGET from tu where TUGRPID='TARGET' group by STUDYID,USUBJID;  create table measures as select a.* from tr as a inner join tu as b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID and a.TRLNKID=b.TULNKID where a.TRGRPID='TARGET' and a.TRTESTCD='LDIAM' and b.TUGRPID='TARGET';  create table totals as select STUDYID,USUBJID,AVISIT,sum(TRSTRESN) as AVAL,count(TRSTRESN) as NMEAS from measures group by STUDYID,USUBJID,AVISIT;  create table combined as select a.*,b.AVAL,b.NMEAS,c.NTARGET from trvisit as a left join totals as b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID and a.AVISIT=b.AVISIT left join inventory as c on a.STUDYID=c.STUDYID and a.USUBJID=c.USUBJID;quit;data result;  length PARAMCD $8 PARAM $40 ANL01FL $1;  set combined; PARAMCD='SDIAM'; PARAM='Sum of Target Lesion Diameters (mm)';  if not missing(NMEAS) and not missing(NTARGET) and NMEAS=NTARGET then ANL01FL='Y';run;proc sql;  create table final as select STUDYID, USUBJID, AVISIT, AVISITN, ADT, PARAMCD, PARAM, AVAL, NMEAS, NTARGET, ANL01FL from result;quit;proc export data=final outfile="/app/output/adtr.csv" dbms=csv replace; run;