yamaa As a clinical trial statistician or programmer, challenge anything in this benchmark: describe a problem in Comments, or propose a fix with the Edit button on any source file.
- Task: Would a real study need this derivation, specified this way? Does it follow CDISC SDTM: variable names, controlled terms, record structure? Flag any rule that is ambiguous or that a sponsor would handle differently.
- Inputs: Do they look like data collected in a real trial, and do they include the situations you would expect to meet in one?
- Expected output: Would independent QC programming reproduce every value from these inputs and rules?
The other tabs beside Comments are background for assessment, not under review.
Summary
EditUnder reviewWhat this benchmark meansREADME.mdGoal: build one record per collected tumor assessment, carrying
the collected test, result, unit, method, evaluator, and visit
through and adding the standardized numeric result, text result,
and standard unit: TRTESTCD, TRTEST, TRORRES, TRORRESU,
TRSTRESN, TRSTRESC, TRSTRESU, and TRSTAT.
Input: collected lesion assessments with one row per lesion per
visit, carrying the collected result and unit plus study, subject,
sequence (TRSEQ), link, method, evaluator, visit, and date
identifiers.
Variables:
TRLNKIDis the link identifier tying the assessment to its lesion in the tumor identification (TU) domain; target, non-target, and new lesions each keep their own identifier.TRTESTCDis the test code as collected:DIAMETERfor a measured lesion orTUMSTATEfor a lesion state.TRTESTis the test name as collected:DiameterorTumor State.TRORRESis the result exactly as collected, never overwritten: a diameter,TOO SMALL TO MEASURE, a state such asPRESENT, or empty when the lesion was not assessed.TRORRESUis the unit exactly as collected:mmorcmfor a diameter, empty otherwise.TRSTRESCis the standardized character result: a diameter in mm written as text,5for a lesion too small to measure, a lesion state kept as collected, and empty when the lesion was not assessed.TRSTRESNis the standardized numeric result in mm: a diameter passes through, a value collected incmis multiplied by 10, and a lesion too small to measure takes the study convention of 5 mm; it stays empty for lesion states and for lesions not assessed.TRSTRESUismmwhenever a standardized numeric result exists, empty otherwise.TRSTATisNOT DONEfor a lesion not assessed at a visit, empty otherwise.TRMETHODis the measurement method as collected.TREVALnames the evaluator as collected.VISITNUMis the visit number as collected.TRDTCis the assessment date as collected, empty when the lesion was not assessed.
Note: a lymph-node target lesion contributes its short axis as its diameter, following the study's response criteria; that axis is already the value the sum of diameters needs, so it is carried like any other diameter. A lesion too small to measure is not a zero: it takes the study's conventional 5 mm, and a lesion not assessed stays out of every standardized column rather than counting as zero.
Inputs
Under review1 source fileinput/tr_raw.csv
Edit10 rows| STUDYID | USUBJID | TRSEQ | TRLNKID | TRTESTCD | TRTEST | TRORRES | TRORRESU | TRSTAT | TRMETHOD | TREVAL | VISITNUM | TRDTC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| YAMAA-01 | 1 | T01 | DIAMETER | Diameter | 32 | mm | — | CT SCAN | INVESTIGATOR | 1 | 2026-01-05 | |
| YAMAA-01 | 2 | T01 | DIAMETER | Diameter | 2.8 | cm | — | CT SCAN | INVESTIGATOR | 2 | 2026-03-02 | |
| YAMAA-01 | 3 | T01 | DIAMETER | Diameter | 25 | mm | — | CT SCAN | INVESTIGATOR | 3 | 2026-04-27 | |
| YAMAA-01 | 4 | T02 | DIAMETER | Diameter | 18 | mm | — | CT SCAN | INVESTIGATOR | 1 | 2026-01-05 | |
| YAMAA-01 | 5 | T02 | DIAMETER | Diameter | — | — | NOT DONE | CT SCAN | INVESTIGATOR | 2 | — | |
| YAMAA-01 | 6 | T02 | DIAMETER | Diameter | TOO SMALL TO MEASURE | mm | — | CT SCAN | INVESTIGATOR | 3 | 2026-04-27 | |
| YAMAA-01 | 7 | NT01 | TUMSTATE | Tumor State | PRESENT | — | — | CT SCAN | INVESTIGATOR | 1 | 2026-01-05 | |
| YAMAA-01 | 8 | NT01 | TUMSTATE | Tumor State | PRESENT | — | — | CT SCAN | INVESTIGATOR | 2 | 2026-03-02 | |
| YAMAA-01 | 9 | NT01 | TUMSTATE | Tumor State | ABSENT | — | — | CT SCAN | INVESTIGATOR | 3 | 2026-04-27 | |
| YAMAA-01 | 10 | NEW01 | TUMSTATE | Tumor State | PRESENT | — | — | CT SCAN | INVESTIGATOR | 3 | 2026-04-27 |
Expected output
Under review10 expected rowsexpected/tr.csv
Edit10 rows| DOMAIN | STUDYID | USUBJID | TRSEQ | TRLNKID | TRTESTCD | TRTEST | TRORRES | TRORRESU | TRSTRESC | TRSTRESN | TRSTRESU | TRSTAT | TRMETHOD | TREVAL | VISITNUM | TRDTC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| TR | YAMAA-01 | 1 | T01 | DIAMETER | Diameter | 32 | mm | 32 | 32 | mm | — | CT SCAN | INVESTIGATOR | 1 | 2026-01-05 | |
| TR | YAMAA-01 | 2 | T01 | DIAMETER | Diameter | 2.8 | cm | 28 | 28 | mm | — | CT SCAN | INVESTIGATOR | 2 | 2026-03-02 | |
| TR | YAMAA-01 | 3 | T01 | DIAMETER | Diameter | 25 | mm | 25 | 25 | mm | — | CT SCAN | INVESTIGATOR | 3 | 2026-04-27 | |
| TR | YAMAA-01 | 4 | T02 | DIAMETER | Diameter | 18 | mm | 18 | 18 | mm | — | CT SCAN | INVESTIGATOR | 1 | 2026-01-05 | |
| TR | YAMAA-01 | 5 | T02 | DIAMETER | Diameter | — | — | — | — | — | NOT DONE | CT SCAN | INVESTIGATOR | 2 | — | |
| TR | YAMAA-01 | 6 | T02 | DIAMETER | Diameter | TOO SMALL TO MEASURE | mm | 5 | 5 | mm | — | CT SCAN | INVESTIGATOR | 3 | 2026-04-27 | |
| TR | YAMAA-01 | 7 | NT01 | TUMSTATE | Tumor State | PRESENT | — | PRESENT | — | — | — | CT SCAN | INVESTIGATOR | 1 | 2026-01-05 | |
| TR | YAMAA-01 | 8 | NT01 | TUMSTATE | Tumor State | PRESENT | — | PRESENT | — | — | — | CT SCAN | INVESTIGATOR | 2 | 2026-03-02 | |
| TR | YAMAA-01 | 9 | NT01 | TUMSTATE | Tumor State | ABSENT | — | ABSENT | — | — | — | CT SCAN | INVESTIGATOR | 3 | 2026-04-27 | |
| TR | YAMAA-01 | 10 | NEW01 | TUMSTATE | Tumor State | PRESENT | — | PRESENT | — | — | — | CT SCAN | INVESTIGATOR | 3 | 2026-04-27 |
Comments and assessment materials
| Variable Name | Variable Label | Type | Length | Controlled Terms or Format | Origin | Core | Conversion Definition | Variable Type | Variable Order | Comments for Define |
|---|---|---|---|---|---|---|---|---|---|---|
| DOMAIN | Domain Abbreviation | Char | Assigned | Set constant value "TR". | SDTM | 1 | ||||
| STUDYID | Study Identifier | Char | Collected | Copy value from TR_RAW.STUDYID. | SDTM | 2 | ||||
| USUBJID | Unique Subject Identifier | Char | Collected | Copy value from TR_RAW.USUBJID. | SDTM | 3 | ||||
| TRSEQ | Sequence Number | Num | Collected | Copy value from TR_RAW.TRSEQ. | SDTM | 4 | ||||
| TRLNKID | Link ID | Char | Collected | Copy value from TR_RAW.TRLNKID. | SDTM | 5 | ||||
| TRTESTCD | Tumor Result Test Short Name | Char | DIAMETER, TUMSTATE | Collected | Copy value from TR_RAW.TRTESTCD. | SDTM | 6 | |||
| TRTEST | Tumor Result Test Name | Char | Collected | Copy value from TR_RAW.TRTEST. | SDTM | 7 | ||||
| TRORRES | Result or Finding in Original Units | Char | Collected | Copy value from TR_RAW.TRORRES. | SDTM | 8 | ||||
| TRORRESU | Original Units | Char | mm, cm | Collected | Copy value from TR_RAW.TRORRESU. | SDTM | 9 | |||
| TRSTRESN | Numeric Result/Finding in Standard Units | Num | Derived | If TR_RAW.TRORRES = 'TOO SMALL TO MEASURE' then "5"; If TR_RAW.TRTESTCD = 'DIAMETER' AND TR_RAW.TRORRESU = 'cm' then compute TRORRES_N * 10; If TR_RAW.TRTESTCD = 'DIAMETER' then compute TRORRES_N; otherwise blank. | SDTM | 10 | ||||
| TRSTRESC | Character Result/Finding in Std Format | Char | Derived | If TR_RAW.TRTESTCD = 'DIAMETER' then "TRSTRESN"; otherwise "TR_RAW.TRORRES". | SDTM | 11 | ||||
| TRSTRESU | Standard Units | Char | mm | Derived | If TRSTRESN IS NOT NULL then "mm"; otherwise blank. | SDTM | 12 | |||
| TRSTAT | Completion Status | Char | NOT DONE | Collected | Copy value from TR_RAW.TRSTAT. | SDTM | 13 | |||
| TRMETHOD | Method of Test or Examination | Char | Collected | Copy value from TR_RAW.TRMETHOD. | SDTM | 14 | ||||
| TREVAL | Evaluator | Char | Collected | Copy value from TR_RAW.TREVAL. | SDTM | 15 | ||||
| VISITNUM | Visit Number | Num | Collected | Copy value from TR_RAW.VISITNUM. | SDTM | 16 | ||||
| TRDTC | Date/Time of Tumor Measurement | Char | ISO 8601 | Collected | Copy value from TR_RAW.TRDTC. | SDTM | 17 |
| Variable | Permitted values |
|---|---|
| TRTESTCD | DIAMETER, TUMSTATE |
| TRORRESU | mm, cm |
| TRSTRESU | mm |
| TRSTAT | NOT DONE |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/sdtm-tr-tumor-measurements/full.md
Edit34 linesFollowing CDISC SDTM standards, use the provided TR_RAW dataset to create a TR dataset with one record per collected tumor assessment.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, TRSEQ, TRLNKID, TRTESTCD, TRTEST, TRORRES, TRORRESU, TRSTRESC, TRSTRESN, TRSTRESU, TRSTAT, TRMETHOD, TREVAL, VISITNUM, TRDTC
TRLNKID is the link identifier tying the assessment to its lesion in the tumor identification domain. TRTESTCD is DIAMETER for a measured lesion or TUMSTATE for a lesion state; TRTEST is Diameter or Tumor State. TRORRES is the result exactly as collected, never overwritten: a diameter, TOO SMALL TO MEASURE, a state such as PRESENT, or has no value when the lesion was not assessed. TRORRESU is the unit exactly as collected (mm or cm for a diameter), and has no value otherwise.
TRSTRESC is the diameter in mm written as text (a whole number has no decimal point), 5 for a lesion too small to measure, or a lesion state kept as collected; it has no value when the lesion was not assessed. TRSTRESN is the diameter in mm as a number (a value collected in cm is multiplied by 10), or 5 for a lesion too small to measure; it has no value for lesion states and for lesions not assessed. TRSTRESU is mm whenever a standardized numeric result exists, and has no value otherwise. TRSTAT is NOT DONE for a lesion not assessed at a visit, and has no value otherwise. TRMETHOD, TREVAL, and VISITNUM are as collected; TRDTC is the assessment date as collected, and has no value when the lesion was not assessed.
A lymph-node target lesion contributes its short axis as its diameter. A lesion too small to measure is not a zero: it takes the study convention of 5 mm. A lesion not assessed stays out of every standardized column rather than counting as zero.
Read the source datasets from /app/input and save the completed dataset as /app/output/tr.csv.
prompts/sdtm-tr-tumor-measurements/conventions.md
Edit17 linesFollowing CDISC SDTM standards, use the provided TR_RAW dataset to create a TR dataset with one record per collected tumor assessment.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, TRSEQ, TRLNKID, TRTESTCD, TRTEST, TRORRES, TRORRESU, TRSTRESC, TRSTRESN, TRSTRESU, TRSTAT, TRMETHOD, TREVAL, VISITNUM, TRDTC
TRTESTCD is DIAMETER for a measured lesion or TUMSTATE for a lesion state; TRTEST is Diameter or Tumor State. TRORRES is a diameter, TOO SMALL TO MEASURE, a state such as PRESENT, or has no value.
In TRSTRESC a whole number has no decimal point. TRSTRESC and TRSTRESN are 5 for a lesion too small to measure. TRSTAT is NOT DONE or has no value.
Read the source datasets from /app/input and save the completed dataset as /app/output/tr.csv.
prompts/sdtm-tr-tumor-measurements/brief.md
Edit10 linesFollowing CDISC SDTM standards, use the provided TR_RAW dataset to create a TR dataset with one record per collected tumor assessment.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, TRSEQ, TRLNKID, TRTESTCD, TRTEST, TRORRES, TRORRESU, TRSTRESC, TRSTRESN, TRSTRESU, TRSTAT, TRMETHOD, TREVAL, VISITNUM, TRDTC
Read the source datasets from /app/input and save the completed dataset as /app/output/tr.csv.
run.R
Edit4 lineslibrary(yamaa)tr <- yamaa_domain("spec.yaml")$outputtrrun.py
Edit4 linesimport yamaatr = yamaa.yamaa_domain("spec.yaml").outputtrsolutions/sdtm-tr-tumor-measurements/result.R
Edit61 lines# Reference solution for the yamaa benchmark sdtm-tr-tumor-measurements (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)fmt_num <- function(x) { if (is.na(x)) { return(NA_character_) } if (x == floor(x)) { formatC(x, format = "f", digits = 0) } else { formatted <- format(x, scientific = FALSE, trim = TRUE, digits = 15) sub("\\.0+$", "", formatted) }}raw <- read_csv( "/app/input/tr_raw.csv", col_types = cols( TRSEQ = col_integer(), VISITNUM = col_integer(), .default = col_character() ))tr <- raw |> mutate( orres_num = suppressWarnings(as.numeric(TRORRES)), mm_value = if_else(TRORRESU == "cm", orres_num * 10, orres_num), TRSTRESN = case_when( TRTESTCD == "DIAMETER" & TRORRES == "TOO SMALL TO MEASURE" ~ 5, TRTESTCD == "DIAMETER" & !is.na(TRORRES) & TRORRES != "" & TRORRES != "TOO SMALL TO MEASURE" ~ mm_value, .default = NA_real_ ), TRSTRESC = case_when( TRTESTCD == "DIAMETER" & TRORRES == "TOO SMALL TO MEASURE" ~ "5", TRTESTCD == "DIAMETER" & !is.na(TRORRES) & TRORRES != "" & TRORRES != "TOO SMALL TO MEASURE" ~ vapply( TRSTRESN, fmt_num, character(1) ), TRTESTCD != "DIAMETER" & !is.na(TRORRES) & TRORRES != "" ~ TRORRES, .default = NA_character_ ), DOMAIN = "TR", TRSTRESU = if_else(!is.na(TRSTRESN), "mm", NA_character_) ) |> arrange(USUBJID, TRSEQ) |> select( DOMAIN, STUDYID, USUBJID, TRSEQ, TRLNKID, TRTESTCD, TRTEST, TRORRES, TRORRESU, TRSTRESC, TRSTRESN, TRSTRESU, TRSTAT, TRMETHOD, TREVAL, VISITNUM, TRDTC )dir.create("/app/output", showWarnings = FALSE)write_csv(tr, "/app/output/tr.csv", na = "")solutions/sdtm-tr-tumor-measurements/result.py
Edit103 lines# Reference solution for the yamaa benchmark sdtm-tr-tumor-measurements (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.from pathlib import Pathimport polars as pldef fmt_num(x: float | None) -> str | None: import math if x is None: return None if isinstance(x, float) and math.isnan(x): return None if float(x).is_integer(): return str(int(float(x))) text = repr(float(x)) if "e" in text or "E" in text: text = f"{float(x):.15f}".rstrip("0").rstrip(".") else: text = text.rstrip("0").rstrip(".") if "." in text else text return textraw = pl.read_csv("/app/input/tr_raw.csv", infer_schema=False).with_columns( pl.col("TRSEQ").cast(pl.Int64), pl.col("VISITNUM").cast(pl.Int64),)is_diameter = pl.col("TRTESTCD") == "DIAMETER"is_too_small = pl.col("TRORRES") == "TOO SMALL TO MEASURE"is_missing = pl.col("TRORRES").is_null() | (pl.col("TRORRES") == "")orres_num = pl.col("TRORRES").cast(pl.Float64, strict=False)mm_value = ( pl.when(pl.col("TRORRESU") == "cm") .then(orres_num * 10) .otherwise(orres_num))stresn = ( pl.when(is_diameter & is_too_small) .then(pl.lit(5.0)) .when(is_diameter & ~is_missing & ~is_too_small) .then(mm_value) .otherwise(pl.lit(None, dtype=pl.Float64)))tr = raw.with_columns(TRSTRESN=stresn)# Standardized text: 5 for too small, the lesion state as collected, the# diameter in mm without a decimal point for whole numbers, else no value.rows = tr.to_dicts()for row in rows: if row["TRTESTCD"] == "DIAMETER": if row["TRORRES"] == "TOO SMALL TO MEASURE": row["TRSTRESC"] = "5" elif row["TRORRES"] is None or row["TRORRES"] == "": row["TRSTRESC"] = None else: row["TRSTRESC"] = fmt_num(row["TRSTRESN"]) else: if row["TRORRES"] is None or row["TRORRES"] == "": row["TRSTRESC"] = None else: row["TRSTRESC"] = row["TRORRES"]tr = ( pl.DataFrame(rows) .with_columns( DOMAIN=pl.lit("TR"), TRSTRESU=pl.when(pl.col("TRSTRESN").is_not_null()) .then(pl.lit("mm")) .otherwise(pl.lit(None, dtype=pl.String)), ) .select( "DOMAIN", "STUDYID", "USUBJID", "TRSEQ", "TRLNKID", "TRTESTCD", "TRTEST", "TRORRES", "TRORRESU", "TRSTRESC", "TRSTRESN", "TRSTRESU", "TRSTAT", "TRMETHOD", "TREVAL", "VISITNUM", "TRDTC", ) .sort(["USUBJID", "TRSEQ"]))Path("/app/output").mkdir(exist_ok=True)tr.write_csv("/app/output/tr.csv")solutions/sdtm-tr-tumor-measurements/result.sas
Edit19 lines/* opensas reference solution for the yamaa benchmark sdtm-tr-tumor-measurements. Reads the benchmark inputs and derives the requested output. Execute with the existing opensas runtime: opensas /app/output/result.sas. */data tr_raw; length STUDYID USUBJID TRLNKID TRTESTCD TRTEST TRORRES TRORRESU TRSTAT TRMETHOD TREVAL TRDTC $1024; infile "/app/input/tr_raw.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. TRSEQ TRLNKID : $char1024. TRTESTCD : $char1024. TRTEST : $char1024. TRORRES : $char1024. TRORRESU : $char1024. TRSTAT : $char1024. TRMETHOD : $char1024. TREVAL : $char1024. VISITNUM TRDTC : $char1024.;run;data result; set tr_raw; length DOMAIN $2 TRSTRESC TRSTRESU $1024; DOMAIN='TR'; TRSTRESN=.; TRSTRESC=''; TRSTRESU='';if TRSTAT='NOT DONE' or missing(TRORRES) then do; TRSTAT='NOT DONE'; TRORRES=''; TRORRESU=''; TRDTC=''; end;else if TRTESTCD='TUMSTATE' then TRSTRESC=TRORRES;else do; if TRORRES='TOO SMALL TO MEASURE' then TRSTRESN=5; else do; TRSTRESN=input(TRORRES,?? best32.); if TRORRESU='cm' then TRSTRESN=TRSTRESN*10; end;if not missing(TRSTRESN) then do; TRSTRESC=strip(put(TRSTRESN,best32.)); TRSTRESU='mm'; end; end; run;proc sql; create table final as select DOMAIN, STUDYID, USUBJID, TRSEQ, TRLNKID, TRTESTCD, TRTEST, TRORRES, TRORRESU, TRSTRESC, TRSTRESN, TRSTRESU, TRSTAT, TRMETHOD, TREVAL, VISITNUM, TRDTC from result;quit;proc export data=final outfile="/app/output/tr.csv" dbms=csv replace; run;
Comments
Signed in with GitHub; saved in repository Discussions