yamaa As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.
We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.
Summary
EditUnder reviewWhat this benchmark meansREADME.mdPublic entries: Python and R. Passing coverage requires an installed inventory.
Goal: build one record per collected tumor assessment, carrying
the collected test, result, unit, method, evaluator, and visit
through and adding the standardized numeric result, text result,
and standard unit: TRTESTCD, TRTEST, TRORRES, TRORRESU,
TRSTRESN, TRSTRESC, TRSTRESU, and TRSTAT.
Input: collected lesion assessments with one row per lesion per
visit, carrying the collected result and unit plus study, subject,
sequence (TRSEQ), link, method, evaluator, visit, and date
identifiers.
Variables:
TRLNKIDis the link identifier tying the assessment to its lesion in the tumor identification (TU) domain; target, non-target, and new lesions each keep their own identifier.TRTESTCDis the test code as collected:DIAMETERfor a measured lesion orTUMSTATEfor a lesion state.TRTESTis the test name as collected:DiameterorTumor State.TRORRESis the result exactly as collected, never overwritten: a diameter,TOO SMALL TO MEASURE, a state such asPRESENT, or empty when the lesion was not assessed.TRORRESUis the unit exactly as collected:mmorcmfor a diameter, empty otherwise.TRSTRESCis the standardized character result: a diameter in mm written as text,5for a lesion too small to measure, a lesion state kept as collected, and empty when the lesion was not assessed.TRSTRESNis the standardized numeric result in mm: a diameter passes through, a value collected incmis multiplied by 10, and a lesion too small to measure takes the study convention of 5 mm; it stays empty for lesion states and for lesions not assessed.TRSTRESUismmwhenever a standardized numeric result exists, empty otherwise.TRSTATisNOT DONEfor a lesion not assessed at a visit, empty otherwise.TRMETHODis the measurement method as collected.TREVALnames the evaluator as collected.VISITNUMis the visit number as collected.TRDTCis the assessment date as collected, empty when the lesion was not assessed.
Note: a lymph-node target lesion contributes its short axis as its diameter, following the study's response criteria; that axis is already the value the sum of diameters needs, so it is carried like any other diameter. A lesion too small to measure is not a zero: it takes the study's conventional 5 mm, and a lesion not assessed stays out of every standardized column rather than counting as zero.
Inputs
Under review1 source fileinput/tr_raw.csv
Edit10 rows| STUDYID | USUBJID | TRSEQ | TRLNKID | TRTESTCD | TRTEST | TRORRES | TRORRESU | TRSTAT | TRMETHOD | TREVAL | VISITNUM | TRDTC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| YAMAA-01 | 1 | T01 | DIAMETER | Diameter | 32 | mm | — | CT SCAN | INVESTIGATOR | 1 | 2026-01-05 | |
| YAMAA-01 | 2 | T01 | DIAMETER | Diameter | 2.8 | cm | — | CT SCAN | INVESTIGATOR | 2 | 2026-03-02 | |
| YAMAA-01 | 3 | T01 | DIAMETER | Diameter | 25 | mm | — | CT SCAN | INVESTIGATOR | 3 | 2026-04-27 | |
| YAMAA-01 | 4 | T02 | DIAMETER | Diameter | 18 | mm | — | CT SCAN | INVESTIGATOR | 1 | 2026-01-05 | |
| YAMAA-01 | 5 | T02 | DIAMETER | Diameter | — | — | NOT DONE | CT SCAN | INVESTIGATOR | 2 | — | |
| YAMAA-01 | 6 | T02 | DIAMETER | Diameter | TOO SMALL TO MEASURE | mm | — | CT SCAN | INVESTIGATOR | 3 | 2026-04-27 | |
| YAMAA-01 | 7 | NT01 | TUMSTATE | Tumor State | PRESENT | — | — | CT SCAN | INVESTIGATOR | 1 | 2026-01-05 | |
| YAMAA-01 | 8 | NT01 | TUMSTATE | Tumor State | PRESENT | — | — | CT SCAN | INVESTIGATOR | 2 | 2026-03-02 | |
| YAMAA-01 | 9 | NT01 | TUMSTATE | Tumor State | ABSENT | — | — | CT SCAN | INVESTIGATOR | 3 | 2026-04-27 | |
| YAMAA-01 | 10 | NEW01 | TUMSTATE | Tumor State | PRESENT | — | — | CT SCAN | INVESTIGATOR | 3 | 2026-04-27 |
Expected output
Under review10 expected rowsexpected/tr.csv
Edit10 rows| DOMAIN | STUDYID | USUBJID | TRSEQ | TRLNKID | TRTESTCD | TRTEST | TRORRES | TRORRESU | TRSTRESC | TRSTRESN | TRSTRESU | TRSTAT | TRMETHOD | TREVAL | VISITNUM | TRDTC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| TR | YAMAA-01 | 1 | T01 | DIAMETER | Diameter | 32 | mm | 32 | 32 | mm | — | CT SCAN | INVESTIGATOR | 1 | 2026-01-05 | |
| TR | YAMAA-01 | 2 | T01 | DIAMETER | Diameter | 2.8 | cm | 28 | 28 | mm | — | CT SCAN | INVESTIGATOR | 2 | 2026-03-02 | |
| TR | YAMAA-01 | 3 | T01 | DIAMETER | Diameter | 25 | mm | 25 | 25 | mm | — | CT SCAN | INVESTIGATOR | 3 | 2026-04-27 | |
| TR | YAMAA-01 | 4 | T02 | DIAMETER | Diameter | 18 | mm | 18 | 18 | mm | — | CT SCAN | INVESTIGATOR | 1 | 2026-01-05 | |
| TR | YAMAA-01 | 5 | T02 | DIAMETER | Diameter | — | — | — | — | — | NOT DONE | CT SCAN | INVESTIGATOR | 2 | — | |
| TR | YAMAA-01 | 6 | T02 | DIAMETER | Diameter | TOO SMALL TO MEASURE | mm | 5 | 5 | mm | — | CT SCAN | INVESTIGATOR | 3 | 2026-04-27 | |
| TR | YAMAA-01 | 7 | NT01 | TUMSTATE | Tumor State | PRESENT | — | PRESENT | — | — | — | CT SCAN | INVESTIGATOR | 1 | 2026-01-05 | |
| TR | YAMAA-01 | 8 | NT01 | TUMSTATE | Tumor State | PRESENT | — | PRESENT | — | — | — | CT SCAN | INVESTIGATOR | 2 | 2026-03-02 | |
| TR | YAMAA-01 | 9 | NT01 | TUMSTATE | Tumor State | ABSENT | — | ABSENT | — | — | — | CT SCAN | INVESTIGATOR | 3 | 2026-04-27 | |
| TR | YAMAA-01 | 10 | NEW01 | TUMSTATE | Tumor State | PRESENT | — | PRESENT | — | — | — | CT SCAN | INVESTIGATOR | 3 | 2026-04-27 |
Human review and assessment
| Variable Name | Variable Label | Type | Length | Controlled Terms or Format | Origin | Core | Conversion Definition | Variable Type | Variable Order | Comments for Define |
|---|---|---|---|---|---|---|---|---|---|---|
| DOMAIN | Domain Abbreviation | Char | Assigned | Set constant value "TR". | SDTM | 1 | ||||
| STUDYID | Study Identifier | Char | Collected | Copy value from TR_RAW.STUDYID. | SDTM | 2 | ||||
| USUBJID | Unique Subject Identifier | Char | Collected | Copy value from TR_RAW.USUBJID. | SDTM | 3 | ||||
| TRSEQ | Sequence Number | Num | Collected | Copy value from TR_RAW.TRSEQ. | SDTM | 4 | ||||
| TRLNKID | Link ID | Char | Collected | Copy value from TR_RAW.TRLNKID. | SDTM | 5 | ||||
| TRTESTCD | Tumor Result Test Short Name | Char | DIAMETER, TUMSTATE | Collected | Copy value from TR_RAW.TRTESTCD. | SDTM | 6 | |||
| TRTEST | Tumor Result Test Name | Char | Collected | Copy value from TR_RAW.TRTEST. | SDTM | 7 | ||||
| TRORRES | Result or Finding in Original Units | Char | Collected | Copy value from TR_RAW.TRORRES. | SDTM | 8 | ||||
| TRORRESU | Original Units | Char | mm, cm | Collected | Copy value from TR_RAW.TRORRESU. | SDTM | 9 | |||
| TRSTRESN | Numeric Result/Finding in Standard Units | Num | Derived | If TR_RAW.TRORRES = 'TOO SMALL TO MEASURE' then "5"; If TR_RAW.TRTESTCD = 'DIAMETER' AND TR_RAW.TRORRESU = 'cm' then compute TRORRES_N * 10; If TR_RAW.TRTESTCD = 'DIAMETER' then "TRORRES_N"; otherwise blank. | SDTM | 10 | ||||
| TRSTRESC | Character Result/Finding in Std Format | Char | Derived | If TR_RAW.TRTESTCD = 'DIAMETER' then "TRSTRESN"; otherwise "TR_RAW.TRORRES". | SDTM | 11 | ||||
| TRSTRESU | Standard Units | Char | mm | Derived | If TRSTRESN IS NOT NULL then "mm"; otherwise blank. | SDTM | 12 | |||
| TRSTAT | Completion Status | Char | NOT DONE | Collected | Copy value from TR_RAW.TRSTAT. | SDTM | 13 | |||
| TRMETHOD | Method of Test or Examination | Char | Collected | Copy value from TR_RAW.TRMETHOD. | SDTM | 14 | ||||
| TREVAL | Evaluator | Char | Collected | Copy value from TR_RAW.TREVAL. | SDTM | 15 | ||||
| VISITNUM | Visit Number | Num | Collected | Copy value from TR_RAW.VISITNUM. | SDTM | 16 | ||||
| TRDTC | Date/Time of Tumor Measurement | Char | ISO 8601 | Collected | Copy value from TR_RAW.TRDTC. | SDTM | 17 |
| Variable | Permitted values |
|---|---|
| TRTESTCD | DIAMETER, TUMSTATE |
| TRORRESU | mm, cm |
| TRSTRESU | mm |
| TRSTAT | NOT DONE |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.7 | 2026-10-20 | yamaa docs build | Read a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default | ||
| 1.6 | 2026-10-17 | yamaa docs build | Describe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight | ||
| 1.5 | 2026-10-17 | yamaa docs build | Name collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part | ||
| 1.4 | 2026-10-15 | yamaa docs build | State datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks | ||
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/sdtm-tr-tumor-measurements/full.md
Edit30 linesFollowing CDISC SDTM standards, use the provided TR_RAW dataset to create a TR dataset with one record per collected tumor assessment.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, TRSEQ, TRLNKID, TRTESTCD, TRTEST, TRORRES, TRORRESU, TRSTRESC, TRSTRESN, TRSTRESU, TRSTAT, TRMETHOD, TREVAL, VISITNUM, TRDTC
DOMAIN is "TR". STUDYID and USUBJID are as given in TR_RAW, and TRSEQ is the sequence number as collected. TRLNKID is the link identifier tying the assessment to its lesion in the tumor identification (TU) domain; target, non-target, and new lesions each keep their own identifier. TRTESTCD is the test code as collected: "DIAMETER" for a measured lesion or "TUMSTATE" for a lesion state. TRTEST is the test name as collected: "Diameter" or "Tumor State". TRORRES is the result exactly as collected: a diameter, "TOO SMALL TO MEASURE", a state such as "PRESENT", or no value when the lesion was not assessed. TRORRESU is the unit exactly as collected: "mm" or "cm" for a diameter, and no value otherwise. TRSTRESN is the standardized numeric result in mm: a diameter collected in mm keeps its collected value, a diameter collected in cm is multiplied by 10, and a lesion too small to measure takes 5. TRSTRESN has no value for a lesion state or for a lesion that was not assessed. TRSTRESC is the standardized character result: for a diameter, the standardized numeric result in mm written as text, so a lesion too small to measure gives "5"; a lesion state is kept as collected; and it has no value when the lesion was not assessed. TRSTRESU is "mm" whenever TRSTRESN has a value, and has no value otherwise. TRSTAT is "NOT DONE" for a lesion not assessed at a visit, and has no value otherwise.
Read the source datasets from /app/input and save the completed dataset as /app/output/tr.csv.
prompts/sdtm-tr-tumor-measurements/conventions.md
Edit12 linesFollowing CDISC SDTM standards, use the provided TR_RAW dataset to create a TR dataset with one record per collected tumor assessment.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, TRSEQ, TRLNKID, TRTESTCD, TRTEST, TRORRES, TRORRESU, TRSTRESC, TRSTRESN, TRSTRESU, TRSTAT, TRMETHOD, TREVAL, VISITNUM, TRDTC
DOMAIN is "TR". TRTESTCD is "DIAMETER" or "TUMSTATE", and TRTEST is "Diameter" or "Tumor State". TRSTAT is "NOT DONE" or has no value.
Read the source datasets from /app/input and save the completed dataset as /app/output/tr.csv.
prompts/sdtm-tr-tumor-measurements/brief.md
Edit9 linesFollowing CDISC SDTM standards, use the provided TR_RAW dataset to create a TR dataset with one record per collected tumor assessment.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, TRSEQ, TRLNKID, TRTESTCD, TRTEST, TRORRES, TRORRESU, TRSTRESC, TRSTRESN, TRSTRESU, TRSTAT, TRMETHOD, TREVAL, VISITNUM, TRDTC
Read the source datasets from /app/input and save the completed dataset as /app/output/tr.csv.
R
Edit3 lineslibrary(yamaa)yamaa_spec("spec.yaml")$run()$outputPython
Edit3 linesimport yamaayamaa.spec("spec.yaml").run().outputsolutions/sdtm-tr-tumor-measurements/result.R
Edit61 lines# Reference solution for the yamaa benchmark sdtm-tr-tumor-measurements (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)fmt_num <- function(x) { if (is.na(x)) { return(NA_character_) } if (x == floor(x)) { formatC(x, format = "f", digits = 0) } else { formatted <- format(x, scientific = FALSE, trim = TRUE, digits = 15) sub("\\.0+$", "", formatted) }}raw <- read_csv( "/app/input/tr_raw.csv", col_types = cols( TRSEQ = col_integer(), VISITNUM = col_integer(), .default = col_character() ))tr <- raw |> mutate( orres_num = suppressWarnings(as.numeric(TRORRES)), mm_value = if_else(TRORRESU == "cm", orres_num * 10, orres_num), TRSTRESN = case_when( TRTESTCD == "DIAMETER" & TRORRES == "TOO SMALL TO MEASURE" ~ 5, TRTESTCD == "DIAMETER" & !is.na(TRORRES) & TRORRES != "" & TRORRES != "TOO SMALL TO MEASURE" ~ mm_value, .default = NA_real_ ), TRSTRESC = case_when( TRTESTCD == "DIAMETER" & TRORRES == "TOO SMALL TO MEASURE" ~ "5", TRTESTCD == "DIAMETER" & !is.na(TRORRES) & TRORRES != "" & TRORRES != "TOO SMALL TO MEASURE" ~ vapply( TRSTRESN, fmt_num, character(1) ), TRTESTCD != "DIAMETER" & !is.na(TRORRES) & TRORRES != "" ~ TRORRES, .default = NA_character_ ), DOMAIN = "TR", TRSTRESU = if_else(!is.na(TRSTRESN), "mm", NA_character_) ) |> arrange(USUBJID, TRSEQ) |> select( DOMAIN, STUDYID, USUBJID, TRSEQ, TRLNKID, TRTESTCD, TRTEST, TRORRES, TRORRESU, TRSTRESC, TRSTRESN, TRSTRESU, TRSTAT, TRMETHOD, TREVAL, VISITNUM, TRDTC )dir.create("/app/output", showWarnings = FALSE)write_csv(tr, "/app/output/tr.csv", na = "")solutions/sdtm-tr-tumor-measurements/result.py
Edit95 lines# Reference solution for the yamaa benchmark sdtm-tr-tumor-measurements (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1."""Build the SDTM TR dataset from TR_RAW.One record per collected tumor assessment. Diameter results arestandardized to mm: a value collected in mm keeps its value, a valuecollected in cm is multiplied by 10, and a lesion too small to measuretakes 5. Lesion states are kept as collected. A lesion that was notassessed carries TRSTAT "NOT DONE" and no standardized result."""import csvimport osINPUT_DIR = "/app/input"OUTPUT_DIR = "/app/output"TOO_SMALL = "TOO SMALL TO MEASURE"TOO_SMALL_MM = 5.0OUT_COLUMNS = [ "DOMAIN", "STUDYID", "USUBJID", "TRSEQ", "TRLNKID", "TRTESTCD", "TRTEST", "TRORRES", "TRORRESU", "TRSTRESC", "TRSTRESN", "TRSTRESU", "TRSTAT", "TRMETHOD", "TREVAL", "VISITNUM", "TRDTC",]def fmt_number(value): """Write a float without a trailing .0 or float artifacts.""" if value == int(value): return str(int(value)) return repr(value)def main(): with open(os.path.join(INPUT_DIR, "tr_raw.csv"), newline="") as fh: rows = list(csv.DictReader(fh)) out_rows = [] for row in rows: testcd = row["TRTESTCD"] orres = row["TRORRES"] orresu = row["TRORRESU"] stat = row["TRSTAT"] stresn = "" stresc = "" if testcd == "DIAMETER": if orres == TOO_SMALL: stresn = fmt_number(TOO_SMALL_MM) stresc = stresn elif orres != "": value = float(orres) if orresu == "cm": value = value * 10 stresn = fmt_number(value) stresc = stresn elif testcd == "TUMSTATE": stresc = orres stresu = "mm" if stresn != "" else "" out_rows.append({ "DOMAIN": "TR", "STUDYID": row["STUDYID"], "USUBJID": row["USUBJID"], "TRSEQ": row["TRSEQ"], "TRLNKID": row["TRLNKID"], "TRTESTCD": testcd, "TRTEST": row["TRTEST"], "TRORRES": orres, "TRORRESU": orresu, "TRSTRESC": stresc, "TRSTRESN": stresn, "TRSTRESU": stresu, "TRSTAT": stat, "TRMETHOD": row["TRMETHOD"], "TREVAL": row["TREVAL"], "VISITNUM": row["VISITNUM"], "TRDTC": row["TRDTC"], }) os.makedirs(OUTPUT_DIR, exist_ok=True) with open(os.path.join(OUTPUT_DIR, "tr.csv"), "w", newline="") as fh: writer = csv.DictWriter(fh, fieldnames=OUT_COLUMNS) writer.writeheader() writer.writerows(out_rows)if __name__ == "__main__": main()solutions/sdtm-tr-tumor-measurements/result.sas
Edit19 lines/* opensas reference solution for the yamaa benchmark sdtm-tr-tumor-measurements. Reads the benchmark inputs and derives the requested output. Execute with the existing opensas runtime: opensas /app/output/result.sas. */data tr_raw; length STUDYID USUBJID TRLNKID TRTESTCD TRTEST TRORRES TRORRESU TRSTAT TRMETHOD TREVAL TRDTC $1024; infile "/app/input/tr_raw.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. TRSEQ TRLNKID : $char1024. TRTESTCD : $char1024. TRTEST : $char1024. TRORRES : $char1024. TRORRESU : $char1024. TRSTAT : $char1024. TRMETHOD : $char1024. TREVAL : $char1024. VISITNUM TRDTC : $char1024.;run;data result; set tr_raw; length DOMAIN $2 TRSTRESC TRSTRESU $1024; DOMAIN='TR'; TRSTRESN=.; TRSTRESC=''; TRSTRESU='';if TRSTAT='NOT DONE' or missing(TRORRES) then do; TRSTAT='NOT DONE'; TRORRES=''; TRORRESU=''; TRDTC=''; end;else if TRTESTCD='TUMSTATE' then TRSTRESC=TRORRES;else do; if TRORRES='TOO SMALL TO MEASURE' then TRSTRESN=5; else do; TRSTRESN=input(TRORRES,?? best32.); if TRORRESU='cm' then TRSTRESN=TRSTRESN*10; end;if not missing(TRSTRESN) then do; TRSTRESC=strip(put(TRSTRESN,best32.)); TRSTRESU='mm'; end; end; run;proc sql; create table final as select DOMAIN, STUDYID, USUBJID, TRSEQ, TRLNKID, TRTESTCD, TRTEST, TRORRES, TRORRESU, TRSTRESC, TRSTRESN, TRSTRESU, TRSTAT, TRMETHOD, TREVAL, VISITNUM, TRDTC from result;quit;proc export data=final outfile="/app/output/tr.csv" dbms=csv replace; run;