yamaa As a clinical trial statistician or programmer, challenge anything in this benchmark: describe a problem in Comments, or propose a fix with the Edit button on any source file.
- Task: Would a real study need this derivation, specified this way? Does it follow CDISC SDTM: variable names, controlled terms, record structure? Flag any rule that is ambiguous or that a sponsor would handle differently.
- Inputs: Do they look like data collected in a real trial, and do they include the situations you would expect to meet in one?
- Expected output: Would independent QC programming reproduce every value from these inputs and rules?
The other tabs beside Comments are background for assessment, not under review.
Summary
EditUnder reviewWhat this benchmark meansREADME.mdGoal: build one record per identified tumor or lesion, carrying
the link identifier, the identification test and result, the
anatomical location, the identification method, the evaluator, and
the visit: TULNKID, TUTESTCD, TUTEST, TUORRES, TUSTRESC,
TULOC, TULAT, TUMETHOD, TUEVAL, VISITNUM, and TUDTC.
Input: collected lesion identification with one row per lesion,
carrying the collected lesion category, lesion number, location,
laterality, method, evaluator, visit, and identification date, plus
study, subject, and sequence (TUSEQ) identifiers.
Variables:
TUSEQnumbers a subject's identification records in collection order, starting at 1.TULNKIDis the link identifier tying the lesion to its assessments in the tumor results (TR) domain:Tplus the lesion number for a target lesion,NTfor a non-target lesion,NEWfor a new lesion.TUTESTCDisTUMIDENTandTUTESTisTumor Identificationon every record.TUORRESis the lesion category as collected, standardized toTARGET,NON-TARGET, orNEW.TUSTRESCrepeats the standardized category.TULOCis the anatomical location as collected, standardized to uppercase.TULATis the laterality as collected, standardized toLEFTorRIGHT; empty when the site recorded none.TUMETHODis the identification method as collected.TUEVALnames the evaluator as collected.VISITNUMis the visit number as collected.TUDTCis the date of the scan or examination that identified the lesion, as collected.
Note: a lymph-node target lesion is identified like any other
target lesion: it keeps a T link identifier and its category stays
TARGET; its location names it as a lymph node, and the results
domain measures its short axis. A new lesion is identified once, at
the visit where it first appears, and earlier visits keep no record
of it.
Inputs
Under review1 source fileinput/tu_raw.csv
Edit6 rows| STUDYID | USUBJID | TUSEQ | LESION_CATEGORY | LESION_NUM | LOCATION | LATERALITY | METHOD | EVALUATOR | VISITNUM | VISIT | TUDTC |
|---|---|---|---|---|---|---|---|---|---|---|---|
| YAMAA-01 | 1 | Target | 01 | Liver | Right | CT SCAN | INVESTIGATOR | 1 | SCREENING | 2026-01-05 | |
| YAMAA-01 | 2 | Target | 02 | Mediastinal lymph node | — | CT SCAN | INVESTIGATOR | 1 | SCREENING | 2026-01-05 | |
| YAMAA-01 | 3 | Non-target | 01 | Lung | Left | CT SCAN | INVESTIGATOR | 1 | SCREENING | 2026-01-05 | |
| YAMAA-01 | 4 | New | 01 | Adrenal gland | Right | CT SCAN | INVESTIGATOR | 4 | WEEK 12 | 2026-03-30 | |
| YAMAA-01 | 1 | Target | 01 | Liver | Left | CT SCAN | INVESTIGATOR | 1 | SCREENING | 2026-01-12 | |
| YAMAA-01 | 2 | Non-target | 01 | Bone | — | CT SCAN | INVESTIGATOR | 1 | SCREENING | 2026-01-12 |
Expected output
Under review6 expected rowsexpected/tu.csv
Edit6 rows| DOMAIN | STUDYID | USUBJID | TUSEQ | TULNKID | TUTESTCD | TUTEST | TUORRES | TUSTRESC | TULOC | TULAT | TUMETHOD | TUEVAL | VISITNUM | VISIT | TUDTC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| TU | YAMAA-01 | 1 | T01 | TUMIDENT | Tumor Identification | TARGET | TARGET | LIVER | RIGHT | CT SCAN | INVESTIGATOR | 1 | SCREENING | 2026-01-05 | |
| TU | YAMAA-01 | 2 | T02 | TUMIDENT | Tumor Identification | TARGET | TARGET | MEDIASTINAL LYMPH NODE | — | CT SCAN | INVESTIGATOR | 1 | SCREENING | 2026-01-05 | |
| TU | YAMAA-01 | 3 | NT01 | TUMIDENT | Tumor Identification | NON-TARGET | NON-TARGET | LUNG | LEFT | CT SCAN | INVESTIGATOR | 1 | SCREENING | 2026-01-05 | |
| TU | YAMAA-01 | 4 | NEW01 | TUMIDENT | Tumor Identification | NEW | NEW | ADRENAL GLAND | RIGHT | CT SCAN | INVESTIGATOR | 4 | WEEK 12 | 2026-03-30 | |
| TU | YAMAA-01 | 1 | T01 | TUMIDENT | Tumor Identification | TARGET | TARGET | LIVER | LEFT | CT SCAN | INVESTIGATOR | 1 | SCREENING | 2026-01-12 | |
| TU | YAMAA-01 | 2 | NT01 | TUMIDENT | Tumor Identification | NON-TARGET | NON-TARGET | BONE | — | CT SCAN | INVESTIGATOR | 1 | SCREENING | 2026-01-12 |
Comments and assessment materials
| Variable Name | Variable Label | Type | Length | Controlled Terms or Format | Origin | Core | Conversion Definition | Variable Type | Variable Order | Comments for Define |
|---|---|---|---|---|---|---|---|---|---|---|
| DOMAIN | Domain Abbreviation | Char | Assigned | Set constant value "TU". | SDTM | 1 | ||||
| STUDYID | Study Identifier | Char | Collected | Copy value from TU_RAW.STUDYID. | SDTM | 2 | ||||
| USUBJID | Unique Subject Identifier | Char | Collected | Copy value from TU_RAW.USUBJID. | SDTM | 3 | ||||
| TUSEQ | Sequence Number | Num | Collected | Copy value from TU_RAW.TUSEQ. | SDTM | 4 | ||||
| TULNKID | Link ID | Char | Derived | If TU_RAW.LESION_CATEGORY = 'Target' then fill the template "T{TU_RAW.LESION_NUM}"; If TU_RAW.LESION_CATEGORY = 'Non-target' then fill the template "NT{TU_RAW.LESION_NUM}"; If TU_RAW.LESION_CATEGORY = 'New' then fill the template "NEW{TU_RAW.LESION_NUM}"; otherwise blank. | SDTM | 5 | ||||
| TUTESTCD | Tumor Identification Test Short Name | Char | TUMIDENT | Assigned | Set constant value "TUMIDENT". | SDTM | 6 | |||
| TUTEST | Tumor Identification Test Name | Char | Assigned | Set constant value "Tumor Identification". | SDTM | 7 | ||||
| TUORRES | Tumor Identification Result | Char | TARGET, NON-TARGET, NEW | Derived | Convert TU_RAW.LESION_CATEGORY to upper case. | SDTM | 8 | |||
| TUSTRESC | Tumor Identification Result in Std Format | Char | Derived | Copy value from TUORRES. | SDTM | 9 | ||||
| TULOC | Location of the Tumor | Char | Derived | Convert TU_RAW.LOCATION to upper case. | SDTM | 10 | ||||
| TULAT | Laterality | Char | Derived | Convert TU_RAW.LATERALITY to upper case; a missing source → blank. | SDTM | 11 | ||||
| TUMETHOD | Method of Identification | Char | Collected | Copy value from TU_RAW.METHOD. | SDTM | 12 | ||||
| TUEVAL | Evaluator | Char | Collected | Copy value from TU_RAW.EVALUATOR. | SDTM | 13 | ||||
| VISITNUM | Visit Number | Num | Collected | Copy value from TU_RAW.VISITNUM. | SDTM | 14 | ||||
| VISIT | Visit Name | Char | Collected | Copy value from TU_RAW.VISIT. | SDTM | 15 | ||||
| TUDTC | Date/Time of Tumor Identification | Char | ISO 8601 | Collected | Copy value from TU_RAW.TUDTC. | SDTM | 16 |
| Variable | Permitted values |
|---|---|
| TUTESTCD | TUMIDENT |
| TUORRES | TARGET, NON-TARGET, NEW |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/sdtm-tu-tumor-identification/full.md
Edit27 linesFollowing CDISC SDTM standards, use the provided TU_RAW dataset to create a TU dataset with one record per identified tumor or lesion.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, TUSEQ, TULNKID, TUTESTCD, TUTEST, TUORRES, TUSTRESC, TULOC, TULAT, TUMETHOD, TUEVAL, VISITNUM, VISIT, TUDTC
TUSEQ numbers a subject's identification records in collection order, starting at 1. TULNKID ties the lesion to its assessments in the tumor results domain: T plus the lesion number for a target lesion, NT plus the number for a non-target lesion, and NEW plus the number for a new lesion, the number written in two digits as collected (target lesion 3 is T03). TUTESTCD is TUMIDENT and TUTEST is Tumor Identification on every record. TUORRES is the lesion category as collected, standardized to TARGET, NON-TARGET, or NEW; TUSTRESC repeats the standardized category. TULOC is the anatomical location as collected, standardized to uppercase. TULAT is the laterality as collected, standardized to LEFT or RIGHT, and has no value when the site recorded none. TUMETHOD, TUEVAL, VISITNUM, and TUDTC are as collected.
A lymph-node target lesion is identified like any other target lesion: it keeps a T link identifier and its category stays TARGET, while its location names it as a lymph node. A new lesion is identified once, at the visit where it first appears.
Read the source datasets from /app/input and save the completed dataset as /app/output/tu.csv.
prompts/sdtm-tu-tumor-identification/conventions.md
Edit16 linesFollowing CDISC SDTM standards, use the provided TU_RAW dataset to create a TU dataset with one record per identified tumor or lesion.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, TUSEQ, TULNKID, TUTESTCD, TUTEST, TUORRES, TUSTRESC, TULOC, TULAT, TUMETHOD, TUEVAL, VISITNUM, VISIT, TUDTC
TULNKID is T plus the lesion number for a target lesion, NT plus the number for a non-target lesion, and NEW plus the number for a new lesion, the number written in two digits as collected (target lesion 3 is T03). TUTESTCD is TUMIDENT and TUTEST is Tumor Identification on every record. TUORRES is standardized to TARGET, NON-TARGET, or NEW. TULOC is standardized to uppercase. TULAT is standardized to LEFT or RIGHT, or has no value.
Read the source datasets from /app/input and save the completed dataset as /app/output/tu.csv.
prompts/sdtm-tu-tumor-identification/brief.md
Edit9 linesFollowing CDISC SDTM standards, use the provided TU_RAW dataset to create a TU dataset with one record per identified tumor or lesion.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, TUSEQ, TULNKID, TUTESTCD, TUTEST, TUORRES, TUSTRESC, TULOC, TULAT, TUMETHOD, TUEVAL, VISITNUM, VISIT, TUDTC
Read the source datasets from /app/input and save the completed dataset as /app/output/tu.csv.
run.R
Edit4 lineslibrary(yamaa)tu <- yamaa_domain("spec.yaml")$outputturun.py
Edit4 linesimport yamaatu = yamaa.yamaa_domain("spec.yaml").outputtusolutions/sdtm-tu-tumor-identification/result.R
Edit48 lines# Reference solution for the yamaa benchmark sdtm-tu-tumor-identification (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)library(stringr)raw <- read_csv( "/app/input/tu_raw.csv", col_types = cols( TUSEQ = col_integer(), VISITNUM = col_integer(), .default = col_character() ))tu <- raw |> mutate( DOMAIN = "TU", TULNKID = case_when( LESION_CATEGORY == "Target" ~ paste0("T", LESION_NUM), LESION_CATEGORY == "Non-target" ~ paste0("NT", LESION_NUM), .default = paste0("NEW", LESION_NUM) ), TUTESTCD = "TUMIDENT", TUTEST = "Tumor Identification", TUORRES = case_when( LESION_CATEGORY == "Target" ~ "TARGET", LESION_CATEGORY == "Non-target" ~ "NON-TARGET", .default = "NEW" ), TUSTRESC = TUORRES, TULOC = str_to_upper(LOCATION), TULAT = str_to_upper(LATERALITY), TUMETHOD = METHOD, TUEVAL = EVALUATOR ) |> arrange(USUBJID, TUSEQ) |> select( DOMAIN, STUDYID, USUBJID, TUSEQ, TULNKID, TUTESTCD, TUTEST, TUORRES, TUSTRESC, TULOC, TULAT, TUMETHOD, TUEVAL, VISITNUM, VISIT, TUDTC )dir.create("/app/output", showWarnings = FALSE)write_csv(tu, "/app/output/tu.csv", na = "")solutions/sdtm-tu-tumor-identification/result.py
Edit66 lines# Reference solution for the yamaa benchmark sdtm-tu-tumor-identification (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.from pathlib import Pathimport polars as plraw = pl.read_csv("/app/input/tu_raw.csv", infer_schema=False).with_columns( pl.col("TUSEQ").cast(pl.Int64), pl.col("VISITNUM").cast(pl.Int64),)prefix = ( pl.when(pl.col("LESION_CATEGORY") == "Target") .then(pl.lit("T")) .when(pl.col("LESION_CATEGORY") == "Non-target") .then(pl.lit("NT")) .otherwise(pl.lit("NEW")))category = ( pl.when(pl.col("LESION_CATEGORY") == "Target") .then(pl.lit("TARGET")) .when(pl.col("LESION_CATEGORY") == "Non-target") .then(pl.lit("NON-TARGET")) .otherwise(pl.lit("NEW")))tu = ( raw.with_columns( DOMAIN=pl.lit("TU"), TULNKID=prefix + pl.col("LESION_NUM"), TUTESTCD=pl.lit("TUMIDENT"), TUTEST=pl.lit("Tumor Identification"), TUORRES=category, TUSTRESC=category, TULOC=pl.col("LOCATION").str.to_uppercase(), TULAT=pl.col("LATERALITY").str.to_uppercase(), TUMETHOD=pl.col("METHOD"), TUEVAL=pl.col("EVALUATOR"), ) .sort(["USUBJID", "TUSEQ"]) .select( "DOMAIN", "STUDYID", "USUBJID", "TUSEQ", "TULNKID", "TUTESTCD", "TUTEST", "TUORRES", "TUSTRESC", "TULOC", "TULAT", "TUMETHOD", "TUEVAL", "VISITNUM", "VISIT", "TUDTC", ))Path("/app/output").mkdir(exist_ok=True)tu.write_csv("/app/output/tu.csv")solutions/sdtm-tu-tumor-identification/result.sas
Edit16 lines/* opensas reference solution for the yamaa benchmark sdtm-tu-tumor-identification. Reads the benchmark inputs and derives the requested output. Execute with the existing opensas runtime: opensas /app/output/result.sas. */data tu_raw; length STUDYID USUBJID LESION_CATEGORY LESION_NUM LOCATION LATERALITY METHOD EVALUATOR VISIT TUDTC $1024; infile "/app/input/tu_raw.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. TUSEQ LESION_CATEGORY : $char1024. LESION_NUM : $char1024. LOCATION : $char1024. LATERALITY : $char1024. METHOD : $char1024. EVALUATOR : $char1024. VISITNUM VISIT : $char1024. TUDTC : $char1024.;run;data result; set tu_raw; length DOMAIN $2 TULNKID TUTESTCD TUTEST TUORRES TUSTRESC TULOC TULAT TUMETHOD TUEVAL PREFIX $1024;DOMAIN='TU'; TUORRES=upcase(LESION_CATEGORY); TUSTRESC=TUORRES; TUTESTCD='TUMIDENT'; TUTEST='Tumor Identification'; TULOC=upcase(LOCATION); TULAT=upcase(LATERALITY); TUMETHOD=METHOD; TUEVAL=EVALUATOR; PREFIX='NEW'; if TUORRES='TARGET' then PREFIX='T'; if TUORRES='NON-TARGET' then PREFIX='NT'; TULNKID=cats(PREFIX,put(input(LESION_NUM,best32.),z2.)); run;proc sql; create table final as select DOMAIN, STUDYID, USUBJID, TUSEQ, TULNKID, TUTESTCD, TUTEST, TUORRES, TUSTRESC, TULOC, TULAT, TUMETHOD, TUEVAL, VISITNUM, VISIT, TUDTC from result;quit;proc export data=final outfile="/app/output/tu.csv" dbms=csv replace; run;
Comments
Signed in with GitHub; saved in repository Discussions