yamaa As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.
We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.
Summary
EditUnder reviewWhat this benchmark meansREADME.mdPublic entries: Python and R. Passing coverage requires an installed inventory.
Goal: build one record per identified tumor or lesion, carrying
the link identifier, the identification test and result, the
anatomical location, the identification method, the evaluator, and
the visit: TULNKID, TUTESTCD, TUTEST, TUORRES, TUSTRESC,
TULOC, TULAT, TUMETHOD, TUEVAL, VISITNUM, and TUDTC.
Input: collected lesion identification with one row per lesion,
carrying the collected lesion category, lesion number, location,
laterality, method, evaluator, visit, and identification date, plus
study, subject, and sequence (TUSEQ) identifiers.
Variables:
TUSEQnumbers a subject's identification records in collection order, starting at 1.TULNKIDis the link identifier tying the lesion to its assessments in the tumor results (TR) domain:Tplus the lesion number for a target lesion,NTfor a non-target lesion,NEWfor a new lesion.TUTESTCDisTUMIDENTandTUTESTisTumor Identificationon every record.TUORRESis the lesion category as collected, standardized toTARGET,NON-TARGET, orNEW.TUSTRESCrepeats the standardized category.TULOCis the anatomical location as collected, standardized to uppercase.TULATis the laterality as collected, standardized toLEFTorRIGHT; empty when the site recorded none.TUMETHODis the identification method as collected.TUEVALnames the evaluator as collected.VISITNUMis the visit number as collected.TUDTCis the date of the scan or examination that identified the lesion, as collected.
Note: a lymph-node target lesion is identified like any other
target lesion: it keeps a T link identifier and its category stays
TARGET; its location names it as a lymph node, and the results
domain measures its short axis. A new lesion is identified once, at
the visit where it first appears, and earlier visits keep no record
of it.
Inputs
Under review1 source fileinput/tu_raw.csv
Edit6 rows| STUDYID | USUBJID | TUSEQ | LESION_CATEGORY | LESION_NUM | LOCATION | LATERALITY | METHOD | EVALUATOR | VISITNUM | VISIT | TUDTC |
|---|---|---|---|---|---|---|---|---|---|---|---|
| YAMAA-01 | 1 | Target | 01 | Liver | Right | CT SCAN | INVESTIGATOR | 1 | SCREENING | 2026-01-05 | |
| YAMAA-01 | 2 | Target | 02 | Mediastinal lymph node | — | CT SCAN | INVESTIGATOR | 1 | SCREENING | 2026-01-05 | |
| YAMAA-01 | 3 | Non-target | 01 | Lung | Left | CT SCAN | INVESTIGATOR | 1 | SCREENING | 2026-01-05 | |
| YAMAA-01 | 4 | New | 01 | Adrenal gland | Right | CT SCAN | INVESTIGATOR | 4 | WEEK 12 | 2026-03-30 | |
| YAMAA-01 | 1 | Target | 01 | Liver | Left | CT SCAN | INVESTIGATOR | 1 | SCREENING | 2026-01-12 | |
| YAMAA-01 | 2 | Non-target | 01 | Bone | — | CT SCAN | INVESTIGATOR | 1 | SCREENING | 2026-01-12 |
Expected output
Under review6 expected rowsexpected/tu.csv
Edit6 rows| DOMAIN | STUDYID | USUBJID | TUSEQ | TULNKID | TUTESTCD | TUTEST | TUORRES | TUSTRESC | TULOC | TULAT | TUMETHOD | TUEVAL | VISITNUM | VISIT | TUDTC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| TU | YAMAA-01 | 1 | T01 | TUMIDENT | Tumor Identification | TARGET | TARGET | LIVER | RIGHT | CT SCAN | INVESTIGATOR | 1 | SCREENING | 2026-01-05 | |
| TU | YAMAA-01 | 2 | T02 | TUMIDENT | Tumor Identification | TARGET | TARGET | MEDIASTINAL LYMPH NODE | — | CT SCAN | INVESTIGATOR | 1 | SCREENING | 2026-01-05 | |
| TU | YAMAA-01 | 3 | NT01 | TUMIDENT | Tumor Identification | NON-TARGET | NON-TARGET | LUNG | LEFT | CT SCAN | INVESTIGATOR | 1 | SCREENING | 2026-01-05 | |
| TU | YAMAA-01 | 4 | NEW01 | TUMIDENT | Tumor Identification | NEW | NEW | ADRENAL GLAND | RIGHT | CT SCAN | INVESTIGATOR | 4 | WEEK 12 | 2026-03-30 | |
| TU | YAMAA-01 | 1 | T01 | TUMIDENT | Tumor Identification | TARGET | TARGET | LIVER | LEFT | CT SCAN | INVESTIGATOR | 1 | SCREENING | 2026-01-12 | |
| TU | YAMAA-01 | 2 | NT01 | TUMIDENT | Tumor Identification | NON-TARGET | NON-TARGET | BONE | — | CT SCAN | INVESTIGATOR | 1 | SCREENING | 2026-01-12 |
Human review and assessment
| Variable Name | Variable Label | Type | Length | Controlled Terms or Format | Origin | Core | Conversion Definition | Variable Type | Variable Order | Comments for Define |
|---|---|---|---|---|---|---|---|---|---|---|
| DOMAIN | Domain Abbreviation | Char | Assigned | Set constant value "TU". | SDTM | 1 | ||||
| STUDYID | Study Identifier | Char | Collected | Copy value from TU_RAW.STUDYID. | SDTM | 2 | ||||
| USUBJID | Unique Subject Identifier | Char | Collected | Copy value from TU_RAW.USUBJID. | SDTM | 3 | ||||
| TUSEQ | Sequence Number | Num | Collected | Copy value from TU_RAW.TUSEQ. | SDTM | 4 | ||||
| TULNKID | Link ID | Char | Derived | If TU_RAW.LESION_CATEGORY = 'Target' then fill the template "T{TU_RAW.LESION_NUM}"; If TU_RAW.LESION_CATEGORY = 'Non-target' then fill the template "NT{TU_RAW.LESION_NUM}"; If TU_RAW.LESION_CATEGORY = 'New' then fill the template "NEW{TU_RAW.LESION_NUM}"; otherwise blank. | SDTM | 5 | ||||
| TUTESTCD | Tumor Identification Test Short Name | Char | TUMIDENT | Assigned | Set constant value "TUMIDENT". | SDTM | 6 | |||
| TUTEST | Tumor Identification Test Name | Char | Assigned | Set constant value "Tumor Identification". | SDTM | 7 | ||||
| TUORRES | Tumor Identification Result | Char | TARGET, NON-TARGET, NEW | Derived | Convert TU_RAW.LESION_CATEGORY to upper case. | SDTM | 8 | |||
| TUSTRESC | Tumor Identification Result in Std Format | Char | Derived | Copy value from TUORRES. | SDTM | 9 | ||||
| TULOC | Location of the Tumor | Char | Derived | Convert TU_RAW.LOCATION to upper case. | SDTM | 10 | ||||
| TULAT | Laterality | Char | Derived | Convert TU_RAW.LATERALITY to upper case. | SDTM | 11 | ||||
| TUMETHOD | Method of Identification | Char | Collected | Copy value from TU_RAW.METHOD. | SDTM | 12 | ||||
| TUEVAL | Evaluator | Char | Collected | Copy value from TU_RAW.EVALUATOR. | SDTM | 13 | ||||
| VISITNUM | Visit Number | Num | Collected | Copy value from TU_RAW.VISITNUM. | SDTM | 14 | ||||
| VISIT | Visit Name | Char | Collected | Copy value from TU_RAW.VISIT. | SDTM | 15 | ||||
| TUDTC | Date/Time of Tumor Identification | Char | ISO 8601 | Collected | Copy value from TU_RAW.TUDTC. | SDTM | 16 |
| Variable | Permitted values |
|---|---|
| TUTESTCD | TUMIDENT |
| TUORRES | TARGET, NON-TARGET, NEW |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.7 | 2026-10-20 | yamaa docs build | Read a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default | ||
| 1.6 | 2026-10-17 | yamaa docs build | Describe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight | ||
| 1.5 | 2026-10-17 | yamaa docs build | Name collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part | ||
| 1.4 | 2026-10-15 | yamaa docs build | State datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks | ||
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/sdtm-tu-tumor-identification/full.md
Edit30 linesFollowing CDISC SDTM standards, use the provided TU_RAW dataset to create a TU dataset with one record per identified tumor or lesion.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, TUSEQ, TULNKID, TUTESTCD, TUTEST, TUORRES, TUSTRESC, TULOC, TULAT, TUMETHOD, TUEVAL, VISITNUM, VISIT, TUDTC
DOMAIN is "TU" on every record. TUSEQ numbers a subject's identification records in collection order, starting at 1. TULNKID is the link identifier tying the lesion to its assessments in the tumor results (TR) domain: "T" plus the lesion number for a target lesion, "NT" plus the lesion number for a non-target lesion, and "NEW" plus the lesion number for a new lesion. TUTESTCD is "TUMIDENT" and TUTEST is "Tumor Identification" on every record. TUORRES is the lesion category as collected, standardized to "TARGET", "NON-TARGET", or "NEW". TUSTRESC repeats the standardized category in TUORRES. TULOC is the anatomical location as collected, standardized to uppercase. TULAT is the laterality as collected, standardized to "LEFT" or "RIGHT", and has no value when the site recorded no laterality. TUMETHOD is the identification method as collected. TUEVAL names the evaluator as collected. VISITNUM and VISIT are the visit number and visit name as collected. TUDTC is the date of the scan or examination that identified the lesion, as collected. A lymph-node target lesion is identified like any other target lesion: it keeps a "T" link identifier, its category stays "TARGET", and its location names it as a lymph node. A new lesion is identified once, at the visit where it first appears, and earlier visits keep no record of it.
Read the source datasets from /app/input and save the completed dataset as /app/output/tu.csv.
prompts/sdtm-tu-tumor-identification/conventions.md
Edit17 linesFollowing CDISC SDTM standards, use the provided TU_RAW dataset to create a TU dataset with one record per identified tumor or lesion.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, TUSEQ, TULNKID, TUTESTCD, TUTEST, TUORRES, TUSTRESC, TULOC, TULAT, TUMETHOD, TUEVAL, VISITNUM, VISIT, TUDTC
DOMAIN is "TU" on every record. TULNKID is "T" plus the lesion number for a target lesion, "NT" plus the lesion number for a non-target lesion, and "NEW" plus the lesion number for a new lesion. TUTESTCD is "TUMIDENT" and TUTEST is "Tumor Identification" on every record. TUORRES is "TARGET", "NON-TARGET", or "NEW". TULOC is the anatomical location as collected, standardized to uppercase. TULAT is "LEFT" or "RIGHT", or has no value.
Read the source datasets from /app/input and save the completed dataset as /app/output/tu.csv.
prompts/sdtm-tu-tumor-identification/brief.md
Edit9 linesFollowing CDISC SDTM standards, use the provided TU_RAW dataset to create a TU dataset with one record per identified tumor or lesion.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, TUSEQ, TULNKID, TUTESTCD, TUTEST, TUORRES, TUSTRESC, TULOC, TULAT, TUMETHOD, TUEVAL, VISITNUM, VISIT, TUDTC
Read the source datasets from /app/input and save the completed dataset as /app/output/tu.csv.
R
Edit3 lineslibrary(yamaa)yamaa_spec("spec.yaml")$run()$outputPython
Edit3 linesimport yamaayamaa.spec("spec.yaml").run().outputsolutions/sdtm-tu-tumor-identification/result.R
Edit48 lines# Reference solution for the yamaa benchmark sdtm-tu-tumor-identification (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)library(stringr)raw <- read_csv( "/app/input/tu_raw.csv", col_types = cols( TUSEQ = col_integer(), VISITNUM = col_integer(), .default = col_character() ))tu <- raw |> mutate( DOMAIN = "TU", TULNKID = case_when( LESION_CATEGORY == "Target" ~ paste0("T", LESION_NUM), LESION_CATEGORY == "Non-target" ~ paste0("NT", LESION_NUM), .default = paste0("NEW", LESION_NUM) ), TUTESTCD = "TUMIDENT", TUTEST = "Tumor Identification", TUORRES = case_when( LESION_CATEGORY == "Target" ~ "TARGET", LESION_CATEGORY == "Non-target" ~ "NON-TARGET", .default = "NEW" ), TUSTRESC = TUORRES, TULOC = str_to_upper(LOCATION), TULAT = str_to_upper(LATERALITY), TUMETHOD = METHOD, TUEVAL = EVALUATOR ) |> arrange(USUBJID, TUSEQ) |> select( DOMAIN, STUDYID, USUBJID, TUSEQ, TULNKID, TUTESTCD, TUTEST, TUORRES, TUSTRESC, TULOC, TULAT, TUMETHOD, TUEVAL, VISITNUM, VISIT, TUDTC )dir.create("/app/output", showWarnings = FALSE)write_csv(tu, "/app/output/tu.csv", na = "")solutions/sdtm-tu-tumor-identification/result.py
Edit94 lines# Reference solution for the yamaa benchmark sdtm-tu-tumor-identification (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1."""Build the SDTM TU dataset from TU_RAW.One record per identified tumor or lesion. Reads /app/input/tu_raw.csvand writes /app/output/tu.csv."""import csvimport osINPUT_PATH = "/app/input/tu_raw.csv"OUTPUT_PATH = "/app/output/tu.csv"CATEGORY_MAP = { "target": "TARGET", "non-target": "NON-TARGET", "new": "NEW",}LINK_PREFIX = { "TARGET": "T", "NON-TARGET": "NT", "NEW": "NEW",}COLUMNS = [ "DOMAIN", "STUDYID", "USUBJID", "TUSEQ", "TULNKID", "TUTESTCD", "TUTEST", "TUORRES", "TUSTRESC", "TULOC", "TULAT", "TUMETHOD", "TUEVAL", "VISITNUM", "VISIT", "TUDTC",]def main(): with open(INPUT_PATH, newline="", encoding="utf-8") as f: rows = list(csv.DictReader(f)) out = [] seq_by_subject = {} for row in rows: subj = row["USUBJID"] seq_by_subject[subj] = seq_by_subject.get(subj, 0) + 1 category = CATEGORY_MAP[row["LESION_CATEGORY"].strip().lower()] link_id = LINK_PREFIX[category] + row["LESION_NUM"].strip() laterality = row["LATERALITY"].strip() out.append( { "DOMAIN": "TU", "STUDYID": row["STUDYID"], "USUBJID": subj, "TUSEQ": seq_by_subject[subj], "TULNKID": link_id, "TUTESTCD": "TUMIDENT", "TUTEST": "Tumor Identification", "TUORRES": category, "TUSTRESC": category, "TULOC": row["LOCATION"].strip().upper(), "TULAT": laterality.upper() if laterality else "", "TUMETHOD": row["METHOD"], "TUEVAL": row["EVALUATOR"], "VISITNUM": row["VISITNUM"], "VISIT": row["VISIT"], "TUDTC": row["TUDTC"], } ) os.makedirs(os.path.dirname(OUTPUT_PATH), exist_ok=True) with open(OUTPUT_PATH, "w", newline="", encoding="utf-8") as f: writer = csv.DictWriter(f, fieldnames=COLUMNS) writer.writeheader() writer.writerows(out)if __name__ == "__main__": main()solutions/sdtm-tu-tumor-identification/result.sas
Edit16 lines/* opensas reference solution for the yamaa benchmark sdtm-tu-tumor-identification. Reads the benchmark inputs and derives the requested output. Execute with the existing opensas runtime: opensas /app/output/result.sas. */data tu_raw; length STUDYID USUBJID LESION_CATEGORY LESION_NUM LOCATION LATERALITY METHOD EVALUATOR VISIT TUDTC $1024; infile "/app/input/tu_raw.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. TUSEQ LESION_CATEGORY : $char1024. LESION_NUM : $char1024. LOCATION : $char1024. LATERALITY : $char1024. METHOD : $char1024. EVALUATOR : $char1024. VISITNUM VISIT : $char1024. TUDTC : $char1024.;run;data result; set tu_raw; length DOMAIN $2 TULNKID TUTESTCD TUTEST TUORRES TUSTRESC TULOC TULAT TUMETHOD TUEVAL PREFIX $1024;DOMAIN='TU'; TUORRES=upcase(LESION_CATEGORY); TUSTRESC=TUORRES; TUTESTCD='TUMIDENT'; TUTEST='Tumor Identification'; TULOC=upcase(LOCATION); TULAT=upcase(LATERALITY); TUMETHOD=METHOD; TUEVAL=EVALUATOR; PREFIX='NEW'; if TUORRES='TARGET' then PREFIX='T'; if TUORRES='NON-TARGET' then PREFIX='NT'; TULNKID=cats(PREFIX,put(input(LESION_NUM,best32.),z2.)); run;proc sql; create table final as select DOMAIN, STUDYID, USUBJID, TUSEQ, TULNKID, TUTESTCD, TUTEST, TUORRES, TUSTRESC, TULOC, TULAT, TUMETHOD, TUEVAL, VISITNUM, VISIT, TUDTC from result;quit;proc export data=final outfile="/app/output/tu.csv" dbms=csv replace; run;