As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.
We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.
Summary
README.mdPublic entries: Python and R. Passing coverage requires an installed inventory.
Goal: build one laboratory record per collected neutrophil or
hemoglobin result, carrying LBTESTCD, LBSTRESN, and LBTOXGR
from the Common Terminology Criteria for Adverse Events
(CTCAE) v5.0 bands for that test.
Input: collected laboratory results with test, sex, standardized numeric result, and an independently recorded grade confirming the assigned grade.
Variables:
LBTESTCDis the test short name as collected:ANC(absolute neutrophil count) orHGB(hemoglobin).LBSTRESNis the standardized numeric result as collected.LBTOXGRis the standard toxicity grade for the result, one of4,3,2,1, or0, from the band set for that test and sex; each band includes its lower limit and excludes its upper one.
Note: neutrophil counts use one band set whatever the sex, and
the example lab's lower limit of normal is 1.8 x 10^9/L. The ANC
bands are in 10^9/L: below 0.5 is grade 4, 0.5 to below 1.0 is
grade 3, 1.0 to below 1.5 is grade 2, and 1.5 to below 1.8 is
grade 1. Hemoglobin uses one band set per sex. A result for any other
test, or a hemoglobin result with a sex other than M or F, gets no record.
A missing result, or an assigned grade that differs from the
independently recorded one, stops the run.
Inputs
Under review1 source fileinput/lb_raw.csv
Edit10 rows| STUDYID | USUBJID | LBSEQ | LBTESTCD | SEX | LBSTRESN | REPROGRADE |
|---|---|---|---|---|---|---|
| PILOT7 | 1 | ANC | M | 1.6 | 1 | |
| PILOT7 | 2 | ANC | M | 0.75 | 3 | |
| PILOT7 | 1 | ANC | F | 0.08 | 4 | |
| PILOT7 | 1 | HGB | M | 12.5 | 1 | |
| PILOT7 | 1 | HGB | F | 12.5 | 0 | |
| PILOT7 | 2 | HGB | F | 9 | 2 | |
| PILOT7 | 1 | ANC | F | 0.5 | 3 | |
| PILOT7 | 2 | ANC | F | 1.0 | 2 | |
| PILOT7 | 3 | ANC | F | 1.5 | 1 | |
| PILOT7 | 4 | ANC | F | 1.8 | 0 |
Expected output
Under review10 expected rowsexpected/lb.csv
Edit10 rows| DOMAIN | STUDYID | USUBJID | LBSEQ | LBTESTCD | LBSTRESN | LBTOXGR |
|---|---|---|---|---|---|---|
| LB | PILOT7 | 1 | ANC | 1.6 | 1 | |
| LB | PILOT7 | 2 | ANC | 0.75 | 3 | |
| LB | PILOT7 | 1 | ANC | 0.08 | 4 | |
| LB | PILOT7 | 1 | ANC | 0.5 | 3 | |
| LB | PILOT7 | 2 | ANC | 1 | 2 | |
| LB | PILOT7 | 3 | ANC | 1.5 | 1 | |
| LB | PILOT7 | 4 | ANC | 1.8 | 0 | |
| LB | PILOT7 | 1 | HGB | 12.5 | 1 | |
| LB | PILOT7 | 1 | HGB | 12.5 | 0 | |
| LB | PILOT7 | 2 | HGB | 9 | 2 |
Human review and assessment
| Variable Name | Variable Label | Type | Length | Controlled Terms or Format | Origin | Core | Conversion Definition | Variable Type | Variable Order | Comments for Define |
|---|---|---|---|---|---|---|---|---|---|---|
| DOMAIN | Domain Abbreviation | Char | Assigned | Set constant value "LB". | SDTM | 1 | ||||
| STUDYID | Study Identifier | Char | Collected | Copy value from LB_RAW.STUDYID. | SDTM | 2 | ||||
| USUBJID | Unique Subject Identifier | Char | Collected | Copy value from LB_RAW.USUBJID. | SDTM | 3 | ||||
| LBSEQ | Sequence Number | Num | Collected | Copy value from LB_RAW.LBSEQ. | SDTM | 4 | ||||
| LBTESTCD | Lab Test or Examination Short Name | Char | Collected | Copy value from LB_RAW.LBTESTCD. | SDTM | 5 | ||||
| LBSTRESN | Numeric Result/Finding in Standard Units | Num | Collected | Copy value from LB_RAW.LBSTRESN. | SDTM | 6 | ||||
| LBTOXGR | Standard Toxicity Grade | Char | Derived | Set per constructed row; see the Row construction sheet. | SDTM | 7 |
| Template | Include when | LBTOXGR |
|---|---|---|
| neutrophils | LB_RAW.LBTESTCD = 'ANC' | Band LB_RAW.LBSTRESN at 0.5, 1.0, 1.5, 1.8 into "4", "3", "2", "1" and "0", intervals closed on the left |
| haemoglobin-male | LB_RAW.LBTESTCD = 'HGB' AND LB_RAW.SEX = 'M' | Band LB_RAW.LBSTRESN at 8.0, 10.0, 13.5 into "3", "2", "1" and "0", intervals closed on the left |
| haemoglobin-female | LB_RAW.LBTESTCD = 'HGB' AND LB_RAW.SEX = 'F' | Band LB_RAW.LBSTRESN at 8.0, 10.0, 12.0 into "3", "2", "1" and "0", intervals closed on the left |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.7 | 2026-10-20 | yamaa docs build | Read a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default | ||
| 1.6 | 2026-10-17 | yamaa docs build | Describe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight | ||
| 1.5 | 2026-10-17 | yamaa docs build | Name collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part | ||
| 1.4 | 2026-10-15 | yamaa docs build | State datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks | ||
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/sdtm-lb-grading/full.md
Edit22 linesFollowing CDISC SDTM standards, use the provided LB_RAW dataset to create an LB dataset with one record per collected neutrophil or hemoglobin result.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBSTRESN, LBTOXGR
DOMAIN is "LB". LBTOXGR is the standard toxicity grade for the result, one of "4", "3", "2", "1", or "0", from the band set for that test and sex; each band includes its lower limit and excludes its upper one. Neutrophil counts use one band set whatever the sex, with bands in 10^9/L: below 0.5 is grade "4", 0.5 to below 1.0 is grade "3", 1.0 to below 1.5 is grade "2", 1.5 to below 1.8 is grade "1", and 1.8 or above is grade "0". Hemoglobin uses one band set per sex: for "M", below 8.0 is grade "3", 8.0 to below 10.0 is grade "2", 10.0 to below 13.5 is grade "1", and 13.5 or above is grade "0"; for "F", below 8.0 is grade "3", 8.0 to below 10.0 is grade "2", 10.0 to below 12.0 is grade "1", and 12.0 or above is grade "0". A result for any other test, or a hemoglobin result with a sex other than "M" or "F", gets no record.
Read the source datasets from /app/input and save the completed dataset as /app/output/lb.csv.
prompts/sdtm-lb-grading/conventions.md
Edit11 linesFollowing CDISC SDTM standards, use the provided LB_RAW dataset to create an LB dataset with one record per collected neutrophil or hemoglobin result.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBSTRESN, LBTOXGR
DOMAIN is "LB". LBTOXGR is the standard toxicity grade for the result, one of "4", "3", "2", "1", or "0".
Read the source datasets from /app/input and save the completed dataset as /app/output/lb.csv.
prompts/sdtm-lb-grading/brief.md
Edit8 linesFollowing CDISC SDTM standards, use the provided LB_RAW dataset to create an LB dataset with one record per collected neutrophil or hemoglobin result.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBSTRESN, LBTOXGR
Read the source datasets from /app/input and save the completed dataset as /app/output/lb.csv.
R
Edit3 lineslibrary(yamaa)yamaa_spec("spec.yaml")$run()$outputPython
Edit3 linesimport yamaayamaa.spec("spec.yaml").run().outputsolutions/sdtm-lb-grading/result.R
Edit49 lines# Reference solution for the yamaa benchmark sdtm-lb-grading (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)raw <- read_csv( "/app/input/lb_raw.csv", col_types = cols( LBSEQ = col_integer(), LBSTRESN = col_double(), .default = col_character() ))toxgrade <- function(test, sex, value) { if (is.na(value)) return(NA_character_) if (test == "ANC") { if (value < 0.5) return("4") if (value < 1.0) return("3") if (value < 1.5) return("2") if (value < 1.8) return("1") return("0") } if (test == "HGB") { if (!sex %in% c("M", "F")) return(NA_character_) if (value < 8.0) return("3") if (value < 10.0) return("2") limit <- if (sex == "M") 13.5 else 12.0 if (value < limit) return("1") return("0") } NA_character_}lb <- raw |> filter(LBTESTCD %in% c("ANC", "HGB"), !(LBTESTCD == "HGB" & !SEX %in% c("M", "F"))) |> rowwise() |> mutate(LBTOXGR = toxgrade(LBTESTCD, SEX, LBSTRESN)) |> ungroup() |> mutate(DOMAIN = "LB") |> select(DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBSTRESN, LBTOXGR) |> arrange(STUDYID, USUBJID, LBSEQ)dir.create("/app/output", showWarnings = FALSE)write_csv(lb, "/app/output/lb.csv", na = "")solutions/sdtm-lb-grading/result.py
Edit85 lines# Reference solution for the yamaa benchmark sdtm-lb-grading (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1."""Build the SDTM LB dataset from LB_RAW.One record per collected neutrophil (ANC) or hemoglobin (HGB) result,with LBTOXGR the standard toxicity grade from the band set for thetest (and, for hemoglobin, the sex). Each band includes its lowerlimit and excludes its upper one. Results for any other test, andhemoglobin results with a sex other than "M" or "F", get no record."""import csvimport osINPUT_DIR = "/app/input"OUTPUT_DIR = "/app/output"# (lower limit inclusive, grade) pairs in ascending order; a value# takes the grade of the last band whose lower limit it reaches.ANC_BANDS = [(0.0, "4"), (0.5, "3"), (1.0, "2"), (1.5, "1"), (1.8, "0")]HGB_BANDS = { "M": [(0.0, "3"), (8.0, "2"), (10.0, "1"), (13.5, "0")], "F": [(0.0, "3"), (8.0, "2"), (10.0, "1"), (12.0, "0")],}def grade(value, bands): result = None for lower, band_grade in bands: if value >= lower: result = band_grade return resultdef main(): with open(os.path.join(INPUT_DIR, "lb_raw.csv"), newline="") as f: rows = list(csv.DictReader(f)) out = [] for row in rows: test = row["LBTESTCD"] value = float(row["LBSTRESN"]) if test == "ANC": toxgr = grade(value, ANC_BANDS) elif test == "HGB": bands = HGB_BANDS.get(row["SEX"]) if bands is None: continue toxgr = grade(value, bands) else: continue out.append( { "DOMAIN": "LB", "STUDYID": row["STUDYID"], "USUBJID": row["USUBJID"], "LBSEQ": row["LBSEQ"], "LBTESTCD": test, "LBSTRESN": row["LBSTRESN"], "LBTOXGR": toxgr, } ) os.makedirs(OUTPUT_DIR, exist_ok=True) columns = [ "DOMAIN", "STUDYID", "USUBJID", "LBSEQ", "LBTESTCD", "LBSTRESN", "LBTOXGR", ] with open(os.path.join(OUTPUT_DIR, "lb.csv"), "w", newline="") as f: writer = csv.DictWriter(f, fieldnames=columns) writer.writeheader() writer.writerows(out)if __name__ == "__main__": main()solutions/sdtm-lb-grading/result.sas
Edit17 lines/* opensas reference solution for the yamaa benchmark sdtm-lb-grading. Reads the benchmark inputs and derives the requested output. Execute with the existing opensas runtime: opensas /app/output/result.sas. */data lb_raw; length STUDYID USUBJID LBTESTCD SEX REPROGRADE $1024; infile "/app/input/lb_raw.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. LBSEQ LBTESTCD : $char1024. SEX : $char1024. LBSTRESN REPROGRADE : $char1024.;run;data result; set lb_raw; length DOMAIN $2; DOMAIN='LB'; LBTOXGR=.;if LBTESTCD='ANC' then do; if missing(LBSTRESN) then LBTOXGR=.; else if LBSTRESN<0.5 then LBTOXGR=4; else if LBSTRESN<1 then LBTOXGR=3; else if LBSTRESN<1.5 then LBTOXGR=2; else if LBSTRESN<1.8 then LBTOXGR=1; else LBTOXGR=0; end;else if LBTESTCD='HGB' and SEX in ('M','F') then do; if missing(LBSTRESN) then LBTOXGR=.; else if LBSTRESN<8 then LBTOXGR=3; else if LBSTRESN<10 then LBTOXGR=2; else if (SEX='M' and LBSTRESN<13.5) or (SEX='F' and LBSTRESN<12) then LBTOXGR=1; else LBTOXGR=0; end; else delete; run;proc sql; create table final as select DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBSTRESN, LBTOXGR from result;quit;proc export data=final outfile="/app/output/lb.csv" dbms=csv replace; run;
yamaa