Review ask

As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.

We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.

Summary

Lifecycle: reviewedEditUnder review
What this benchmark means
README.md

Public entries: Python and R. Passing coverage requires an installed inventory.

Goal: build one laboratory record per collected neutrophil or hemoglobin result, carrying LBTESTCD, LBSTRESN, and LBTOXGR from the Common Terminology Criteria for Adverse Events (CTCAE) v5.0 bands for that test.

Input: collected laboratory results with test, sex, standardized numeric result, and an independently recorded grade confirming the assigned grade.

Variables:

  • LBTESTCD is the test short name as collected: ANC (absolute neutrophil count) or HGB (hemoglobin).
  • LBSTRESN is the standardized numeric result as collected.
  • LBTOXGR is the standard toxicity grade for the result, one of 4, 3, 2, 1, or 0, from the band set for that test and sex; each band includes its lower limit and excludes its upper one.

Note: neutrophil counts use one band set whatever the sex, and the example lab's lower limit of normal is 1.8 x 10^9/L. The ANC bands are in 10^9/L: below 0.5 is grade 4, 0.5 to below 1.0 is grade 3, 1.0 to below 1.5 is grade 2, and 1.5 to below 1.8 is grade 1. Hemoglobin uses one band set per sex. A result for any other test, or a hemoglobin result with a sex other than M or F, gets no record. A missing result, or an assigned grade that differs from the independently recorded one, stops the run.

Inputs

Under review
1 source file

input/lb_raw.csv

Edit
10 rows
input/lb_raw.csv
STUDYIDUSUBJIDLBSEQLBTESTCDSEXLBSTRESNREPROGRADE
PILOT71ANCM1.61
PILOT72ANCM0.753
PILOT71ANCF0.084
PILOT71HGBM12.51
PILOT71HGBF12.50
PILOT72HGBF92
PILOT71ANCF0.53
PILOT72ANCF1.02
PILOT73ANCF1.51
PILOT74ANCF1.80

Expected output

Under review
10 expected rows

expected/lb.csv

Edit
10 rows
expected/lb.csv
DOMAINSTUDYIDUSUBJIDLBSEQLBTESTCDLBSTRESNLBTOXGR
LBPILOT71ANC1.61
LBPILOT72ANC0.753
LBPILOT71ANC0.084
LBPILOT71ANC0.53
LBPILOT72ANC12
LBPILOT73ANC1.51
LBPILOT74ANC1.80
LBPILOT71HGB12.51
LBPILOT71HGB12.50
LBPILOT72HGB92
Shaded columns are derived— = missing value

Human review and assessment

Your review

3 questions and your comments · View benchmark discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Variable NameVariable LabelTypeLengthControlled Terms or FormatOriginCoreConversion DefinitionVariable TypeVariable OrderComments for Define
DOMAINDomain AbbreviationCharAssignedSet constant value "LB".SDTM1
STUDYIDStudy IdentifierCharCollectedCopy value from LB_RAW.STUDYID.SDTM2
USUBJIDUnique Subject IdentifierCharCollectedCopy value from LB_RAW.USUBJID.SDTM3
LBSEQSequence NumberNumCollectedCopy value from LB_RAW.LBSEQ.SDTM4
LBTESTCDLab Test or Examination Short NameCharCollectedCopy value from LB_RAW.LBTESTCD.SDTM5
LBSTRESNNumeric Result/Finding in Standard UnitsNumCollectedCopy value from LB_RAW.LBSTRESN.SDTM6
LBTOXGRStandard Toxicity GradeCharDerivedSet per constructed row; see the Row construction sheet.SDTM7
TemplateInclude whenLBTOXGR
neutrophilsLB_RAW.LBTESTCD = 'ANC'Band LB_RAW.LBSTRESN at 0.5, 1.0, 1.5, 1.8 into "4", "3", "2", "1" and "0", intervals closed on the left
haemoglobin-maleLB_RAW.LBTESTCD = 'HGB' AND LB_RAW.SEX = 'M'Band LB_RAW.LBSTRESN at 8.0, 10.0, 13.5 into "3", "2", "1" and "0", intervals closed on the left
haemoglobin-femaleLB_RAW.LBTESTCD = 'HGB' AND LB_RAW.SEX = 'F'Band LB_RAW.LBSTRESN at 8.0, 10.0, 12.0 into "3", "2", "1" and "0", intervals closed on the left
VersionDateAuthorDescriptionReviewerSign-off
1.72026-10-20yamaa docs buildRead a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default
1.62026-10-17yamaa docs buildDescribe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight
1.52026-10-17yamaa docs buildName collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part
1.42026-10-15yamaa docs buildState datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/sdtm-lb-grading/full.md

Edit
22 lines

Following CDISC SDTM standards, use the provided LB_RAW dataset to create an LB dataset with one record per collected neutrophil or hemoglobin result.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBSTRESN, LBTOXGR

DOMAIN is "LB". LBTOXGR is the standard toxicity grade for the result, one of "4", "3", "2", "1", or "0", from the band set for that test and sex; each band includes its lower limit and excludes its upper one. Neutrophil counts use one band set whatever the sex, with bands in 10^9/L: below 0.5 is grade "4", 0.5 to below 1.0 is grade "3", 1.0 to below 1.5 is grade "2", 1.5 to below 1.8 is grade "1", and 1.8 or above is grade "0". Hemoglobin uses one band set per sex: for "M", below 8.0 is grade "3", 8.0 to below 10.0 is grade "2", 10.0 to below 13.5 is grade "1", and 13.5 or above is grade "0"; for "F", below 8.0 is grade "3", 8.0 to below 10.0 is grade "2", 10.0 to below 12.0 is grade "1", and 12.0 or above is grade "0". A result for any other test, or a hemoglobin result with a sex other than "M" or "F", gets no record.

Read the source datasets from /app/input and save the completed dataset as /app/output/lb.csv.

prompts/sdtm-lb-grading/conventions.md

Edit
11 lines

Following CDISC SDTM standards, use the provided LB_RAW dataset to create an LB dataset with one record per collected neutrophil or hemoglobin result.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBSTRESN, LBTOXGR

DOMAIN is "LB". LBTOXGR is the standard toxicity grade for the result, one of "4", "3", "2", "1", or "0".

Read the source datasets from /app/input and save the completed dataset as /app/output/lb.csv.

prompts/sdtm-lb-grading/brief.md

Edit
8 lines

Following CDISC SDTM standards, use the provided LB_RAW dataset to create an LB dataset with one record per collected neutrophil or hemoglobin result.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBSTRESN, LBTOXGR

Read the source datasets from /app/input and save the completed dataset as /app/output/lb.csv.

yamaa code

Public calls generated from benchmark.yaml; results must match the expected output

R

Edit
3 lines
library(yamaa)yamaa_spec("spec.yaml")$run()$output

Python

Edit
3 lines
import yamaayamaa.spec("spec.yaml").run().output

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/sdtm-lb-grading/result.R

Edit
49 lines
# Reference solution for the yamaa benchmark sdtm-lb-grading (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)raw <- read_csv(  "/app/input/lb_raw.csv",  col_types = cols(    LBSEQ = col_integer(),    LBSTRESN = col_double(),    .default = col_character()  ))toxgrade <- function(test, sex, value) {  if (is.na(value)) return(NA_character_)  if (test == "ANC") {    if (value < 0.5) return("4")    if (value < 1.0) return("3")    if (value < 1.5) return("2")    if (value < 1.8) return("1")    return("0")  }  if (test == "HGB") {    if (!sex %in% c("M", "F")) return(NA_character_)    if (value < 8.0) return("3")    if (value < 10.0) return("2")    limit <- if (sex == "M") 13.5 else 12.0    if (value < limit) return("1")    return("0")  }  NA_character_}lb <- raw |>  filter(LBTESTCD %in% c("ANC", "HGB"), !(LBTESTCD == "HGB" & !SEX %in% c("M", "F"))) |>  rowwise() |>  mutate(LBTOXGR = toxgrade(LBTESTCD, SEX, LBSTRESN)) |>  ungroup() |>  mutate(DOMAIN = "LB") |>  select(DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBSTRESN, LBTOXGR) |>  arrange(STUDYID, USUBJID, LBSEQ)dir.create("/app/output", showWarnings = FALSE)write_csv(lb, "/app/output/lb.csv", na = "")

solutions/sdtm-lb-grading/result.py

Edit
85 lines
# Reference solution for the yamaa benchmark sdtm-lb-grading (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1."""Build the SDTM LB dataset from LB_RAW.One record per collected neutrophil (ANC) or hemoglobin (HGB) result,with LBTOXGR the standard toxicity grade from the band set for thetest (and, for hemoglobin, the sex). Each band includes its lowerlimit and excludes its upper one. Results for any other test, andhemoglobin results with a sex other than "M" or "F", get no record."""import csvimport osINPUT_DIR = "/app/input"OUTPUT_DIR = "/app/output"# (lower limit inclusive, grade) pairs in ascending order; a value# takes the grade of the last band whose lower limit it reaches.ANC_BANDS = [(0.0, "4"), (0.5, "3"), (1.0, "2"), (1.5, "1"), (1.8, "0")]HGB_BANDS = {    "M": [(0.0, "3"), (8.0, "2"), (10.0, "1"), (13.5, "0")],    "F": [(0.0, "3"), (8.0, "2"), (10.0, "1"), (12.0, "0")],}def grade(value, bands):    result = None    for lower, band_grade in bands:        if value >= lower:            result = band_grade    return resultdef main():    with open(os.path.join(INPUT_DIR, "lb_raw.csv"), newline="") as f:        rows = list(csv.DictReader(f))    out = []    for row in rows:        test = row["LBTESTCD"]        value = float(row["LBSTRESN"])        if test == "ANC":            toxgr = grade(value, ANC_BANDS)        elif test == "HGB":            bands = HGB_BANDS.get(row["SEX"])            if bands is None:                continue            toxgr = grade(value, bands)        else:            continue        out.append(            {                "DOMAIN": "LB",                "STUDYID": row["STUDYID"],                "USUBJID": row["USUBJID"],                "LBSEQ": row["LBSEQ"],                "LBTESTCD": test,                "LBSTRESN": row["LBSTRESN"],                "LBTOXGR": toxgr,            }        )    os.makedirs(OUTPUT_DIR, exist_ok=True)    columns = [        "DOMAIN",        "STUDYID",        "USUBJID",        "LBSEQ",        "LBTESTCD",        "LBSTRESN",        "LBTOXGR",    ]    with open(os.path.join(OUTPUT_DIR, "lb.csv"), "w", newline="") as f:        writer = csv.DictWriter(f, fieldnames=columns)        writer.writeheader()        writer.writerows(out)if __name__ == "__main__":    main()

solutions/sdtm-lb-grading/result.sas

Edit
17 lines
/* opensas reference solution for the yamaa benchmark sdtm-lb-grading.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data lb_raw;  length STUDYID USUBJID LBTESTCD SEX REPROGRADE $1024;  infile "/app/input/lb_raw.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. LBSEQ LBTESTCD : $char1024. SEX : $char1024. LBSTRESN REPROGRADE : $char1024.;run;data result; set lb_raw; length DOMAIN $2; DOMAIN='LB'; LBTOXGR=.;if LBTESTCD='ANC' then do; if missing(LBSTRESN) then LBTOXGR=.; else if LBSTRESN<0.5 then LBTOXGR=4; else if LBSTRESN<1 then LBTOXGR=3; else if LBSTRESN<1.5 then LBTOXGR=2; else if LBSTRESN<1.8 then LBTOXGR=1; else LBTOXGR=0; end;else if LBTESTCD='HGB' and SEX in ('M','F') then do; if missing(LBSTRESN) then LBTOXGR=.; else if LBSTRESN<8 then LBTOXGR=3; else if LBSTRESN<10 then LBTOXGR=2; else if (SEX='M' and LBSTRESN<13.5) or (SEX='F' and LBSTRESN<12) then LBTOXGR=1; else LBTOXGR=0; end; else delete; run;proc sql;  create table final as select DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBSTRESN, LBTOXGR from result;quit;proc export data=final outfile="/app/output/lb.csv" dbms=csv replace; run;