Review ask

As a clinical trial statistician or programmer, challenge anything in this benchmark: describe a problem in Comments, or propose a fix with the Edit button on any source file.

  1. Task: Would a real study need this derivation, specified this way? Does it follow CDISC SDTM: variable names, controlled terms, record structure? Flag any rule that is ambiguous or that a sponsor would handle differently.
  2. Inputs: Do they look like data collected in a real trial, and do they include the situations you would expect to meet in one?
  3. Expected output: Would independent QC programming reproduce every value from these inputs and rules?

The other tabs beside Comments are background for assessment, not under review.

Summary

Lifecycle: reviewedEditUnder review
What this benchmark means
README.md

Goal: build one laboratory record per collected neutrophil or hemoglobin result, carrying LBTESTCD, LBSTRESN, and LBTOXGR from the Common Terminology Criteria for Adverse Events (CTCAE) v5.0 bands for that test.

Input: collected laboratory results with test, sex, standardized numeric result, and an independently recorded grade confirming the assigned grade.

Variables:

  • LBTESTCD is the test short name as collected: ANC (absolute neutrophil count) or HGB (hemoglobin).
  • LBSTRESN is the standardized numeric result as collected.
  • LBTOXGR is the standard toxicity grade for the result, one of 4, 3, 2, 1, or 0, from the band set for that test and sex; each band includes its lower limit and excludes its upper one.

Note: neutrophil counts use one band set whatever the sex, and the example lab's lower limit of normal is 1.8 x 10^9/L. The ANC bands are in 10^9/L: below 0.5 is grade 4, 0.5 to below 1.0 is grade 3, 1.0 to below 1.5 is grade 2, and 1.5 to below 1.8 is grade 1. Hemoglobin uses one band set per sex. A result for any other test, or a hemoglobin result with a sex other than M or F, gets no record. A missing result, or an assigned grade that differs from the independently recorded one, stops the run.

Inputs

Under review
1 source file

input/lb_raw.csv

Edit
10 rows
input/lb_raw.csv
STUDYIDUSUBJIDLBSEQLBTESTCDSEXLBSTRESNREPROGRADE
PILOT71ANCM1.61
PILOT72ANCM0.753
PILOT71ANCF0.084
PILOT71HGBM12.51
PILOT71HGBF12.50
PILOT72HGBF92
PILOT71ANCF0.53
PILOT72ANCF1.02
PILOT73ANCF1.51
PILOT74ANCF1.80

Expected output

Under review
10 expected rows

expected/lb.csv

Edit
10 rows
expected/lb.csv
DOMAINSTUDYIDUSUBJIDLBSEQLBTESTCDLBSTRESNLBTOXGR
LBPILOT71ANC1.61
LBPILOT72ANC0.753
LBPILOT71ANC0.084
LBPILOT71ANC0.53
LBPILOT72ANC12
LBPILOT73ANC1.51
LBPILOT74ANC1.80
LBPILOT71HGB12.51
LBPILOT71HGB12.50
LBPILOT72HGB92
Shaded columns are derived— = missing value

Comments and assessment materials

Comments

Signed in with GitHub; saved in repository Discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Variable NameVariable LabelTypeLengthControlled Terms or FormatOriginCoreConversion DefinitionVariable TypeVariable OrderComments for Define
DOMAINDomain AbbreviationCharAssignedSet constant value "LB".SDTM1
STUDYIDStudy IdentifierCharCollectedCopy value from LB_RAW.STUDYID.SDTM2
USUBJIDUnique Subject IdentifierCharCollectedCopy value from LB_RAW.USUBJID.SDTM3
LBSEQSequence NumberNumCollectedCopy value from LB_RAW.LBSEQ.SDTM4
LBTESTCDLab Test or Examination Short NameCharCollectedCopy value from LB_RAW.LBTESTCD.SDTM5
LBSTRESNNumeric Result/Finding in Standard UnitsNumCollectedCopy value from LB_RAW.LBSTRESN.SDTM6
LBTOXGRStandard Toxicity GradeCharDerivedSet per constructed row; see the Row construction sheet.SDTM7
TemplateInclude whenLBTOXGR
neutrophilsLB_RAW.LBTESTCD = 'ANC'Band LB_RAW.LBSTRESN at 0.5, 1.0, 1.5, 1.8 into "4", "3", "2", "1" and "0", intervals closed on the left
haemoglobin-maleLB_RAW.LBTESTCD = 'HGB' AND LB_RAW.SEX = 'M'Band LB_RAW.LBSTRESN at 8.0, 10.0, 13.5 into "3", "2", "1" and "0", intervals closed on the left
haemoglobin-femaleLB_RAW.LBTESTCD = 'HGB' AND LB_RAW.SEX = 'F'Band LB_RAW.LBSTRESN at 8.0, 10.0, 12.0 into "3", "2", "1" and "0", intervals closed on the left
VersionDateAuthorDescriptionReviewerSign-off
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/sdtm-lb-grading/full.md

Edit
25 lines

Following CDISC SDTM standards, use the provided LB_RAW dataset to create an LB dataset with one laboratory record per collected neutrophil or hemoglobin result.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBSTRESN, LBTOXGR

LBSEQ is the sequence number as collected. LBTESTCD is the test short name as collected: ANC for absolute neutrophil count or HGB for hemoglobin. LBSTRESN is the standardized numeric result as collected. LBTOXGR is the standard toxicity grade for the result: 4, 3, 2, 1, or 0. Each band includes its lower limit and excludes its upper one.

Neutrophil counts use one band set whatever the sex, with the example lab's lower limit of normal at 1.8: below 0.5 is grade 4, 0.5 to below 1.0 is grade 3, 1.0 to below 1.5 is grade 2, 1.5 to below 1.8 is grade 1, and 1.8 or above is grade 0. Hemoglobin uses one band set per sex: below 8.0 is grade 3, below 10.0 is grade 2, and below 13.5 for males (below 12.0 for females) is grade 1; higher results are grade 0.

A result for any other test, or a hemoglobin result with a sex other than M or F, gets no record.

Read the source datasets from /app/input and save the completed dataset as /app/output/lb.csv.

prompts/sdtm-lb-grading/conventions.md

Edit
12 lines

Following CDISC SDTM standards, use the provided LB_RAW dataset to create an LB dataset with one laboratory record per collected neutrophil or hemoglobin result.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBSTRESN, LBTOXGR

LBTESTCD is ANC for absolute neutrophil count or HGB for hemoglobin. LBTOXGR is 4, 3, 2, 1, or 0.

Read the source datasets from /app/input and save the completed dataset as /app/output/lb.csv.

prompts/sdtm-lb-grading/brief.md

Edit
9 lines

Following CDISC SDTM standards, use the provided LB_RAW dataset to create an LB dataset with one laboratory record per collected neutrophil or hemoglobin result.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBSTRESN, LBTOXGR

Read the source datasets from /app/input and save the completed dataset as /app/output/lb.csv.

yamaa code

Runs the yamaa specification; its result must match the expected output

run.R

Edit
4 lines
library(yamaa)lb <- yamaa_domain("spec.yaml")$outputlb

run.py

Edit
4 lines
import yamaalb = yamaa.yamaa_domain("spec.yaml").outputlb

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/sdtm-lb-grading/result.R

Edit
49 lines
# Reference solution for the yamaa benchmark sdtm-lb-grading (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)raw <- read_csv(  "/app/input/lb_raw.csv",  col_types = cols(    LBSEQ = col_integer(),    LBSTRESN = col_double(),    .default = col_character()  ))toxgrade <- function(test, sex, value) {  if (is.na(value)) return(NA_character_)  if (test == "ANC") {    if (value < 0.5) return("4")    if (value < 1.0) return("3")    if (value < 1.5) return("2")    if (value < 1.8) return("1")    return("0")  }  if (test == "HGB") {    if (!sex %in% c("M", "F")) return(NA_character_)    if (value < 8.0) return("3")    if (value < 10.0) return("2")    limit <- if (sex == "M") 13.5 else 12.0    if (value < limit) return("1")    return("0")  }  NA_character_}lb <- raw |>  filter(LBTESTCD %in% c("ANC", "HGB"), !(LBTESTCD == "HGB" & !SEX %in% c("M", "F"))) |>  rowwise() |>  mutate(LBTOXGR = toxgrade(LBTESTCD, SEX, LBSTRESN)) |>  ungroup() |>  mutate(DOMAIN = "LB") |>  select(DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBSTRESN, LBTOXGR) |>  arrange(STUDYID, USUBJID, LBSEQ)dir.create("/app/output", showWarnings = FALSE)write_csv(lb, "/app/output/lb.csv", na = "")

solutions/sdtm-lb-grading/result.py

Edit
77 lines
# Reference solution for the yamaa benchmark sdtm-lb-grading (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.from pathlib import Pathimport polars as plraw = pl.read_csv("/app/input/lb_raw.csv", infer_schema=False).with_columns(    pl.col("LBSEQ").cast(pl.Int64),    pl.col("LBSTRESN").cast(pl.Float64, strict=False),)def toxgrade(test, sex, value):    if value is None:        return None    if test == "ANC":        if value < 0.5:            return "4"        if value < 1.0:            return "3"        if value < 1.5:            return "2"        if value < 1.8:            return "1"        return "0"    if test == "HGB":        if sex not in ("M", "F"):            return None        if value < 8.0:            return "3"        if value < 10.0:            return "2"        limit = 13.5 if sex == "M" else 12.0        if value < limit:            return "1"        return "0"    return Nonerows = []for r in raw.to_dicts():    grade = toxgrade(r["LBTESTCD"], r["SEX"], r["LBSTRESN"])    if grade is None and r["LBTESTCD"] not in ("ANC", "HGB"):        continue    if r["LBTESTCD"] == "HGB" and r["SEX"] not in ("M", "F"):        continue    rows.append(        {            "DOMAIN": "LB",            "STUDYID": r["STUDYID"],            "USUBJID": r["USUBJID"],            "LBSEQ": r["LBSEQ"],            "LBTESTCD": r["LBTESTCD"],            "LBSTRESN": r["LBSTRESN"],            "LBTOXGR": grade,        }    )lb = pl.DataFrame(    rows,    schema={        "DOMAIN": pl.String,        "STUDYID": pl.String,        "USUBJID": pl.String,        "LBSEQ": pl.Int64,        "LBTESTCD": pl.String,        "LBSTRESN": pl.Float64,        "LBTOXGR": pl.String,    },).sort(["STUDYID", "USUBJID", "LBSEQ"])Path("/app/output").mkdir(parents=True, exist_ok=True)lb.write_csv("/app/output/lb.csv")

solutions/sdtm-lb-grading/result.sas

Edit
17 lines
/* opensas reference solution for the yamaa benchmark sdtm-lb-grading.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data lb_raw;  length STUDYID USUBJID LBTESTCD SEX REPROGRADE $1024;  infile "/app/input/lb_raw.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. LBSEQ LBTESTCD : $char1024. SEX : $char1024. LBSTRESN REPROGRADE : $char1024.;run;data result; set lb_raw; length DOMAIN $2; DOMAIN='LB'; LBTOXGR=.;if LBTESTCD='ANC' then do; if missing(LBSTRESN) then LBTOXGR=.; else if LBSTRESN<0.5 then LBTOXGR=4; else if LBSTRESN<1 then LBTOXGR=3; else if LBSTRESN<1.5 then LBTOXGR=2; else if LBSTRESN<1.8 then LBTOXGR=1; else LBTOXGR=0; end;else if LBTESTCD='HGB' and SEX in ('M','F') then do; if missing(LBSTRESN) then LBTOXGR=.; else if LBSTRESN<8 then LBTOXGR=3; else if LBSTRESN<10 then LBTOXGR=2; else if (SEX='M' and LBSTRESN<13.5) or (SEX='F' and LBSTRESN<12) then LBTOXGR=1; else LBTOXGR=0; end; else delete; run;proc sql;  create table final as select DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBSTRESN, LBTOXGR from result;quit;proc export data=final outfile="/app/output/lb.csv" dbms=csv replace; run;