yamaa As a clinical trial statistician or programmer, challenge anything in this benchmark: describe a problem in Comments, or propose a fix with the Edit button on any source file.
- Task: Would a real study need this derivation, specified this way? Does it follow CDISC SDTM: variable names, controlled terms, record structure? Flag any rule that is ambiguous or that a sponsor would handle differently.
- Inputs: Do they look like data collected in a real trial, and do they include the situations you would expect to meet in one?
- Expected output: Would independent QC programming reproduce every value from these inputs and rules?
The other tabs beside Comments are background for assessment, not under review.
Summary
EditUnder reviewWhat this benchmark meansREADME.mdGoal: attach the reference unit and limits to each collected
test result and flag the result against them: LBSTRESU,
LBSTNRLO, LBSTNRHI, and LBNRIND.
Input: collected results, one row per result carrying the test code and numeric result; a demographics input carrying sex by subject; plus a reference dictionary keyed by test code and sex giving the unit and the lower and upper limits.
Variables:
LBTESTCDis the test code as collected; together with sex from DM it selects the reference entry.LBSTRESNis the numeric result in standard units as collected; missing when the result was not collected.LBSTRESUis the reference unit for the test-and-sex combination, in standard units.LBSTNRLOis the lower reference limit in standard units.LBSTNRHIis the upper reference limit in standard units.LBNRINDisLOWwhen the result is below the lower limit,HIGHwhen it is above the upper limit, andNORMALotherwise; blank when the result or either reference limit is missing.
Note: a test-and-sex combination with two reference entries stops the run. One with no entry leaves the unit and both limits blank, and its range indicator stays blank. A record whose result is missing still carries its unit and limits.
Inputs
Under review3 source filesinput/dm.csv
Edit8 rows| STUDYID | USUBJID | SEX |
|---|---|---|
| PILOT7 | M | |
| PILOT7 | F | |
| PILOT7 | M | |
| PILOT7 | F | |
| PILOT7 | M | |
| PILOT7 | F | |
| PILOT7 | M | |
| PILOT7 | M |
input/lb_raw.csv
Edit8 rows| STUDYID | USUBJID | LBSEQ | LBTESTCD | LBSTRESN |
|---|---|---|---|---|
| PILOT7 | 1 | ALT | 15 | |
| PILOT7 | 1 | ALT | 55 | |
| PILOT7 | 1 | HGB | 12.5 | |
| PILOT7 | 1 | HGB | 12.5 | |
| PILOT7 | 1 | CREAT | 0.9 | |
| PILOT7 | 1 | POTASSIUM | 3.2 | |
| PILOT7 | 1 | ALT | — | |
| PILOT7 | 1 | GLUCOSE | 5 |
input/lbrange.csv
Edit6 rows| LBTESTCD | SEX | UNIT | NRLO | NRHI |
|---|---|---|---|---|
| ALT | M | U/L | 0 | 40 |
| ALT | F | U/L | 0 | 40 |
| HGB | M | g/dL | 13.5 | 17.5 |
| HGB | F | g/dL | 12 | 16 |
| CREAT | M | mg/dL | 0.7 | 1.3 |
| POTASSIUM | F | mmol/L | 3.5 | 5.1 |
Expected output
Under review8 expected rowsexpected/lb.csv
Edit8 rows| DOMAIN | STUDYID | USUBJID | LBSEQ | LBTESTCD | LBSTRESN | LBSTRESU | LBSTNRLO | LBSTNRHI | LBNRIND |
|---|---|---|---|---|---|---|---|---|---|
| LB | PILOT7 | 1 | ALT | 15 | U/L | 0 | 40 | NORMAL | |
| LB | PILOT7 | 1 | ALT | 55 | U/L | 0 | 40 | HIGH | |
| LB | PILOT7 | 1 | HGB | 12.5 | g/dL | 13.5 | 17.5 | LOW | |
| LB | PILOT7 | 1 | HGB | 12.5 | g/dL | 12 | 16 | NORMAL | |
| LB | PILOT7 | 1 | CREAT | 0.9 | mg/dL | 0.7 | 1.3 | NORMAL | |
| LB | PILOT7 | 1 | POTASSIUM | 3.2 | mmol/L | 3.5 | 5.1 | LOW | |
| LB | PILOT7 | 1 | ALT | — | U/L | 0 | 40 | — | |
| LB | PILOT7 | 1 | GLUCOSE | 5 | — | — | — | — |
Comments and assessment materials
| Variable Name | Variable Label | Type | Length | Controlled Terms or Format | Origin | Core | Conversion Definition | Variable Type | Variable Order | Comments for Define |
|---|---|---|---|---|---|---|---|---|---|---|
| DOMAIN | Domain Abbreviation | Char | Assigned | Set constant value "LB". | SDTM | 1 | ||||
| STUDYID | Study Identifier | Char | Collected | Copy value from LB_RAW.STUDYID. | SDTM | 2 | ||||
| USUBJID | Unique Subject Identifier | Char | Collected | Copy value from LB_RAW.USUBJID. | SDTM | 3 | ||||
| LBSEQ | Sequence Number | Num | Collected | Copy value from LB_RAW.LBSEQ. | SDTM | 4 | ||||
| LBTESTCD | Lab Test or Examination Short Name | Char | Collected | Copy value from LB_RAW.LBTESTCD. | SDTM | 5 | ||||
| LBSTRESN | Numeric Result/Finding in Standard Units | Num | Collected | Copy value from LB_RAW.LBSTRESN. | SDTM | 6 | ||||
| LBSTRESU | Standard Units | Char | Derived | Take UNIT from the LBRANGE record matching on LBTESTCD and SEX, and no match yields blank. | SDTM | 7 | ||||
| LBSTNRLO | Reference Range Lower Limit-Std Units | Num | Derived | Take NRLO from the LBRANGE record matching on LBTESTCD and SEX, and no match yields blank. | SDTM | 8 | ||||
| LBSTNRHI | Reference Range Upper Limit-Std Units | Num | Derived | Take NRHI from the LBRANGE record matching on LBTESTCD and SEX, and no match yields blank. | SDTM | 9 | ||||
| LBNRIND | Reference Range Indicator | Char | Derived | If LBSTRESN IS NULL OR LBSTNRLO IS NULL OR LBSTNRHI IS NULL then "None"; If LBSTRESN < LBSTNRLO then "LOW"; If LBSTRESN > LBSTNRHI then "HIGH"; otherwise "NORMAL". | SDTM | 10 |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/sdtm-lb-ranges/full.md
Edit25 linesFollowing CDISC SDTM standards, use the provided LB_RAW, LBRANGE, and DM datasets to create an LB dataset with one record per collected test result.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBSTRESN, LBSTRESU, LBSTNRLO, LBSTNRHI, LBNRIND
LBSEQ is the sequence number as collected. LBTESTCD is the test code as collected; together with sex from DM it selects the reference entry. LBSTRESN is the numeric result in standard units as collected, and has no value when the result was not collected. LBSTRESU, LBSTNRLO, and LBSTNRHI are the reference unit and the lower and upper limits for that test-and-sex combination, in standard units.
LBNRIND is LOW when the result is below the lower limit, HIGH when it is above the upper limit, and NORMAL otherwise; it has no value when the result or either reference limit is missing.
A combination with no reference entry leaves the unit and both limits with no value, and its range indicator has no value. A record whose result is missing still carries its unit and limits.
Read the source datasets from /app/input and save the completed dataset as /app/output/lb.csv.
prompts/sdtm-lb-ranges/conventions.md
Edit12 linesFollowing CDISC SDTM standards, use the provided LB_RAW, LBRANGE, and DM datasets to create an LB dataset with one record per collected test result.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBSTRESN, LBSTRESU, LBSTNRLO, LBSTNRHI, LBNRIND
LBNRIND is LOW, HIGH, or NORMAL, or has no value.
Read the source datasets from /app/input and save the completed dataset as /app/output/lb.csv.
prompts/sdtm-lb-ranges/brief.md
Edit10 linesFollowing CDISC SDTM standards, use the provided LB_RAW, LBRANGE, and DM datasets to create an LB dataset with one record per collected test result.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBSTRESN, LBSTRESU, LBSTNRLO, LBSTNRHI, LBNRIND
Read the source datasets from /app/input and save the completed dataset as /app/output/lb.csv.
run.R
Edit4 lineslibrary(yamaa)lb <- yamaa_domain("spec.yaml")$outputlbrun.py
Edit4 linesimport yamaalb = yamaa.yamaa_domain("spec.yaml").outputlbsolutions/sdtm-lb-ranges/result.R
Edit51 lines# Reference solution for the yamaa benchmark sdtm-lb-ranges (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)lb_raw <- read_csv( "/app/input/lb_raw.csv", col_types = cols(LBSEQ = col_integer(), .default = col_character()))dm <- read_csv( "/app/input/dm.csv", col_types = cols(.default = col_character()))lbrange <- read_csv( "/app/input/lbrange.csv", col_types = cols(.default = col_character()))lb <- lb_raw |> left_join(dm |> select(USUBJID, SEX), by = "USUBJID") |> left_join(lbrange, by = c("LBTESTCD", "SEX")) |> mutate( DOMAIN = "LB", LBSTRESU = UNIT, LBSTNRLO = NRLO, LBSTNRHI = NRHI, LBSTRESN = as.character(LBSTRESN), LBNRIND = case_when( is.na(LBSTRESN) | LBSTRESN == "" ~ NA_character_, is.na(NRLO) | NRLO == "" ~ NA_character_, is.na(NRHI) | NRHI == "" ~ NA_character_, suppressWarnings(as.numeric(LBSTRESN)) < suppressWarnings(as.numeric(NRLO)) ~ "LOW", suppressWarnings(as.numeric(LBSTRESN)) > suppressWarnings(as.numeric(NRHI)) ~ "HIGH", TRUE ~ "NORMAL" ), LBSTRESN = na_if(LBSTRESN, "") ) |> arrange(USUBJID, LBSEQ) |> select( DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBSTRESN, LBSTRESU, LBSTNRLO, LBSTNRHI, LBNRIND )dir.create("/app/output", showWarnings = FALSE)write_csv(lb, "/app/output/lb.csv", na = "")solutions/sdtm-lb-ranges/result.py
Edit89 lines# Reference solution for the yamaa benchmark sdtm-lb-ranges (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.from pathlib import Pathimport polars as pldef _present(value) -> bool: return value is not None and str(value) != ""lb_raw = pl.read_csv("/app/input/lb_raw.csv", infer_schema=False)dm = pl.read_csv("/app/input/dm.csv", infer_schema=False)lbrange = pl.read_csv("/app/input/lbrange.csv", infer_schema=False)base = lb_raw.join(dm.select("USUBJID", "SEX"), on="USUBJID", how="left")joined = base.join(lbrange, on=["LBTESTCD", "SEX"], how="left")def _indicator(row) -> str | None: result, lo, hi = row["LBSTRESN"], row["NRLO"], row["NRHI"] if not _present(result) or not _present(lo) or not _present(hi): return None try: value, low, high = float(result), float(lo), float(hi) except ValueError: return None if value < low: return "LOW" if value > high: return "HIGH" return "NORMAL"rows = []for row in joined.to_dicts(): rows.append( { "DOMAIN": "LB", "STUDYID": row["STUDYID"], "USUBJID": row["USUBJID"], "LBSEQ": row["LBSEQ"], "LBTESTCD": row["LBTESTCD"], "LBSTRESN": row["LBSTRESN"] if _present(row["LBSTRESN"]) else None, "LBSTRESU": row["UNIT"] if _present(row.get("UNIT")) else None, "LBSTNRLO": row["NRLO"] if _present(row.get("NRLO")) else None, "LBSTNRHI": row["NRHI"] if _present(row.get("NRHI")) else None, "LBNRIND": _indicator(row), } )lb = ( pl.DataFrame( rows, schema={ "DOMAIN": pl.String, "STUDYID": pl.String, "USUBJID": pl.String, "LBSEQ": pl.String, "LBTESTCD": pl.String, "LBSTRESN": pl.String, "LBSTRESU": pl.String, "LBSTNRLO": pl.String, "LBSTNRHI": pl.String, "LBNRIND": pl.String, }, ) .with_columns(pl.col("LBSEQ").cast(pl.Int64).alias("_seq")) .sort(["USUBJID", "_seq"]) .select( "DOMAIN", "STUDYID", "USUBJID", "LBSEQ", "LBTESTCD", "LBSTRESN", "LBSTRESU", "LBSTNRLO", "LBSTNRHI", "LBNRIND", ))Path("/app/output").mkdir(exist_ok=True)lb.write_csv("/app/output/lb.csv")solutions/sdtm-lb-ranges/result.sas
Edit29 lines/* opensas reference solution for the yamaa benchmark sdtm-lb-ranges. Reads the benchmark inputs and derives the requested output. Execute with the existing opensas runtime: opensas /app/output/result.sas. */data dm; length STUDYID USUBJID SEX $1024; infile "/app/input/dm.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. SEX : $char1024.;run;data lb_raw; length STUDYID USUBJID LBTESTCD $1024; infile "/app/input/lb_raw.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. LBSEQ LBTESTCD : $char1024. LBSTRESN;run;data lbrange; length LBTESTCD SEX UNIT $1024; infile "/app/input/lbrange.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input LBTESTCD : $char1024. SEX : $char1024. UNIT : $char1024. NRLO NRHI;run;proc sql;create table joined as select a.*,c.UNIT as LBSTRESU,c.NRLO as LBSTNRLO,c.NRHI as LBSTNRHI from lb_raw a left join dm b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID left join lbrange c on a.LBTESTCD=c.LBTESTCD and b.SEX=c.SEX; quit;data result; set joined; length DOMAIN $2 LBNRIND $8; DOMAIN='LB'; LBNRIND=''; if not missing(LBSTRESN) and not missing(LBSTNRLO) and not missing(LBSTNRHI) then do; if LBSTRESN<LBSTNRLO then LBNRIND='LOW'; else if LBSTRESN>LBSTNRHI then LBNRIND='HIGH'; else LBNRIND='NORMAL'; end; run;proc sql; create table final as select DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBSTRESN, LBSTRESU, LBSTNRLO, LBSTNRHI, LBNRIND from result;quit;proc export data=final outfile="/app/output/lb.csv" dbms=csv replace; run;
Comments
Signed in with GitHub; saved in repository Discussions