Review ask

As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.

We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.

Summary

Lifecycle: reviewedEditUnder review
What this benchmark means
README.md

Public entries: Python and R. Passing coverage requires an installed inventory.

Goal: attach the reference unit and limits to each collected test result and flag the result against them: LBSTRESU, LBSTNRLO, LBSTNRHI, and LBNRIND.

Input: collected results, one row per result carrying the test code and numeric result; a demographics input carrying sex by subject; plus a reference dictionary keyed by test code and sex giving the unit and the lower and upper limits.

Variables:

  • LBTESTCD is the test code as collected; together with sex from DM it selects the reference entry.
  • LBSTRESN is the numeric result in standard units as collected; missing when the result was not collected.
  • LBSTRESU is the reference unit for the test-and-sex combination, in standard units.
  • LBSTNRLO is the lower reference limit in standard units.
  • LBSTNRHI is the upper reference limit in standard units.
  • LBNRIND is LOW when the result is below the lower limit, HIGH when it is above the upper limit, and NORMAL otherwise; blank when the result or either reference limit is missing.

Note: a test-and-sex combination with two reference entries stops the run. One with no entry leaves the unit and both limits blank, and its range indicator stays blank. A record whose result is missing still carries its unit and limits.

Inputs

Under review
3 source files

input/dm.csv

Edit
8 rows
input/dm.csv
STUDYIDUSUBJIDSEX
PILOT7M
PILOT7F
PILOT7M
PILOT7F
PILOT7M
PILOT7F
PILOT7M
PILOT7M

input/lb_raw.csv

Edit
8 rows
input/lb_raw.csv
STUDYIDUSUBJIDLBSEQLBTESTCDLBSTRESN
PILOT71ALT15
PILOT71ALT55
PILOT71HGB12.5
PILOT71HGB12.5
PILOT71CREAT0.9
PILOT71POTASSIUM3.2
PILOT71ALT—
PILOT71GLUCOSE5

input/lbrange.csv

Edit
6 rows
input/lbrange.csv
LBTESTCDSEXUNITNRLONRHI
ALTMU/L040
ALTFU/L040
HGBMg/dL13.517.5
HGBFg/dL1216
CREATMmg/dL0.71.3
POTASSIUMFmmol/L3.55.1

Expected output

Under review
8 expected rows

expected/lb.csv

Edit
8 rows
expected/lb.csv
DOMAINSTUDYIDUSUBJIDLBSEQLBTESTCDLBSTRESNLBSTRESULBSTNRLOLBSTNRHILBNRIND
LBPILOT71ALT15U/L040NORMAL
LBPILOT71ALT55U/L040HIGH
LBPILOT71HGB12.5g/dL13.517.5LOW
LBPILOT71HGB12.5g/dL1216NORMAL
LBPILOT71CREAT0.9mg/dL0.71.3NORMAL
LBPILOT71POTASSIUM3.2mmol/L3.55.1LOW
LBPILOT71ALT—U/L040—
LBPILOT71GLUCOSE5————
Shaded columns are derived— = missing value

Human review and assessment

Your review

3 questions and your comments · View benchmark discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Variable NameVariable LabelTypeLengthControlled Terms or FormatOriginCoreConversion DefinitionVariable TypeVariable OrderComments for Define
DOMAINDomain AbbreviationCharAssignedSet constant value "LB".SDTM1
STUDYIDStudy IdentifierCharCollectedCopy value from LB_RAW.STUDYID.SDTM2
USUBJIDUnique Subject IdentifierCharCollectedCopy value from LB_RAW.USUBJID.SDTM3
LBSEQSequence NumberNumCollectedCopy value from LB_RAW.LBSEQ.SDTM4
LBTESTCDLab Test or Examination Short NameCharCollectedCopy value from LB_RAW.LBTESTCD.SDTM5
LBSTRESNNumeric Result/Finding in Standard UnitsNumCollectedCopy value from LB_RAW.LBSTRESN.SDTM6
LBSTRESUStandard UnitsCharDerivedTake UNIT from the LBRANGE record matching on LBTESTCD and SEX, and no match yields blank.SDTM7
LBSTNRLOReference Range Lower Limit-Std UnitsNumDerivedTake NRLO from the LBRANGE record matching on LBTESTCD and SEX, and no match yields blank.SDTM8
LBSTNRHIReference Range Upper Limit-Std UnitsNumDerivedTake NRHI from the LBRANGE record matching on LBTESTCD and SEX, and no match yields blank.SDTM9
LBNRINDReference Range IndicatorCharDerivedIf LBSTRESN IS NULL OR LBSTNRLO IS NULL OR LBSTNRHI IS NULL then "None"; If LBSTRESN < LBSTNRLO then "LOW"; If LBSTRESN > LBSTNRHI then "HIGH"; otherwise "NORMAL".SDTM10
VersionDateAuthorDescriptionReviewerSign-off
1.72026-10-20yamaa docs buildRead a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default
1.62026-10-17yamaa docs buildDescribe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight
1.52026-10-17yamaa docs buildName collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part
1.42026-10-15yamaa docs buildState datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/sdtm-lb-ranges/full.md

Edit
20 lines

Following CDISC SDTM standards, use the provided DM, LBRANGE, and LB_RAW datasets to create an LB dataset with one record per collected test result.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBSTRESN, LBSTRESU, LBSTNRLO, LBSTNRHI, LBNRIND

DOMAIN is "LB". The record's test code and the subject's sex in DM together select the reference entry in LBRANGE. LBSTRESU is that entry's unit, LBSTNRLO is its lower limit, and LBSTNRHI is its upper limit. When no entry matches the record's test code and the subject's sex, LBSTRESU, LBSTNRLO, and LBSTNRHI have no value. LBNRIND is "LOW" when the result is below the lower limit, "HIGH" when it is above the upper limit, and "NORMAL" otherwise. LBNRIND has no value when the result or either limit has no value. A record whose result was not collected still carries its unit and its lower and upper limits.

Read the source datasets from /app/input and save the completed dataset as /app/output/lb.csv.

prompts/sdtm-lb-ranges/conventions.md

Edit
12 lines

Following CDISC SDTM standards, use the provided DM, LBRANGE, and LB_RAW datasets to create an LB dataset with one record per collected test result.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBSTRESN, LBSTRESU, LBSTNRLO, LBSTNRHI, LBNRIND

DOMAIN is "LB". LBNRIND is "LOW", "HIGH", or "NORMAL", or has no value.

Read the source datasets from /app/input and save the completed dataset as /app/output/lb.csv.

prompts/sdtm-lb-ranges/brief.md

Edit
10 lines

Following CDISC SDTM standards, use the provided DM, LBRANGE, and LB_RAW datasets to create an LB dataset with one record per collected test result.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBSTRESN, LBSTRESU, LBSTNRLO, LBSTNRHI, LBNRIND

Read the source datasets from /app/input and save the completed dataset as /app/output/lb.csv.

yamaa code

Public calls generated from benchmark.yaml; results must match the expected output

R

Edit
3 lines
library(yamaa)yamaa_spec("spec.yaml")$run()$output

Python

Edit
3 lines
import yamaayamaa.spec("spec.yaml").run().output

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/sdtm-lb-ranges/result.R

Edit
51 lines
# Reference solution for the yamaa benchmark sdtm-lb-ranges (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)lb_raw <- read_csv(  "/app/input/lb_raw.csv",  col_types = cols(LBSEQ = col_integer(), .default = col_character()))dm <- read_csv(  "/app/input/dm.csv",  col_types = cols(.default = col_character()))lbrange <- read_csv(  "/app/input/lbrange.csv",  col_types = cols(.default = col_character()))lb <- lb_raw |>  left_join(dm |> select(USUBJID, SEX), by = "USUBJID") |>  left_join(lbrange, by = c("LBTESTCD", "SEX")) |>  mutate(    DOMAIN = "LB",    LBSTRESU = UNIT,    LBSTNRLO = NRLO,    LBSTNRHI = NRHI,    LBSTRESN = as.character(LBSTRESN),    LBNRIND = case_when(      is.na(LBSTRESN) | LBSTRESN == "" ~ NA_character_,      is.na(NRLO) | NRLO == "" ~ NA_character_,      is.na(NRHI) | NRHI == "" ~ NA_character_,      suppressWarnings(as.numeric(LBSTRESN)) <        suppressWarnings(as.numeric(NRLO)) ~ "LOW",      suppressWarnings(as.numeric(LBSTRESN)) >        suppressWarnings(as.numeric(NRHI)) ~ "HIGH",      TRUE ~ "NORMAL"    ),    LBSTRESN = na_if(LBSTRESN, "")  ) |>  arrange(USUBJID, LBSEQ) |>  select(    DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBSTRESN, LBSTRESU,    LBSTNRLO, LBSTNRHI, LBNRIND  )dir.create("/app/output", showWarnings = FALSE)write_csv(lb, "/app/output/lb.csv", na = "")

solutions/sdtm-lb-ranges/result.py

Edit
98 lines
# Reference solution for the yamaa benchmark sdtm-lb-ranges (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1."""Build the SDTM LB dataset from DM, LBRANGE, and LB_RAW.One record per collected test result in LB_RAW. The reference range(unit and limits) is selected by the record's test code together withthe subject's sex from DM. LBNRIND flags the result against the limits."""import csvimport osINPUT_DIR = "/app/input"OUTPUT_DIR = "/app/output"def read_rows(name):    with open(os.path.join(INPUT_DIR, name), newline="") as handle:        return list(csv.DictReader(handle))def main():    dm = read_rows("dm.csv")    lbrange = read_rows("lbrange.csv")    lb_raw = read_rows("lb_raw.csv")    sex_by_subject = {row["USUBJID"]: row["SEX"] for row in dm}    range_by_key = {        (row["LBTESTCD"], row["SEX"]): row for row in lbrange    }    records = []    for row in lb_raw:        sex = sex_by_subject.get(row["USUBJID"], "")        entry = range_by_key.get((row["LBTESTCD"], sex))        if entry is None:            unit = ""            nrlo = ""            nrhi = ""        else:            unit = entry["UNIT"]            nrlo = entry["NRLO"]            nrhi = entry["NRHI"]        result = row["LBSTRESN"]        nrind = ""        if result != "" and nrlo != "" and nrhi != "":            value = float(result)            if value < float(nrlo):                nrind = "LOW"            elif value > float(nrhi):                nrind = "HIGH"            else:                nrind = "NORMAL"        records.append(            {                "DOMAIN": "LB",                "STUDYID": row["STUDYID"],                "USUBJID": row["USUBJID"],                "LBSEQ": row["LBSEQ"],                "LBTESTCD": row["LBTESTCD"],                "LBSTRESN": result,                "LBSTRESU": unit,                "LBSTNRLO": nrlo,                "LBSTNRHI": nrhi,                "LBNRIND": nrind,            }        )    fieldnames = [        "DOMAIN",        "STUDYID",        "USUBJID",        "LBSEQ",        "LBTESTCD",        "LBSTRESN",        "LBSTRESU",        "LBSTNRLO",        "LBSTNRHI",        "LBNRIND",    ]    os.makedirs(OUTPUT_DIR, exist_ok=True)    with open(        os.path.join(OUTPUT_DIR, "lb.csv"), "w", newline=""    ) as handle:        writer = csv.DictWriter(handle, fieldnames=fieldnames)        writer.writeheader()        writer.writerows(records)if __name__ == "__main__":    main()

solutions/sdtm-lb-ranges/result.sas

Edit
29 lines
/* opensas reference solution for the yamaa benchmark sdtm-lb-ranges.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data dm;  length STUDYID USUBJID SEX $1024;  infile "/app/input/dm.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. SEX : $char1024.;run;data lb_raw;  length STUDYID USUBJID LBTESTCD $1024;  infile "/app/input/lb_raw.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. LBSEQ LBTESTCD : $char1024. LBSTRESN;run;data lbrange;  length LBTESTCD SEX UNIT $1024;  infile "/app/input/lbrange.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input LBTESTCD : $char1024. SEX : $char1024. UNIT : $char1024. NRLO NRHI;run;proc sql;create table joined as select a.*,c.UNIT as LBSTRESU,c.NRLO as LBSTNRLO,c.NRHI as LBSTNRHI from lb_raw a left join dm b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID left join lbrange c on a.LBTESTCD=c.LBTESTCD and b.SEX=c.SEX; quit;data result; set joined; length DOMAIN $2 LBNRIND $8; DOMAIN='LB'; LBNRIND=''; if not missing(LBSTRESN) and not missing(LBSTNRLO) and not missing(LBSTNRHI) then do; if LBSTRESN<LBSTNRLO then LBNRIND='LOW'; else if LBSTRESN>LBSTNRHI then LBNRIND='HIGH'; else LBNRIND='NORMAL'; end; run;proc sql;  create table final as select DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBSTRESN, LBSTRESU, LBSTNRLO, LBSTNRHI, LBNRIND from result;quit;proc export data=final outfile="/app/output/lb.csv" dbms=csv replace; run;