yamaa As a clinical trial statistician or programmer, challenge anything in this benchmark: describe a problem in Comments, or propose a fix with the Edit button on any source file.
- Task: Would a real study need this derivation, specified this way? Does it follow CDISC SDTM: variable names, controlled terms, record structure? Flag any rule that is ambiguous or that a sponsor would handle differently.
- Inputs: Do they look like data collected in a real trial, and do they include the situations you would expect to meet in one?
- Expected output: Would independent QC programming reproduce every value from these inputs and rules?
The other tabs beside Comments are background for assessment, not under review.
Summary
EditUnder reviewWhat this benchmark meansREADME.mdGoal: build one record per collected glucose and creatinine
result, with per-test submission metadata: each LBTESTCD value
gets its own codelist and origin for LBORRESU, overriding the
shared column-level declaration.
Input: long-form Operational Data Model (ODM) rows, one row per collected item, with the collected entry in the value field. Glucose rows carry one item, creatinine rows carry another, and the collection date rides along in the same visit group.
Variables:
LBTESTCDisGLUCfor glucose rows andCREATfor creatinine rows.LBTESTisGlucosewhen the test code isGLUCandCreatininewhen it isCREAT.LBORRESis the collected entry, kept exactly as reported.LBORRESUismg/dLfor every record. Its submission metadata is stated once for the whole column and overridden for each test code: glucose records name the glucose codelist with a vendor origin, creatinine records the creatinine codelist with an investigator origin.LBSTRESNis the numeric form of the collected entry; missing when the entry is not a number.LBSTRESUismg/dLfor every record.LBDTCis the collection date from the same visit group.
Note: a test with no collected entry produces no record, so the records are only results actually reported. The per-value submission metadata is validated when the specification loads; the engine output is unaffected by it.
submission-spec.yaml and submission-environment.yaml are a separate,
independently completed submission witness over the original executable
specification. They add complete dataset/column metadata and retain the
glucose and creatinine codelist declarations. The completion explicitly uses
Sponsor for the creatinine Assigned origin: SDTM admits Vendor or Sponsor for
Assigned, whereas the original family-independent loader fixture retains its
Investigator declaration. Installed Define tests also exercise that original
declaration's family-admission failure.
The fixed-time witness reads the unchanged expected/lb.csv only when
Dataset-JSON is explicitly requested; it does not execute the specification
or open its ODM input. Its independently authored XML structure is validated
against the bundled offline Define-XML XSD. The original declarations, input,
CSV expectation and reference loader controls retain their existing meaning.
Inputs
Under review1 source fileinput/odm.csv
Edit5 rows| StudyOID | MetaDataVersionOID | SubjectKey | StudyEventOID | StudyEventRepeatKey | ItemGroupOID | ItemGroupRepeatKey | ItemOID | Value |
|---|---|---|---|---|---|---|---|---|
| STUDY01 | MDV.1 | 001 | BASELINE | 1 | IG.LB | 1 | IT.LB.LBDTC | 2025-01-01 |
| STUDY01 | MDV.1 | 001 | BASELINE | 1 | IG.LB | 1 | IT.LB.GLUC | 95 |
| STUDY01 | MDV.1 | 001 | BASELINE | 1 | IG.LB | 1 | IT.LB.CREAT | 0.9 |
| STUDY01 | MDV.1 | 002 | BASELINE | 1 | IG.LB | 1 | IT.LB.LBDTC | 2025-01-02 |
| STUDY01 | MDV.1 | 002 | BASELINE | 1 | IG.LB | 1 | IT.LB.GLUC | 101 |
Expected output
Under review3 expected rowsexpected/lb.csv
Edit3 rows| DOMAIN | STUDYID | USUBJID | LBSEQ | LBTESTCD | LBTEST | LBORRES | LBORRESU | LBSTRESN | LBSTRESU | LBDTC |
|---|---|---|---|---|---|---|---|---|---|---|
| LB | STUDY01 | 2 | GLUC | Glucose | 95 | mg/dL | 95 | mg/dL | 2025-01-01 | |
| LB | STUDY01 | 1 | GLUC | Glucose | 101 | mg/dL | 101 | mg/dL | 2025-01-02 | |
| LB | STUDY01 | 1 | CREAT | Creatinine | 0.9 | mg/dL | 0.9 | mg/dL | 2025-01-01 |
Comments and assessment materials
| Variable Name | Variable Label | Type | Length | Controlled Terms or Format | Origin | Core | Conversion Definition | Variable Type | Variable Order | Comments for Define |
|---|---|---|---|---|---|---|---|---|---|---|
| DOMAIN | Domain Abbreviation | Char | Assigned | Set constant value "LB". | SDTM | 1 | ||||
| STUDYID | Study Identifier | Char | Collected | Copy value from ODM.StudyOID. | SDTM | 2 | ||||
| USUBJID | Unique Subject Identifier | Char | Collected | Copy value from ODM.SubjectKey. | SDTM | 3 | ||||
| LBTESTCD | Lab Test or Examination Short Name | Char | Derived | Set per constructed row; see the Row construction sheet. | SDTM | 4 | ||||
| LBTEST | Lab Test or Examination Name | Char | Derived | Set per constructed row; see the Row construction sheet. | SDTM | 5 | ||||
| LBORRES | Result or Finding in Original Units | Char | Derived | Set per constructed row; see the Row construction sheet. | SDTM | 6 | ||||
| LBORRESU | Original Units | Char | CL_LBORRESU | Assigned | Set constant value "mg/dL". | SDTM | 7 | |||
| LBSTRESN | Numeric Result/Finding in Standard Units | Num | Derived | Copy value from LBORRES; a value that cannot be converted → blank. | SDTM | 8 | ||||
| LBSTRESU | Standard Units | Char | Assigned | Set constant value "mg/dL". | SDTM | 9 | ||||
| LBDTC | Date/Time of Specimen Collection | Char | ISO 8601 | Derived | Set per constructed row; see the Row construction sheet. | SDTM | 10 | |||
| LBSEQ | Sequence Number | Num | Derived | Row number within each group of STUDYID, USUBJID, ordered by LBDTC then LBTESTCD. | SDTM | 11 |
| Template | Include when | LBTESTCD | LBTEST | LBORRES | LBDTC |
|---|---|---|---|---|---|
| glucose | LBORRES IS NOT NULL | Constant "GLUC" | Constant "Glucose" | Aggregate ONLY(ODM.Value) over records matching on the shared output keys, where ODM.ItemOID = 'IT.LB.GLUC' | Aggregate ONLY(ODM.Value) over records matching on the shared output keys, where ODM.ItemOID = 'IT.LB.LBDTC' |
| creatinine | LBORRES IS NOT NULL | Constant "CREAT" | Constant "Creatinine" | Aggregate ONLY(ODM.Value) over records matching on the shared output keys, where ODM.ItemOID = 'IT.LB.CREAT' | Aggregate ONLY(ODM.Value) over records matching on the shared output keys, where ODM.ItemOID = 'IT.LB.LBDTC' |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/sdtm-lb-metadata/full.md
Edit24 linesFollowing CDISC SDTM standards, use the provided ODM dataset to create an LB dataset with one record per collected glucose and creatinine result.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBTEST, LBORRES, LBORRESU, LBSTRESN, LBSTRESU, LBDTC
STUDYID is the ODM StudyOID and USUBJID is the SubjectKey, each as given.
LBSEQ numbers the records within a subject by collection date, then test code. LBTESTCD is GLUC for glucose rows and CREAT for creatinine rows; LBTEST is Glucose when the test code is GLUC and Creatinine when it is CREAT.
LBORRES is the collected entry, kept exactly as reported. LBORRESU and LBSTRESU are mg/dL for every record. LBSTRESN is the numeric form of the collected entry, and has no value when the entry is not a number. LBDTC is the collection date from the same visit group.
A test with no collected entry produces no record, so the records are only results actually reported.
Read the source datasets from /app/input and save the completed dataset as /app/output/lb.csv.
prompts/sdtm-lb-metadata/conventions.md
Edit12 linesFollowing CDISC SDTM standards, use the provided ODM dataset to create an LB dataset with one record per collected glucose and creatinine result.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBTEST, LBORRES, LBORRESU, LBSTRESN, LBSTRESU, LBDTC
LBTESTCD is GLUC for glucose rows and CREAT for creatinine rows; LBTEST is Glucose when the test code is GLUC and Creatinine when it is CREAT.
Read the source datasets from /app/input and save the completed dataset as /app/output/lb.csv.
prompts/sdtm-lb-metadata/brief.md
Edit9 linesFollowing CDISC SDTM standards, use the provided ODM dataset to create an LB dataset with one record per collected glucose and creatinine result.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBTEST, LBORRES, LBORRESU, LBSTRESN, LBSTRESU, LBDTC
Read the source datasets from /app/input and save the completed dataset as /app/output/lb.csv.
run.R
Edit4 lineslibrary(yamaa)lb <- yamaa_domain("spec.yaml")$outputlbrun.py
Edit4 linesimport yamaalb = yamaa.yamaa_domain("spec.yaml").outputlbsolutions/sdtm-lb-metadata/result.R
Edit52 lines# Reference solution for the yamaa benchmark sdtm-lb-metadata (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)library(tidyr)odm <- read_csv( "/app/input/odm.csv", col_types = cols(.default = col_character()))key_cols <- c( "StudyOID", "SubjectKey", "StudyEventOID", "StudyEventRepeatKey", "ItemGroupOID", "ItemGroupRepeatKey")dates <- odm |> filter(ItemOID == "IT.LB.LBDTC") |> select(all_of(key_cols), LBDTC = Value)tests <- odm |> filter(ItemOID %in% c("IT.LB.GLUC", "IT.LB.CREAT")) |> left_join(dates, by = key_cols) |> mutate( LBTESTCD = if_else(ItemOID == "IT.LB.GLUC", "GLUC", "CREAT"), LBTEST = if_else(ItemOID == "IT.LB.GLUC", "Glucose", "Creatinine"), LBORRES = na_if(Value, ""), ) |> filter(!is.na(LBORRES)) |> mutate( DOMAIN = "LB", STUDYID = StudyOID, USUBJID = SubjectKey, LBORRESU = "mg/dL", LBSTRESN = suppressWarnings(as.numeric(LBORRES)), LBSTRESU = "mg/dL", ) |> arrange(STUDYID, USUBJID, LBDTC, LBTESTCD) |> group_by(STUDYID, USUBJID) |> mutate(LBSEQ = row_number()) |> ungroup() |> select( DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBTEST, LBORRES, LBORRESU, LBSTRESN, LBSTRESU, LBDTC )dir.create("/app/output", showWarnings = FALSE)write_csv(tests, "/app/output/lb.csv", na = "")solutions/sdtm-lb-metadata/result.py
Edit105 lines# Reference solution for the yamaa benchmark sdtm-lb-metadata (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.from pathlib import Pathimport polars as plodm = pl.read_csv("/app/input/odm.csv", infer_schema=False)# One row per ItemOID within each visit group.key_cols = [ "StudyOID", "SubjectKey", "StudyEventOID", "StudyEventRepeatKey", "ItemGroupOID", "ItemGroupRepeatKey",]wide = ( odm.group_by(key_cols) .agg(pl.struct("ItemOID", "Value").alias("items")) .with_columns( pl.col("items") .list.eval(pl.element().filter(pl.element().struct.field("ItemOID") == "IT.LB.LBDTC")) .list.get(0) .struct.field("Value") .alias("LBDTC"), ) .drop("items"))# Keep the raw long rows for the two lab tests.tests = odm.filter(pl.col("ItemOID").is_in(["IT.LB.GLUC", "IT.LB.CREAT"]))code_of = {"IT.LB.GLUC": "GLUC", "IT.LB.CREAT": "CREAT"}name_of = {"GLUC": "Glucose", "CREAT": "Creatinine"}joined = tests.join(wide, on=key_cols, how="left")def as_number(text): if text is None or text == "": return None try: return float(text) except ValueError: return Nonerows = []for r in joined.to_dicts(): value = r["Value"] if value is None or value == "": continue code = code_of[r["ItemOID"]] rows.append( { "DOMAIN": "LB", "STUDYID": r["StudyOID"], "USUBJID": r["SubjectKey"], "LBDTC": r["LBDTC"], "LBTESTCD": code, "LBTEST": name_of[code], "LBORRES": value, "LBORRESU": "mg/dL", "LBSTRESN": as_number(value), "LBSTRESU": "mg/dL", } )lb = pl.DataFrame( rows, schema={ "DOMAIN": pl.String, "STUDYID": pl.String, "USUBJID": pl.String, "LBDTC": pl.String, "LBTESTCD": pl.String, "LBTEST": pl.String, "LBORRES": pl.String, "LBORRESU": pl.String, "LBSTRESN": pl.Float64, "LBSTRESU": pl.String, },).sort(["STUDYID", "USUBJID", "LBDTC", "LBTESTCD"]).with_columns( pl.int_range(1, pl.len() + 1).over(["STUDYID", "USUBJID"]).alias("LBSEQ")).select( "DOMAIN", "STUDYID", "USUBJID", "LBSEQ", "LBTESTCD", "LBTEST", "LBORRES", "LBORRESU", "LBSTRESN", "LBSTRESU", "LBDTC",)Path("/app/output").mkdir(parents=True, exist_ok=True)lb.write_csv("/app/output/lb.csv")solutions/sdtm-lb-metadata/result.sas
Edit39 lines/* opensas reference solution for the yamaa benchmark sdtm-lb-metadata. Reads the benchmark inputs and derives the requested output. Execute with the existing opensas runtime: opensas /app/output/result.sas. */data odm; length StudyOID MetaDataVersionOID SubjectKey StudyEventOID StudyEventRepeatKey ItemGroupOID ItemGroupRepeatKey ItemOID Value $1024; infile "/app/input/odm.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input StudyOID : $char1024. MetaDataVersionOID : $char1024. SubjectKey : $char1024. StudyEventOID : $char1024. StudyEventRepeatKey : $char1024. ItemGroupOID : $char1024. ItemGroupRepeatKey : $char1024. ItemOID : $char1024. Value : $char1024.;run;proc sort data=odm out=ordered; by StudyOID SubjectKey StudyEventOID StudyEventRepeatKey ItemGroupRepeatKey; run;data dates; length LBDTC $1024; retain LBDTC; set ordered; by StudyOID SubjectKey StudyEventOID StudyEventRepeatKey ItemGroupRepeatKey; if first.ItemGroupRepeatKey then do; LBDTC=''; end; if ItemOID='IT.LB.LBDTC' then LBDTC=Value; if last.ItemGroupRepeatKey then output;run;proc sql;create table joined as select a.*,b.LBDTC from odm a inner join dates b on a.StudyOID=b.StudyOID and a.SubjectKey=b.SubjectKey and a.StudyEventOID=b.StudyEventOID and a.StudyEventRepeatKey=b.StudyEventRepeatKey and a.ItemGroupRepeatKey=b.ItemGroupRepeatKey where a.ItemOID in ('IT.LB.GLUC','IT.LB.CREAT') and not missing(a.Value); quit;data prepared; set joined; length DOMAIN $2 STUDYID USUBJID LBTESTCD LBTEST LBORRES LBORRESU LBSTRESU LBSTAT $1024;DOMAIN='LB'; STUDYID=StudyOID; USUBJID=SubjectKey; LBORRES=Value; LBORRESU='mg/dL'; LBSTRESU='mg/dL'; LBSTRESN=input(Value,?? best32.); LBSTAT='';if ItemOID='IT.LB.CREAT' then do; LBTESTCD='CREAT'; LBTEST='Creatinine'; end; else do; LBTESTCD='GLUC'; LBTEST='Glucose'; end;if Value='NOT DONE' then do; LBORRES=''; LBORRESU=''; LBSTRESU=''; LBSTRESN=.; LBSTAT='NOT DONE'; end; run;proc sort data=prepared; by STUDYID USUBJID LBDTC LBTESTCD; run;data result; set prepared; by STUDYID USUBJID; retain LBSEQ; if first.USUBJID then LBSEQ=0; LBSEQ+1;run;proc sql; create table final as select DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBTEST, LBORRES, LBORRESU, LBSTRESN, LBSTRESU, LBDTC from result;quit;proc export data=final outfile="/app/output/lb.csv" dbms=csv replace; run;
Comments
Signed in with GitHub; saved in repository Discussions