As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.
We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.
Summary
README.mdPublic entries: Python and R. Passing coverage requires an installed inventory.
Goal: build one record per collected glucose and creatinine
result, with per-test submission metadata: each LBTESTCD value
gets its own codelist and origin for LBORRESU, overriding the
shared column-level declaration.
Input: long-form Operational Data Model (ODM) rows, one row per collected item, with the collected entry in the value field. Glucose rows carry one item, creatinine rows carry another, and the collection date rides along in the same visit group.
Variables:
LBTESTCDisGLUCfor glucose rows andCREATfor creatinine rows.LBTESTisGlucosewhen the test code isGLUCandCreatininewhen it isCREAT.LBORRESis the collected entry, kept exactly as reported.LBORRESUismg/dLfor every record. Its submission metadata is stated once for the whole column and overridden for each test code: glucose records name the glucose codelist with a vendor origin, creatinine records the creatinine codelist with an investigator origin.LBSTRESNis the numeric form of the collected entry; missing when the entry is not a number.LBSTRESUismg/dLfor every record.LBDTCis the collection date from the same visit group.
Note: a test with no collected entry produces no record, so the records are only results actually reported. The per-value submission metadata is validated when the specification loads; the engine output is unaffected by it.
The case runs submission-spec.yaml, an independently completed submission
layer over the original executable spec.yaml, with submission-project.yaml,
the project file that declares the unit codelists both layers bind, so every
unit is checked against its list. The completion adds complete dataset/column
metadata and retains the glucose and creatinine codelist declarations. It
restates both test codes' entries, since a later layer replaces the whole
list, and explicitly uses Sponsor for the creatinine Assigned origin: SDTM
admits Vendor or Sponsor for Assigned, whereas the original family-independent
loader fixture retains its Investigator declaration. Installed Define tests
also exercise that original declaration's family-admission failure.
The installed Define tests' fixed-time witness reads the unchanged
expected/lb.csv only when Dataset-JSON is explicitly requested; it does not
execute the specification or open its ODM input. Its independently authored
XML structure is validated against the bundled offline Define-XML XSD. The
original declarations, input, CSV expectation and reference loader controls
retain their existing meaning.
Inputs
Under review1 source fileinput/odm.csv
Edit5 rows| StudyOID | MetaDataVersionOID | SubjectKey | StudyEventOID | StudyEventRepeatKey | ItemGroupOID | ItemGroupRepeatKey | ItemOID | Value |
|---|---|---|---|---|---|---|---|---|
| STUDY01 | MDV.1 | 001 | BASELINE | 1 | IG.LB | 1 | IT.LB.LBDTC | 2025-01-01 |
| STUDY01 | MDV.1 | 001 | BASELINE | 1 | IG.LB | 1 | IT.LB.GLUC | 95 |
| STUDY01 | MDV.1 | 001 | BASELINE | 1 | IG.LB | 1 | IT.LB.CREAT | 0.9 |
| STUDY01 | MDV.1 | 002 | BASELINE | 1 | IG.LB | 1 | IT.LB.LBDTC | 2025-01-02 |
| STUDY01 | MDV.1 | 002 | BASELINE | 1 | IG.LB | 1 | IT.LB.GLUC | 101 |
Expected output
Under review3 expected rowsexpected/lb.csv
Edit3 rows| DOMAIN | STUDYID | USUBJID | LBSEQ | LBTESTCD | LBTEST | LBORRES | LBORRESU | LBSTRESN | LBSTRESU | LBDTC |
|---|---|---|---|---|---|---|---|---|---|---|
| LB | STUDY01 | 2 | GLUC | Glucose | 95 | mg/dL | 95 | mg/dL | 2025-01-01 | |
| LB | STUDY01 | 1 | GLUC | Glucose | 101 | mg/dL | 101 | mg/dL | 2025-01-02 | |
| LB | STUDY01 | 1 | CREAT | Creatinine | 0.9 | mg/dL | 0.9 | mg/dL | 2025-01-01 |
Human review and assessment
| Variable Name | Variable Label | Type | Length | Controlled Terms or Format | Origin | Core | Conversion Definition | Variable Type | Variable Order | Comments for Define |
|---|---|---|---|---|---|---|---|---|---|---|
| DOMAIN | 2 | Assigned | Req | No derivation in this specification; inherited from its parent. | SDTM | 1 | ||||
| STUDYID | 20 | Collected | Req | No derivation in this specification; inherited from its parent. | SDTM | 2 | ||||
| USUBJID | 20 | Collected | Req | No derivation in this specification; inherited from its parent. | SDTM | 3 | ||||
| LBSEQ | 8 | Derived | Req | Sequence within each subject by collection date and test code | SDTM | 4 | ||||
| LBTESTCD | 8 | Assigned | Req | No derivation in this specification; inherited from its parent. | SDTM | 5 | ||||
| LBTEST | 40 | Assigned | Req | No derivation in this specification; inherited from its parent. | SDTM | 6 | ||||
| LBORRES | 20 | Collected | Exp | No derivation in this specification; inherited from its parent. | SDTM | 7 | ||||
| LBORRESU | 8 | Assigned | Exp | No derivation in this specification; inherited from its parent. | SDTM | 8 | ||||
| LBSTRESN | 8 | Derived | Exp | Convert the original result to the declared numeric type | SDTM | 9 | ||||
| LBSTRESU | 8 | Assigned | Exp | No derivation in this specification; inherited from its parent. | SDTM | 10 | ||||
| LBDTC | Collected | Exp | No derivation in this specification; inherited from its parent. | SDTM | 11 |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.7 | 2026-10-20 | yamaa docs build | Read a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default | ||
| 1.6 | 2026-10-17 | yamaa docs build | Describe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight | ||
| 1.5 | 2026-10-17 | yamaa docs build | Name collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part | ||
| 1.4 | 2026-10-15 | yamaa docs build | State datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks | ||
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/sdtm-lb-metadata/full.md
Edit20 linesFollowing CDISC SDTM standards, use the provided ODM dataset to create an LB dataset with one record per collected glucose or creatinine result.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBTEST, LBORRES, LBORRESU, LBSTRESN, LBSTRESU, LBDTC
DOMAIN is "LB". STUDYID is the study OID and USUBJID the subject key, each as given in ODM. Each collected glucose result and each collected creatinine result gives one record, and a test with no collected entry gives no record. LBTESTCD is "GLUC" for a glucose result and "CREAT" for a creatinine result. LBTEST is "Glucose" for "GLUC" and "Creatinine" for "CREAT". LBORRES is the collected entry, kept exactly as reported. LBORRESU is "mg/dL" for every record. LBSTRESN is the numeric form of the collected entry and has no value when the entry is not a number. LBSTRESU is "mg/dL" for every record. LBDTC is the collection date collected in the same visit group. LBSEQ numbers each subject's records from 1 by collection date, then by test code.
Read the source datasets from /app/input and save the completed dataset as /app/output/lb.csv.
prompts/sdtm-lb-metadata/conventions.md
Edit12 linesFollowing CDISC SDTM standards, use the provided ODM dataset to create an LB dataset with one record per collected glucose or creatinine result.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBTEST, LBORRES, LBORRESU, LBSTRESN, LBSTRESU, LBDTC
DOMAIN is "LB". LBTESTCD is "GLUC" for a glucose result and "CREAT" for a creatinine result. LBTEST is "Glucose" for "GLUC" and "Creatinine" for "CREAT".
Read the source datasets from /app/input and save the completed dataset as /app/output/lb.csv.
prompts/sdtm-lb-metadata/brief.md
Edit9 linesFollowing CDISC SDTM standards, use the provided ODM dataset to create an LB dataset with one record per collected glucose or creatinine result.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBTEST, LBORRES, LBORRESU, LBSTRESN, LBSTRESU, LBDTC
Read the source datasets from /app/input and save the completed dataset as /app/output/lb.csv.
R
Edit3 lineslibrary(yamaa)yamaa_project("submission-project.yaml")$spec("LB")$run()$outputPython
Edit3 linesimport yamaayamaa.project("submission-project.yaml").spec("LB").run().outputsolutions/sdtm-lb-metadata/result.R
Edit52 lines# Reference solution for the yamaa benchmark sdtm-lb-metadata (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)library(tidyr)odm <- read_csv( "/app/input/odm.csv", col_types = cols(.default = col_character()))key_cols <- c( "StudyOID", "SubjectKey", "StudyEventOID", "StudyEventRepeatKey", "ItemGroupOID", "ItemGroupRepeatKey")dates <- odm |> filter(ItemOID == "IT.LB.LBDTC") |> select(all_of(key_cols), LBDTC = Value)tests <- odm |> filter(ItemOID %in% c("IT.LB.GLUC", "IT.LB.CREAT")) |> left_join(dates, by = key_cols) |> mutate( LBTESTCD = if_else(ItemOID == "IT.LB.GLUC", "GLUC", "CREAT"), LBTEST = if_else(ItemOID == "IT.LB.GLUC", "Glucose", "Creatinine"), LBORRES = na_if(Value, ""), ) |> filter(!is.na(LBORRES)) |> mutate( DOMAIN = "LB", STUDYID = StudyOID, USUBJID = SubjectKey, LBORRESU = "mg/dL", LBSTRESN = suppressWarnings(as.numeric(LBORRES)), LBSTRESU = "mg/dL", ) |> arrange(STUDYID, USUBJID, LBDTC, LBTESTCD) |> group_by(STUDYID, USUBJID) |> mutate(LBSEQ = row_number()) |> ungroup() |> select( DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBTEST, LBORRES, LBORRESU, LBSTRESN, LBSTRESU, LBDTC )dir.create("/app/output", showWarnings = FALSE)write_csv(tests, "/app/output/lb.csv", na = "")solutions/sdtm-lb-metadata/result.py
Edit115 lines# Reference solution for the yamaa benchmark sdtm-lb-metadata (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1."""Build the SDTM LB dataset from the ODM extract.One record per collected glucose or creatinine result. Reads/app/input/odm.csv and writes /app/output/lb.csv."""import csvfrom collections import OrderedDictINPUT_CSV = "/app/input/odm.csv"OUTPUT_CSV = "/app/output/lb.csv"TESTS = OrderedDict( [ ("IT.LB.GLUC", ("GLUC", "Glucose")), ("IT.LB.CREAT", ("CREAT", "Creatinine")), ])DATE_ITEM = "IT.LB.LBDTC"COLUMNS = [ "DOMAIN", "STUDYID", "USUBJID", "LBSEQ", "LBTESTCD", "LBTEST", "LBORRES", "LBORRESU", "LBSTRESN", "LBSTRESU", "LBDTC",]def numeric_form(text): """Return the numeric rendering of text, or '' when not a number.""" try: value = float(text) except (TypeError, ValueError): return "" if value == int(value): return str(int(value)) return repr(value)def main(): with open(INPUT_CSV, newline="", encoding="utf-8") as handle: rows = list(csv.DictReader(handle)) # Group items by subject and visit group (one collection event). groups = OrderedDict() for row in rows: key = ( row["StudyOID"], row["SubjectKey"], row["StudyEventOID"], row["StudyEventRepeatKey"], row["ItemGroupOID"], row["ItemGroupRepeatKey"], ) groups.setdefault(key, {})[row["ItemOID"]] = row["Value"] records = [] for (study_oid, subject_key, *_), items in groups.items(): lbdtc = items.get(DATE_ITEM, "") for item_oid, (testcd, test) in TESTS.items(): if item_oid not in items: continue entry = items[item_oid] records.append( { "DOMAIN": "LB", "STUDYID": study_oid, "USUBJID": subject_key, "LBTESTCD": testcd, "LBTEST": test, "LBORRES": entry, "LBORRESU": "mg/dL", "LBSTRESN": numeric_form(entry), "LBSTRESU": "mg/dL", "LBDTC": lbdtc, } ) # LBSEQ numbers each subject's records from 1 by collection date, # then by test code. by_subject = {} for record in records: by_subject.setdefault(record["USUBJID"], []).append(record) ordered = [] for subject in by_subject: subject_records = sorted( by_subject[subject], key=lambda r: (r["LBDTC"], r["LBTESTCD"]), ) for index, record in enumerate(subject_records, start=1): record["LBSEQ"] = index ordered.extend(subject_records) with open(OUTPUT_CSV, "w", newline="", encoding="utf-8") as handle: writer = csv.DictWriter(handle, fieldnames=COLUMNS) writer.writeheader() for record in ordered: writer.writerow({column: record[column] for column in COLUMNS})if __name__ == "__main__": main()solutions/sdtm-lb-metadata/result.sas
Edit39 lines/* opensas reference solution for the yamaa benchmark sdtm-lb-metadata. Reads the benchmark inputs and derives the requested output. Execute with the existing opensas runtime: opensas /app/output/result.sas. */data odm; length StudyOID MetaDataVersionOID SubjectKey StudyEventOID StudyEventRepeatKey ItemGroupOID ItemGroupRepeatKey ItemOID Value $1024; infile "/app/input/odm.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input StudyOID : $char1024. MetaDataVersionOID : $char1024. SubjectKey : $char1024. StudyEventOID : $char1024. StudyEventRepeatKey : $char1024. ItemGroupOID : $char1024. ItemGroupRepeatKey : $char1024. ItemOID : $char1024. Value : $char1024.;run;proc sort data=odm out=ordered; by StudyOID SubjectKey StudyEventOID StudyEventRepeatKey ItemGroupRepeatKey; run;data dates; length LBDTC $1024; retain LBDTC; set ordered; by StudyOID SubjectKey StudyEventOID StudyEventRepeatKey ItemGroupRepeatKey; if first.ItemGroupRepeatKey then do; LBDTC=''; end; if ItemOID='IT.LB.LBDTC' then LBDTC=Value; if last.ItemGroupRepeatKey then output;run;proc sql;create table joined as select a.*,b.LBDTC from odm a inner join dates b on a.StudyOID=b.StudyOID and a.SubjectKey=b.SubjectKey and a.StudyEventOID=b.StudyEventOID and a.StudyEventRepeatKey=b.StudyEventRepeatKey and a.ItemGroupRepeatKey=b.ItemGroupRepeatKey where a.ItemOID in ('IT.LB.GLUC','IT.LB.CREAT') and not missing(a.Value); quit;data prepared; set joined; length DOMAIN $2 STUDYID USUBJID LBTESTCD LBTEST LBORRES LBORRESU LBSTRESU LBSTAT $1024;DOMAIN='LB'; STUDYID=StudyOID; USUBJID=SubjectKey; LBORRES=Value; LBORRESU='mg/dL'; LBSTRESU='mg/dL'; LBSTRESN=input(Value,?? best32.); LBSTAT='';if ItemOID='IT.LB.CREAT' then do; LBTESTCD='CREAT'; LBTEST='Creatinine'; end; else do; LBTESTCD='GLUC'; LBTEST='Glucose'; end;if Value='NOT DONE' then do; LBORRES=''; LBORRESU=''; LBSTRESU=''; LBSTRESN=.; LBSTAT='NOT DONE'; end; run;proc sort data=prepared; by STUDYID USUBJID LBDTC LBTESTCD; run;data result; set prepared; by STUDYID USUBJID; retain LBSEQ; if first.USUBJID then LBSEQ=0; LBSEQ+1;run;proc sql; create table final as select DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBTEST, LBORRES, LBORRESU, LBSTRESN, LBSTRESU, LBDTC from result;quit;proc export data=final outfile="/app/output/lb.csv" dbms=csv replace; run;
yamaa