Review ask

As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.

We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.

Summary

Lifecycle: draftEditUnder review
What this benchmark means
README.md

Public entries: Python and R. Passing coverage requires an installed inventory.

Goal: build one record per collected glucose and creatinine result, with per-test submission metadata: each LBTESTCD value gets its own codelist and origin for LBORRESU, overriding the shared column-level declaration.

Input: long-form Operational Data Model (ODM) rows, one row per collected item, with the collected entry in the value field. Glucose rows carry one item, creatinine rows carry another, and the collection date rides along in the same visit group.

Variables:

  • LBTESTCD is GLUC for glucose rows and CREAT for creatinine rows.
  • LBTEST is Glucose when the test code is GLUC and Creatinine when it is CREAT.
  • LBORRES is the collected entry, kept exactly as reported.
  • LBORRESU is mg/dL for every record. Its submission metadata is stated once for the whole column and overridden for each test code: glucose records name the glucose codelist with a vendor origin, creatinine records the creatinine codelist with an investigator origin.
  • LBSTRESN is the numeric form of the collected entry; missing when the entry is not a number.
  • LBSTRESU is mg/dL for every record.
  • LBDTC is the collection date from the same visit group.

Note: a test with no collected entry produces no record, so the records are only results actually reported. The per-value submission metadata is validated when the specification loads; the engine output is unaffected by it.

The case runs submission-spec.yaml, an independently completed submission layer over the original executable spec.yaml, with submission-project.yaml, the project file that declares the unit codelists both layers bind, so every unit is checked against its list. The completion adds complete dataset/column metadata and retains the glucose and creatinine codelist declarations. It restates both test codes' entries, since a later layer replaces the whole list, and explicitly uses Sponsor for the creatinine Assigned origin: SDTM admits Vendor or Sponsor for Assigned, whereas the original family-independent loader fixture retains its Investigator declaration. Installed Define tests also exercise that original declaration's family-admission failure.

The installed Define tests' fixed-time witness reads the unchanged expected/lb.csv only when Dataset-JSON is explicitly requested; it does not execute the specification or open its ODM input. Its independently authored XML structure is validated against the bundled offline Define-XML XSD. The original declarations, input, CSV expectation and reference loader controls retain their existing meaning.

Inputs

Under review
1 source file

input/odm.csv

Edit
5 rows
input/odm.csv
StudyOIDMetaDataVersionOIDSubjectKeyStudyEventOIDStudyEventRepeatKeyItemGroupOIDItemGroupRepeatKeyItemOIDValue
STUDY01MDV.1001BASELINE1IG.LB1IT.LB.LBDTC2025-01-01
STUDY01MDV.1001BASELINE1IG.LB1IT.LB.GLUC95
STUDY01MDV.1001BASELINE1IG.LB1IT.LB.CREAT0.9
STUDY01MDV.1002BASELINE1IG.LB1IT.LB.LBDTC2025-01-02
STUDY01MDV.1002BASELINE1IG.LB1IT.LB.GLUC101

Expected output

Under review
3 expected rows

expected/lb.csv

Edit
3 rows
expected/lb.csv
DOMAINSTUDYIDUSUBJIDLBSEQLBTESTCDLBTESTLBORRESLBORRESULBSTRESNLBSTRESULBDTC
LBSTUDY012GLUCGlucose95mg/dL95mg/dL2025-01-01
LBSTUDY011GLUCGlucose101mg/dL101mg/dL2025-01-02
LBSTUDY011CREATCreatinine0.9mg/dL0.9mg/dL2025-01-01
Shaded columns are derived— = missing value

Human review and assessment

Your review

3 questions and your comments · View benchmark discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Variable NameVariable LabelTypeLengthControlled Terms or FormatOriginCoreConversion DefinitionVariable TypeVariable OrderComments for Define
DOMAIN2AssignedReqNo derivation in this specification; inherited from its parent.SDTM1
STUDYID20CollectedReqNo derivation in this specification; inherited from its parent.SDTM2
USUBJID20CollectedReqNo derivation in this specification; inherited from its parent.SDTM3
LBSEQ8DerivedReqSequence within each subject by collection date and test codeSDTM4
LBTESTCD8AssignedReqNo derivation in this specification; inherited from its parent.SDTM5
LBTEST40AssignedReqNo derivation in this specification; inherited from its parent.SDTM6
LBORRES20CollectedExpNo derivation in this specification; inherited from its parent.SDTM7
LBORRESU8AssignedExpNo derivation in this specification; inherited from its parent.SDTM8
LBSTRESN8DerivedExpConvert the original result to the declared numeric typeSDTM9
LBSTRESU8AssignedExpNo derivation in this specification; inherited from its parent.SDTM10
LBDTCCollectedExpNo derivation in this specification; inherited from its parent.SDTM11
VersionDateAuthorDescriptionReviewerSign-off
1.72026-10-20yamaa docs buildRead a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default
1.62026-10-17yamaa docs buildDescribe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight
1.52026-10-17yamaa docs buildName collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part
1.42026-10-15yamaa docs buildState datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/sdtm-lb-metadata/full.md

Edit
20 lines

Following CDISC SDTM standards, use the provided ODM dataset to create an LB dataset with one record per collected glucose or creatinine result.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBTEST, LBORRES, LBORRESU, LBSTRESN, LBSTRESU, LBDTC

DOMAIN is "LB". STUDYID is the study OID and USUBJID the subject key, each as given in ODM. Each collected glucose result and each collected creatinine result gives one record, and a test with no collected entry gives no record. LBTESTCD is "GLUC" for a glucose result and "CREAT" for a creatinine result. LBTEST is "Glucose" for "GLUC" and "Creatinine" for "CREAT". LBORRES is the collected entry, kept exactly as reported. LBORRESU is "mg/dL" for every record. LBSTRESN is the numeric form of the collected entry and has no value when the entry is not a number. LBSTRESU is "mg/dL" for every record. LBDTC is the collection date collected in the same visit group. LBSEQ numbers each subject's records from 1 by collection date, then by test code.

Read the source datasets from /app/input and save the completed dataset as /app/output/lb.csv.

prompts/sdtm-lb-metadata/conventions.md

Edit
12 lines

Following CDISC SDTM standards, use the provided ODM dataset to create an LB dataset with one record per collected glucose or creatinine result.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBTEST, LBORRES, LBORRESU, LBSTRESN, LBSTRESU, LBDTC

DOMAIN is "LB". LBTESTCD is "GLUC" for a glucose result and "CREAT" for a creatinine result. LBTEST is "Glucose" for "GLUC" and "Creatinine" for "CREAT".

Read the source datasets from /app/input and save the completed dataset as /app/output/lb.csv.

prompts/sdtm-lb-metadata/brief.md

Edit
9 lines

Following CDISC SDTM standards, use the provided ODM dataset to create an LB dataset with one record per collected glucose or creatinine result.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBTEST, LBORRES, LBORRESU, LBSTRESN, LBSTRESU, LBDTC

Read the source datasets from /app/input and save the completed dataset as /app/output/lb.csv.

yamaa code

Public calls generated from benchmark.yaml; results must match the expected output

R

Edit
3 lines
library(yamaa)yamaa_project("submission-project.yaml")$spec("LB")$run()$output

Python

Edit
3 lines
import yamaayamaa.project("submission-project.yaml").spec("LB").run().output

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/sdtm-lb-metadata/result.R

Edit
52 lines
# Reference solution for the yamaa benchmark sdtm-lb-metadata (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)library(tidyr)odm <- read_csv(  "/app/input/odm.csv",  col_types = cols(.default = col_character()))key_cols <- c(  "StudyOID", "SubjectKey", "StudyEventOID", "StudyEventRepeatKey",  "ItemGroupOID", "ItemGroupRepeatKey")dates <- odm |>  filter(ItemOID == "IT.LB.LBDTC") |>  select(all_of(key_cols), LBDTC = Value)tests <- odm |>  filter(ItemOID %in% c("IT.LB.GLUC", "IT.LB.CREAT")) |>  left_join(dates, by = key_cols) |>  mutate(    LBTESTCD = if_else(ItemOID == "IT.LB.GLUC", "GLUC", "CREAT"),    LBTEST = if_else(ItemOID == "IT.LB.GLUC", "Glucose", "Creatinine"),    LBORRES = na_if(Value, ""),  ) |>  filter(!is.na(LBORRES)) |>  mutate(    DOMAIN = "LB",    STUDYID = StudyOID,    USUBJID = SubjectKey,    LBORRESU = "mg/dL",    LBSTRESN = suppressWarnings(as.numeric(LBORRES)),    LBSTRESU = "mg/dL",  ) |>  arrange(STUDYID, USUBJID, LBDTC, LBTESTCD) |>  group_by(STUDYID, USUBJID) |>  mutate(LBSEQ = row_number()) |>  ungroup() |>  select(    DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBTEST, LBORRES,    LBORRESU, LBSTRESN, LBSTRESU, LBDTC  )dir.create("/app/output", showWarnings = FALSE)write_csv(tests, "/app/output/lb.csv", na = "")

solutions/sdtm-lb-metadata/result.py

Edit
115 lines
# Reference solution for the yamaa benchmark sdtm-lb-metadata (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1."""Build the SDTM LB dataset from the ODM extract.One record per collected glucose or creatinine result. Reads/app/input/odm.csv and writes /app/output/lb.csv."""import csvfrom collections import OrderedDictINPUT_CSV = "/app/input/odm.csv"OUTPUT_CSV = "/app/output/lb.csv"TESTS = OrderedDict(    [        ("IT.LB.GLUC", ("GLUC", "Glucose")),        ("IT.LB.CREAT", ("CREAT", "Creatinine")),    ])DATE_ITEM = "IT.LB.LBDTC"COLUMNS = [    "DOMAIN",    "STUDYID",    "USUBJID",    "LBSEQ",    "LBTESTCD",    "LBTEST",    "LBORRES",    "LBORRESU",    "LBSTRESN",    "LBSTRESU",    "LBDTC",]def numeric_form(text):    """Return the numeric rendering of text, or '' when not a number."""    try:        value = float(text)    except (TypeError, ValueError):        return ""    if value == int(value):        return str(int(value))    return repr(value)def main():    with open(INPUT_CSV, newline="", encoding="utf-8") as handle:        rows = list(csv.DictReader(handle))    # Group items by subject and visit group (one collection event).    groups = OrderedDict()    for row in rows:        key = (            row["StudyOID"],            row["SubjectKey"],            row["StudyEventOID"],            row["StudyEventRepeatKey"],            row["ItemGroupOID"],            row["ItemGroupRepeatKey"],        )        groups.setdefault(key, {})[row["ItemOID"]] = row["Value"]    records = []    for (study_oid, subject_key, *_), items in groups.items():        lbdtc = items.get(DATE_ITEM, "")        for item_oid, (testcd, test) in TESTS.items():            if item_oid not in items:                continue            entry = items[item_oid]            records.append(                {                    "DOMAIN": "LB",                    "STUDYID": study_oid,                    "USUBJID": subject_key,                    "LBTESTCD": testcd,                    "LBTEST": test,                    "LBORRES": entry,                    "LBORRESU": "mg/dL",                    "LBSTRESN": numeric_form(entry),                    "LBSTRESU": "mg/dL",                    "LBDTC": lbdtc,                }            )    # LBSEQ numbers each subject's records from 1 by collection date,    # then by test code.    by_subject = {}    for record in records:        by_subject.setdefault(record["USUBJID"], []).append(record)    ordered = []    for subject in by_subject:        subject_records = sorted(            by_subject[subject],            key=lambda r: (r["LBDTC"], r["LBTESTCD"]),        )        for index, record in enumerate(subject_records, start=1):            record["LBSEQ"] = index        ordered.extend(subject_records)    with open(OUTPUT_CSV, "w", newline="", encoding="utf-8") as handle:        writer = csv.DictWriter(handle, fieldnames=COLUMNS)        writer.writeheader()        for record in ordered:            writer.writerow({column: record[column] for column in COLUMNS})if __name__ == "__main__":    main()

solutions/sdtm-lb-metadata/result.sas

Edit
39 lines
/* opensas reference solution for the yamaa benchmark sdtm-lb-metadata.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data odm;  length StudyOID MetaDataVersionOID SubjectKey StudyEventOID StudyEventRepeatKey ItemGroupOID ItemGroupRepeatKey ItemOID Value $1024;  infile "/app/input/odm.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input StudyOID : $char1024. MetaDataVersionOID : $char1024. SubjectKey : $char1024. StudyEventOID : $char1024. StudyEventRepeatKey : $char1024. ItemGroupOID : $char1024. ItemGroupRepeatKey : $char1024. ItemOID : $char1024. Value : $char1024.;run;proc sort data=odm out=ordered; by StudyOID SubjectKey StudyEventOID StudyEventRepeatKey ItemGroupRepeatKey; run;data dates;  length LBDTC $1024;  retain LBDTC;  set ordered;  by StudyOID SubjectKey StudyEventOID StudyEventRepeatKey ItemGroupRepeatKey;  if first.ItemGroupRepeatKey then do;    LBDTC='';  end;    if ItemOID='IT.LB.LBDTC' then LBDTC=Value;  if last.ItemGroupRepeatKey then output;run;proc sql;create table joined as select a.*,b.LBDTC from odm a inner join dates b on a.StudyOID=b.StudyOID and a.SubjectKey=b.SubjectKey and a.StudyEventOID=b.StudyEventOID and a.StudyEventRepeatKey=b.StudyEventRepeatKey and a.ItemGroupRepeatKey=b.ItemGroupRepeatKey where a.ItemOID in ('IT.LB.GLUC','IT.LB.CREAT') and not missing(a.Value); quit;data prepared; set joined; length DOMAIN $2 STUDYID USUBJID LBTESTCD LBTEST LBORRES LBORRESU LBSTRESU LBSTAT $1024;DOMAIN='LB'; STUDYID=StudyOID; USUBJID=SubjectKey; LBORRES=Value; LBORRESU='mg/dL'; LBSTRESU='mg/dL'; LBSTRESN=input(Value,?? best32.); LBSTAT='';if ItemOID='IT.LB.CREAT' then do; LBTESTCD='CREAT'; LBTEST='Creatinine'; end; else do; LBTESTCD='GLUC'; LBTEST='Glucose'; end;if Value='NOT DONE' then do; LBORRES=''; LBORRESU=''; LBSTRESU=''; LBSTRESN=.; LBSTAT='NOT DONE'; end; run;proc sort data=prepared; by STUDYID USUBJID LBDTC LBTESTCD; run;data result; set prepared; by STUDYID USUBJID; retain LBSEQ;  if first.USUBJID then LBSEQ=0;  LBSEQ+1;run;proc sql;  create table final as select DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBTEST, LBORRES, LBORRESU, LBSTRESN, LBSTRESU, LBDTC from result;quit;proc export data=final outfile="/app/output/lb.csv" dbms=csv replace; run;