yamaa As a clinical trial statistician or programmer, challenge anything in this benchmark: describe a problem in Comments, or propose a fix with the Edit button on any source file.
- Task: Would a real study need this derivation, specified this way? Does it follow CDISC SDTM: variable names, controlled terms, record structure? Flag any rule that is ambiguous or that a sponsor would handle differently.
- Inputs: Do they look like data collected in a real trial, and do they include the situations you would expect to meet in one?
- Expected output: Would independent QC programming reproduce every value from these inputs and rules?
The other tabs beside Comments are background for assessment, not under review.
Summary
EditUnder reviewWhat this benchmark meansREADME.mdGoal: build one record per skin compartment a subject has for
the test code (IL13), carrying LBTESTCD, LBTEST, LBSPEC,
LBLOC, LBORRES, LBORRESU, LBSTRESN, and LBSTAT.
Input: one row per subject carrying cohort, lesional result, non-lesional result, and unit.
Variables:
LBTESTCDisIL13on every record.LBTESTisInterleukin 13on every record.LBSPECisSKINon every record.LBLOCisLESIONALon the lesional record andNON-LESIONALon the non-lesional record; the lesional record is built only when a cohort is recorded and is notNONAD, while every subject gets the non-lesional record.LBORRESis the collected result as reported, taken from the lesional result on lesional records and from the non-lesional result on non-lesional records; blank when the expected sample was not analysed.LBORRESUis the collected unit, taken on every record, including records whose result is blank.LBSTRESNis the numeric form of the reported result; missing when the reported result is blank. A result that is not a number stops the run.LBSTATisNOT DONEwhen the reported result is blank; blank otherwise.
Note: a blank result with NOT DONE means the compartment
exists but its sample was not analysed, while a missing lesional
record means the subject has no lesional compartment.
Inputs
Under review1 source fileinput/bx_raw.csv
Edit3 rows| STUDYID | USUBJID | COHORT | LESRES | NLESRES | RESU |
|---|---|---|---|---|---|
| CATH | AD | 4.2 | 1.1 | pg/mL | |
| CATH | PSORIASIS | — | 0.9 | pg/mL | |
| CATH | NONAD | — | 0.8 | pg/mL |
Expected output
Under review5 expected rowsexpected/lb.csv
Edit5 rows| DOMAIN | STUDYID | USUBJID | LBSEQ | LBTESTCD | LBTEST | LBSPEC | LBLOC | LBORRES | LBORRESU | LBSTRESN | LBSTAT |
|---|---|---|---|---|---|---|---|---|---|---|---|
| LB | CATH | 1 | IL13 | Interleukin 13 | SKIN | LESIONAL | 4.2 | pg/mL | 4.2 | — | |
| LB | CATH | 1 | IL13 | Interleukin 13 | SKIN | LESIONAL | — | pg/mL | — | NOT DONE | |
| LB | CATH | 2 | IL13 | Interleukin 13 | SKIN | NON-LESIONAL | 1.1 | pg/mL | 1.1 | — | |
| LB | CATH | 2 | IL13 | Interleukin 13 | SKIN | NON-LESIONAL | 0.9 | pg/mL | 0.9 | — | |
| LB | CATH | 1 | IL13 | Interleukin 13 | SKIN | NON-LESIONAL | 0.8 | pg/mL | 0.8 | — |
Comments and assessment materials
| Variable Name | Variable Label | Type | Length | Controlled Terms or Format | Origin | Core | Conversion Definition | Variable Type | Variable Order | Comments for Define |
|---|---|---|---|---|---|---|---|---|---|---|
| DOMAIN | Domain Abbreviation | Char | Assigned | Set constant value "LB". | SDTM | 1 | ||||
| STUDYID | Study Identifier | Char | Collected | Copy value from BX_RAW.STUDYID. | SDTM | 2 | ||||
| USUBJID | Unique Subject Identifier | Char | Collected | Copy value from BX_RAW.USUBJID. | SDTM | 3 | ||||
| LBTESTCD | Lab Test or Examination Short Name | Char | Assigned | Set constant value "IL13". | SDTM | 4 | ||||
| LBTEST | Lab Test or Examination Name | Char | Assigned | Set constant value "Interleukin 13". | SDTM | 5 | ||||
| LBSPEC | Specimen Material Type | Char | Assigned | Set constant value "SKIN". | SDTM | 6 | ||||
| LBLOC | Location Used for Measurement | Char | Derived | Set per constructed row; see the Row construction sheet. | SDTM | 7 | ||||
| LBSEQ | Sequence Number | Num | Derived | Row number within each group of STUDYID, USUBJID, ordered by LBLOC then LBTESTCD. | SDTM | 8 | ||||
| LBORRES | Result or Finding in Original Units | Char | Derived | Set per constructed row; see the Row construction sheet. | SDTM | 9 | ||||
| LBORRESU | Original Units | Char | Collected | Copy value from BX_RAW.RESU. | SDTM | 10 | ||||
| LBSTRESN | Numeric Result/Finding in Standard Units | Num | Derived | Copy value from LBORRES. | SDTM | 11 | ||||
| LBSTAT | Completion Status | Char | NOT DONE | Derived | If LBORRES IS NULL then "NOT DONE"; otherwise blank. | SDTM | 12 |
| Template | Include when | LBLOC | LBORRES |
|---|---|---|---|
| lesional | BX_RAW.COHORT <> 'NONAD' | Constant "LESIONAL" | Copy BX_RAW.LESRES |
| non_lesional | Constant "NON-LESIONAL" | Copy BX_RAW.NLESRES |
| Variable | Permitted values |
|---|---|
| LBSTAT | NOT DONE |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/sdtm-lb-compartments/full.md
Edit27 linesFollowing CDISC SDTM standards, use the provided BX_RAW dataset to create an LB dataset with one record per skin compartment a subject has.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBTEST, LBSPEC, LBLOC, LBORRES, LBORRESU, LBSTRESN, LBSTAT
LBTESTCD is IL13, LBTEST is Interleukin 13, and LBSPEC is SKIN on every record. LBSEQ numbers each subject's records with the lesional record first. LBLOC is LESIONAL on the lesional record and NON-LESIONAL on the non-lesional record; the lesional record is built only when a cohort is recorded and is not NONAD, while every subject gets the non-lesional record.
LBORRES is the collected result from the matching compartment, and has no value when the expected sample was not analysed. LBORRESU is the collected unit on every record, including records with no result. LBSTRESN is the numeric form of the reported result, and has no value when the reported result is blank. LBSTAT is NOT DONE when the reported result is blank, and has no value otherwise.
A blank result with NOT DONE means the compartment exists but its sample was not analysed, while a missing lesional record means the subject has no lesional compartment.
Read the source datasets from /app/input and save the completed dataset as /app/output/lb.csv.
prompts/sdtm-lb-compartments/conventions.md
Edit15 linesFollowing CDISC SDTM standards, use the provided BX_RAW dataset to create an LB dataset with one record per skin compartment a subject has.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBTEST, LBSPEC, LBLOC, LBORRES, LBORRESU, LBSTRESN, LBSTAT
LBTESTCD is IL13, LBTEST is Interleukin 13, and LBSPEC is SKIN on every record. LBLOC is LESIONAL on the lesional record and NON-LESIONAL on the non-lesional record.
LBSTAT is NOT DONE or has no value.
Read the source datasets from /app/input and save the completed dataset as /app/output/lb.csv.
prompts/sdtm-lb-compartments/brief.md
Edit9 linesFollowing CDISC SDTM standards, use the provided BX_RAW dataset to create an LB dataset with one record per skin compartment a subject has.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBTEST, LBSPEC, LBLOC, LBORRES, LBORRESU, LBSTRESN, LBSTAT
Read the source datasets from /app/input and save the completed dataset as /app/output/lb.csv.
run.R
Edit4 lineslibrary(yamaa)lb <- yamaa_domain("spec.yaml")$outputlbrun.py
Edit4 linesimport yamaalb = yamaa.yamaa_domain("spec.yaml").outputlbsolutions/sdtm-lb-compartments/result.R
Edit50 lines# Reference solution for the yamaa benchmark sdtm-lb-compartments (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)library(purrr)bx <- read_csv( "/app/input/bx_raw.csv", col_types = cols(.default = col_character()))make_records <- function(study, subj, cohort, les, nles, unit) { locs <- list() if (!is.na(cohort) && cohort != "" && cohort != "NONAD") { locs <- append(locs, list(list(loc = "LESIONAL", res = les))) } locs <- append(locs, list(list(loc = "NON-LESIONAL", res = nles))) imap(locs, function(entry, i) { res <- entry$res if (is.na(res) || res == "") res <- NA_character_ num <- suppressWarnings(as.numeric(res)) tibble( DOMAIN = "LB", STUDYID = study, USUBJID = subj, LBSEQ = as.integer(i), LBTESTCD = "IL13", LBTEST = "Interleukin 13", LBSPEC = "SKIN", LBLOC = entry$loc, LBORRES = res, LBORRESU = unit, LBSTRESN = num, LBSTAT = if_else(is.na(res), "NOT DONE", NA_character_) ) }) |> bind_rows()}lb <- pmap_dfr( list(bx$STUDYID, bx$USUBJID, bx$COHORT, bx$LESRES, bx$NLESRES, bx$RESU), make_records) |> arrange(STUDYID, USUBJID, LBSEQ)dir.create("/app/output", showWarnings = FALSE)write_csv(lb, "/app/output/lb.csv", na = "")solutions/sdtm-lb-compartments/result.py
Edit63 lines# Reference solution for the yamaa benchmark sdtm-lb-compartments (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.from pathlib import Pathimport polars as plbx = pl.read_csv("/app/input/bx_raw.csv", infer_schema=False)rows = []for r in bx.to_dicts(): study, subj = r["STUDYID"], r["USUBJID"] cohort, les, nles, unit = r["COHORT"], r["LESRES"], r["NLESRES"], r["RESU"] records = [] if cohort is not None and cohort != "" and cohort != "NONAD": records.append(("LESIONAL", les)) records.append(("NON-LESIONAL", nles)) for i, (loc, res) in enumerate(records, start=1): res_text = res if res not in (None, "") else None try: resn = float(res_text) if res_text is not None else None except ValueError: resn = None rows.append( { "DOMAIN": "LB", "STUDYID": study, "USUBJID": subj, "LBSEQ": i, "LBTESTCD": "IL13", "LBTEST": "Interleukin 13", "LBSPEC": "SKIN", "LBLOC": loc, "LBORRES": res_text, "LBORRESU": unit, "LBSTRESN": resn, "LBSTAT": "NOT DONE" if res_text is None else None, } )lb = pl.DataFrame( rows, schema={ "DOMAIN": pl.String, "STUDYID": pl.String, "USUBJID": pl.String, "LBSEQ": pl.Int64, "LBTESTCD": pl.String, "LBTEST": pl.String, "LBSPEC": pl.String, "LBLOC": pl.String, "LBORRES": pl.String, "LBORRESU": pl.String, "LBSTRESN": pl.Float64, "LBSTAT": pl.String, },).sort(["STUDYID", "USUBJID", "LBSEQ"])Path("/app/output").mkdir(parents=True, exist_ok=True)lb.write_csv("/app/output/lb.csv")solutions/sdtm-lb-compartments/result.sas
Edit23 lines/* opensas reference solution for the yamaa benchmark sdtm-lb-compartments. Reads the benchmark inputs and derives the requested output. Execute with the existing opensas runtime: opensas /app/output/result.sas. */data bx_raw; length STUDYID USUBJID COHORT LESRES NLESRES RESU $1024; infile "/app/input/bx_raw.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. COHORT : $char1024. LESRES : $char1024. NLESRES : $char1024. RESU : $char1024.;run;data prepared; set bx_raw; length DOMAIN $2 LBTESTCD LBTEST LBSPEC LBLOC LBORRES LBORRESU LBSTAT $1024; DOMAIN='LB'; LBTESTCD='IL13'; LBTEST='Interleukin 13'; LBSPEC='SKIN'; LBORRESU=RESU;if not missing(COHORT) and COHORT ne 'NONAD' then do; LBLOC='LESIONAL'; LBORRES=LESRES; LBSTRESN=input(LESRES,?? best32.); LBSTAT=''; if missing(LESRES) then LBSTAT='NOT DONE'; LOCORD=1; output; end;LBLOC='NON-LESIONAL'; LBORRES=NLESRES; LBSTRESN=input(NLESRES,?? best32.); LBSTAT=''; if missing(NLESRES) then LBSTAT='NOT DONE'; LOCORD=2; output; run;proc sort data=prepared; by STUDYID USUBJID LOCORD; run;data result; set prepared; by STUDYID USUBJID; retain LBSEQ; if first.USUBJID then LBSEQ=0; LBSEQ+1;run;proc sql; create table final as select DOMAIN, STUDYID, USUBJID, LBSEQ, LBTESTCD, LBTEST, LBSPEC, LBLOC, LBORRES, LBORRESU, LBSTRESN, LBSTAT from result;quit;proc export data=final outfile="/app/output/lb.csv" dbms=csv replace; run;
Comments
Signed in with GitHub; saved in repository Discussions