Review ask

As a clinical trial statistician or programmer, challenge anything in this benchmark: describe a problem in Comments, or propose a fix with the Edit button on any source file.

  1. Task: Would a real study need this derivation, specified this way? Does it follow CDISC SDTM: variable names, controlled terms, record structure? Flag any rule that is ambiguous or that a sponsor would handle differently.
  2. Inputs: Do they look like data collected in a real trial, and do they include the situations you would expect to meet in one?
  3. Expected output: Would independent QC programming reproduce every value from these inputs and rules?

The other tabs beside Comments are background for assessment, not under review.

Summary

Lifecycle: draftEditUnder review
What this benchmark means
README.md

Goal: create one FA record for each collected reaction finding from a long-form Operational Data Model (ODM) diary, while carrying the reported reaction name and date from the same diary occurrence.

Input: each diary occurrence contains a free-text reaction name and collection date as sibling items, plus separate occurrence, severity, and optional diameter items. One reaction can therefore produce several FA tests. A form repeat number can be reused on another diary day.

Variables:

  • FASEQ numbers the subject's findings by diary day, reaction repeat, and test order.
  • FATESTCD and FATEST identify occurrence, severity, or longest diameter from the collected result item.
  • FAOBJ is the reaction name exactly as reported in the sibling item.
  • FACAT is REACTOGENICITY for every diary finding.
  • FAORRES and FASTRESC keep the collected result; FAORRESU is mm for diameter only.
  • FASTAT is NOT DONE for a result item that is present but blank, and blank for reported results.
  • FATPT identifies the diary day, and FADTC is its collection date.

Note: a reaction without a severity or diameter item produces no such FA test, while a present item with a blank result still produces its FA record.

Inputs

Under review
1 source file

input/odm.csv

Edit
18 rows
input/odm.csv
StudyOIDMetaDataVersionOIDSubjectKeyStudyEventOIDStudyEventRepeatKeyItemGroupOIDItemGroupRepeatKeyItemOIDValue
YAMAA-01MDV.1YAMAA-01-0001DAY11IG.DM1IT.DM.SEXF
YAMAA-01MDV.1YAMAA-01-0001DAY11IG.FA1IT.FA.FAOBJInjection site redness
YAMAA-01MDV.1YAMAA-01-0001DAY11IG.FA1IT.FA.FADTC2025-01-01
YAMAA-01MDV.1YAMAA-01-0001DAY11IG.FA1IT.FA.OCCURY
YAMAA-01MDV.1YAMAA-01-0001DAY11IG.FA1IT.FA.SEVMODERATE
YAMAA-01MDV.1YAMAA-01-0001DAY11IG.FA1IT.FA.LDIAM25
YAMAA-01MDV.1YAMAA-01-0001DAY11IG.FA2IT.FA.FAOBJFatigue
YAMAA-01MDV.1YAMAA-01-0001DAY11IG.FA2IT.FA.FADTC2025-01-01
YAMAA-01MDV.1YAMAA-01-0001DAY11IG.FA2IT.FA.OCCURN
YAMAA-01MDV.1YAMAA-01-0001DAY21IG.FA1IT.FA.FAOBJInjection site redness
YAMAA-01MDV.1YAMAA-01-0001DAY21IG.FA1IT.FA.FADTC2025-01-02
YAMAA-01MDV.1YAMAA-01-0001DAY21IG.FA1IT.FA.OCCURY
YAMAA-01MDV.1YAMAA-01-0001DAY21IG.FA1IT.FA.SEV—
YAMAA-01MDV.1YAMAA-01-0001DAY21IG.FA1IT.FA.LDIAM10
YAMAA-01MDV.1YAMAA-01-0002DAY11IG.FA1IT.FA.FAOBJFatigue
YAMAA-01MDV.1YAMAA-01-0002DAY11IG.FA1IT.FA.FADTC2025-02-01
YAMAA-01MDV.1YAMAA-01-0002DAY11IG.FA1IT.FA.OCCURY
YAMAA-01MDV.1YAMAA-01-0002DAY11IG.FA1IT.FA.SEVMILD

Expected output

Under review
9 expected rows

expected/fa.csv

Edit
9 rows
expected/fa.csv
DOMAINSTUDYIDUSUBJIDFASEQFATESTCDFATESTFAOBJFACATFAORRESFAORRESUFASTRESCFASTATFATPTFADTC
FAYAMAA-011OCCUROccurrence IndicatorInjection site rednessREACTOGENICITYY—Y—END DAY 12025-01-01
FAYAMAA-012SEVSeverity/IntensityInjection site rednessREACTOGENICITYMODERATE—MODERATE—END DAY 12025-01-01
FAYAMAA-013LDIAMLongest DiameterInjection site rednessREACTOGENICITY25mm25—END DAY 12025-01-01
FAYAMAA-014OCCUROccurrence IndicatorFatigueREACTOGENICITYN—N—END DAY 12025-01-01
FAYAMAA-015OCCUROccurrence IndicatorInjection site rednessREACTOGENICITYY—Y—END DAY 22025-01-02
FAYAMAA-016SEVSeverity/IntensityInjection site rednessREACTOGENICITY———NOT DONEEND DAY 22025-01-02
FAYAMAA-017LDIAMLongest DiameterInjection site rednessREACTOGENICITY10mm10—END DAY 22025-01-02
FAYAMAA-011OCCUROccurrence IndicatorFatigueREACTOGENICITYY—Y—END DAY 12025-02-01
FAYAMAA-012SEVSeverity/IntensityFatigueREACTOGENICITYMILD—MILD—END DAY 12025-02-01
Shaded columns are derived— = missing value

Comments and assessment materials

Comments

Signed in with GitHub; saved in repository Discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Variable NameVariable LabelTypeLengthControlled Terms or FormatOriginCoreConversion DefinitionVariable TypeVariable OrderComments for Define
DOMAINDomain AbbreviationCharAssignedSet constant value "FA".SDTM1
STUDYIDStudy IdentifierCharCollectedCopy value from ODM.StudyOID.SDTM2
USUBJIDUnique Subject IdentifierCharCollectedCopy value from ODM.SubjectKey.SDTM3
FATESTCDFindings About Test Short NameCharDerivedSet per constructed row; see the Row construction sheet.SDTM4
FATESTFindings About Test NameCharDerivedSet per constructed row; see the Row construction sheet.SDTM5
FAOBJObject of the ObservationCharDerivedTake Value from the ODM_SIBLING record whose StudyOID equals ODM.StudyOID, SubjectKey equals ODM.SubjectKey, StudyEventOID equals ODM.StudyEventOID, StudyEventRepeatKey equals ODM.StudyEventRepeatKey, ItemGroupOID equals ODM.ItemGroupOID and ItemGroupRepeatKey equals ODM.ItemGroupRepeatKey, where ODM_SIBLING.ItemOID = 'IT.FA.FAOBJ', and a row with no match is an error.SDTM6
FACATCategory for Findings AboutCharAssignedSet constant value "REACTOGENICITY".SDTM7
FAORRESResult or Finding in Original UnitsCharCollectedCopy value from ODM.Value.SDTM8
FAORRESUOriginal UnitsCharDerivedIf FATESTCD = 'LDIAM' then "mm"; otherwise blank.SDTM9
FASTRESCCharacter Result/Finding in Std FormatCharDerivedCopy value from FAORRES.SDTM10
FASTATCompletion StatusCharDerivedIf FAORRES IS NULL then "NOT DONE"; otherwise blank.SDTM11
FATPTPlanned Time Point NameCharCollectedRecode ODM.StudyEventOID: "DAY1" → "END DAY 1", "DAY2" → "END DAY 2"; missing or unlisted values are errors.SDTM12
FADTCDate/Time of CollectionCharDerivedTake Value from the ODM_SIBLING record whose StudyOID equals ODM.StudyOID, SubjectKey equals ODM.SubjectKey, StudyEventOID equals ODM.StudyEventOID, StudyEventRepeatKey equals ODM.StudyEventRepeatKey, ItemGroupOID equals ODM.ItemGroupOID and ItemGroupRepeatKey equals ODM.ItemGroupRepeatKey, where ODM_SIBLING.ItemOID = 'IT.FA.FADTC', and a row with no match is an error.SDTM13
FASEQSequence NumberNumDerivedRow number within each group of STUDYID, USUBJID, ordered by EVENTORD then EVENTREPEAT then REPEATKEY then TESTORD.SDTM14
TemplateInclude whenFATESTCDFATESTTESTORD
reported-findingODM.ItemGroupOID = 'IG.FA' AND ODM.ItemOID IN ('IT.FA.OCCUR', 'IT.FA.SEV', 'IT.FA.LDIAM')Recode ODM.ItemOID: "IT.FA.OCCUR" → "OCCUR", "IT.FA.SEV" → "SEV", "IT.FA.LDIAM" → "LDIAM"; missing or unlisted values → "UNMAPPED"Recode ODM.ItemOID: "IT.FA.OCCUR" → "Occurrence Indicator", "IT.FA.SEV" → "Severity/Intensity", "IT.FA.LDIAM" → "Longest Diameter"; missing or unlisted values → "Unmapped"Recode ODM.ItemOID: "IT.FA.OCCUR" → "1", "IT.FA.SEV" → "2", "IT.FA.LDIAM" → "3"; missing or unlisted values → "99"
VersionDateAuthorDescriptionReviewerSign-off
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/sdtm-fa-odm-multitest/full.md

Edit
26 lines

Following CDISC SDTM standards, use the provided ODM dataset to create an FA dataset with one record per collected reaction finding.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, FASEQ, FATESTCD, FATEST, FAOBJ, FACAT, FAORRES, FAORRESU, FASTRESC, FASTAT, FATPT, FADTC

STUDYID is the ODM StudyOID and USUBJID is the SubjectKey, each as given.

FATESTCD and FATEST are OCCUR and Occurrence Indicator, SEV and Severity/Intensity, or LDIAM and Longest Diameter, following the collected result item. FAOBJ is the reaction name exactly as reported in the sibling item of the same diary occurrence. FACAT is REACTOGENICITY for every diary finding. FAORRES and FASTRESC keep the collected result; FAORRESU is mm for diameter only. FASTAT is NOT DONE for a result item that is present but blank, and has no value for reported results. FATPT is END DAY 1 for a DAY1 occurrence and END DAY 2 for a DAY2 occurrence, and FADTC is that day's collection date from the sibling item. A reaction without a severity or diameter item gives no such test, while a present item with a blank result still gives its record. A form repeat number reused on another diary day belongs to that day. FASEQ numbers the subject's findings by diary day, reaction repeat, and test order.

Read the source datasets from /app/input and save the completed dataset as /app/output/fa.csv.

prompts/sdtm-fa-odm-multitest/conventions.md

Edit
14 lines

Following CDISC SDTM standards, use the provided ODM dataset to create an FA dataset with one record per collected reaction finding.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, FASEQ, FATESTCD, FATEST, FAOBJ, FACAT, FAORRES, FAORRESU, FASTRESC, FASTAT, FATPT, FADTC

FATESTCD and FATEST are OCCUR and Occurrence Indicator, SEV and Severity/Intensity, or LDIAM and Longest Diameter. FACAT is REACTOGENICITY for every diary finding. FASTAT is NOT DONE or has no value. FATPT is END DAY 1 for a DAY1 occurrence and END DAY 2 for a DAY2 occurrence.

Read the source datasets from /app/input and save the completed dataset as /app/output/fa.csv.

prompts/sdtm-fa-odm-multitest/brief.md

Edit
9 lines

Following CDISC SDTM standards, use the provided ODM dataset to create an FA dataset with one record per collected reaction finding.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, FASEQ, FATESTCD, FATEST, FAOBJ, FACAT, FAORRES, FAORRESU, FASTRESC, FASTAT, FATPT, FADTC

Read the source datasets from /app/input and save the completed dataset as /app/output/fa.csv.

yamaa code

Runs the yamaa specification; its result must match the expected output

run.R

Edit
4 lines
library(yamaa)fa <- yamaa_domain("spec.yaml")$outputfa

run.py

Edit
4 lines
import yamaafa = yamaa.yamaa_domain("spec.yaml").outputfa

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/sdtm-fa-odm-multitest/result.R

Edit
93 lines
# Reference solution for the yamaa benchmark sdtm-fa-odm-multitest (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)library(purrr)odm <- read_csv(  "/app/input/odm.csv",  col_types = cols(.default = col_character())) |>  mutate(Value = na_if(Value, ""))forms <- odm |>  filter(ItemGroupOID == "IG.FA") |>  distinct(    StudyOID, SubjectKey, StudyEventOID, StudyEventRepeatKey,    ItemGroupRepeatKey  )one_form <- function(study, subj, event, rep, igrep) {  sub <- odm |>    filter(      SubjectKey == subj,      StudyEventOID == event,      StudyEventRepeatKey == rep,      ItemGroupOID == "IG.FA",      ItemGroupRepeatKey == igrep    )  present <- sub$ItemOID  get <- function(oid) {    hit <- sub |> filter(ItemOID == oid)    if (nrow(hit) == 0) return(list(present = FALSE, value = NA_character_))    list(present = TRUE, value = hit$Value[[1]])  }  obj <- get("IT.FA.FAOBJ")  if (!obj$present || is.na(obj$value)) return(tibble())  fadtc <- get("IT.FA.FADTC")$value  day <- dplyr::recode(event, DAY1 = 1L, DAY2 = 2L, .default = 99L)  fatpt <- dplyr::recode(    event, DAY1 = "END DAY 1", DAY2 = "END DAY 2", .default = NA_character_  )  specs <- list(    list(oid = "IT.FA.OCCUR", code = "OCCUR", name = "Occurrence Indicator"),    list(oid = "IT.FA.SEV", code = "SEV", name = "Severity/Intensity"),    list(oid = "IT.FA.LDIAM", code = "LDIAM", name = "Longest Diameter")  )  imap_dfr(specs, function(spec, i) {    g <- get(spec$oid)    if (!g$present) return(tibble())    blank <- is.na(g$value)    tibble(      DOMAIN = "FA",      STUDYID = study,      USUBJID = subj,      DAY = day,      REP = as.integer(igrep),      ORDER = as.integer(i - 1L),      FATESTCD = spec$code,      FATEST = spec$name,      FAOBJ = obj$value,      FACAT = "REACTOGENICITY",      FAORRES = g$value,      FAORRESU = if_else(spec$oid == "IT.FA.LDIAM", "mm", NA_character_),      FASTRESC = g$value,      FASTAT = if_else(blank, "NOT DONE", NA_character_),      FATPT = fatpt,      FADTC = fadtc    )  })}fa <- pmap_dfr(  list(    forms$StudyOID, forms$SubjectKey, forms$StudyEventOID,    forms$StudyEventRepeatKey, forms$ItemGroupRepeatKey  ),  one_form) |>  arrange(STUDYID, USUBJID, DAY, REP, ORDER) |>  group_by(STUDYID, USUBJID) |>  mutate(FASEQ = row_number()) |>  ungroup() |>  select(    DOMAIN, STUDYID, USUBJID, FASEQ, FATESTCD, FATEST, FAOBJ, FACAT,    FAORRES, FAORRESU, FASTRESC, FASTAT, FATPT, FADTC  )dir.create("/app/output", showWarnings = FALSE)write_csv(fa, "/app/output/fa.csv", na = "")

solutions/sdtm-fa-odm-multitest/result.py

Edit
115 lines
# Reference solution for the yamaa benchmark sdtm-fa-odm-multitest (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.from pathlib import Pathimport polars as plodm = pl.read_csv("/app/input/odm.csv", infer_schema=False)forms = {}study_of = {}for r in odm.to_dicts():    study_of[r["SubjectKey"]] = r["StudyOID"]    if r["ItemGroupOID"] != "IG.FA":        continue    key = (        r["SubjectKey"],        r["StudyEventOID"],        r["StudyEventRepeatKey"],        r["ItemGroupRepeatKey"],    )    forms.setdefault(key, {})[r["ItemOID"]] = r["Value"]day_order = {"DAY1": 1, "DAY2": 2}tpt_of = {"DAY1": "END DAY 1", "DAY2": "END DAY 2"}test_order = {"IT.FA.OCCUR": 0, "IT.FA.SEV": 1, "IT.FA.LDIAM": 2}test_info = {    "IT.FA.OCCUR": ("OCCUR", "Occurrence Indicator"),    "IT.FA.SEV": ("SEV", "Severity/Intensity"),    "IT.FA.LDIAM": ("LDIAM", "Longest Diameter"),}rows = []for (subj, event, _rep, repkey), form in forms.items():    obj = form.get("IT.FA.FAOBJ")    fadtc = form.get("IT.FA.FADTC")    if obj is None or obj == "":        continue    for oid in ("IT.FA.OCCUR", "IT.FA.SEV", "IT.FA.LDIAM"):        if oid not in form:            continue        raw = form[oid]        blank = raw is None or raw == ""        val = None if blank else raw        code, name = test_info[oid]        rows.append(            {                "DOMAIN": "FA",                "STUDYID": study_of[subj],                "USUBJID": subj,                "DAY": day_order.get(event, 99),                "REP": int(repkey),                "ORDER": test_order[oid],                "FATESTCD": code,                "FATEST": name,                "FAOBJ": obj,                "FACAT": "REACTOGENICITY",                "FAORRES": val,                "FAORRESU": "mm" if oid == "IT.FA.LDIAM" else None,                "FASTRESC": val,                "FASTAT": "NOT DONE" if blank else None,                "FATPT": tpt_of.get(event),                "FADTC": fadtc,            }        )fa = (    pl.DataFrame(        rows,        schema={            "DOMAIN": pl.String,            "STUDYID": pl.String,            "USUBJID": pl.String,            "DAY": pl.Int64,            "REP": pl.Int64,            "ORDER": pl.Int64,            "FATESTCD": pl.String,            "FATEST": pl.String,            "FAOBJ": pl.String,            "FACAT": pl.String,            "FAORRES": pl.String,            "FAORRESU": pl.String,            "FASTRESC": pl.String,            "FASTAT": pl.String,            "FATPT": pl.String,            "FADTC": pl.String,        },    )    .sort(["STUDYID", "USUBJID", "DAY", "REP", "ORDER"])    .with_columns(        pl.int_range(1, pl.len() + 1).over(["STUDYID", "USUBJID"]).alias("FASEQ")    )    .select(        "DOMAIN",        "STUDYID",        "USUBJID",        "FASEQ",        "FATESTCD",        "FATEST",        "FAOBJ",        "FACAT",        "FAORRES",        "FAORRESU",        "FASTRESC",        "FASTAT",        "FATPT",        "FADTC",    ))Path("/app/output").mkdir(parents=True, exist_ok=True)fa.write_csv("/app/output/fa.csv")

solutions/sdtm-fa-odm-multitest/result.sas

Edit
42 lines
/* opensas reference solution for the yamaa benchmark sdtm-fa-odm-multitest.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data odm;  length StudyOID MetaDataVersionOID SubjectKey StudyEventOID StudyEventRepeatKey ItemGroupOID ItemGroupRepeatKey ItemOID Value $1024;  infile "/app/input/odm.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input StudyOID : $char1024. MetaDataVersionOID : $char1024. SubjectKey : $char1024. StudyEventOID : $char1024. StudyEventRepeatKey : $char1024. ItemGroupOID : $char1024. ItemGroupRepeatKey : $char1024. ItemOID : $char1024. Value : $char1024.;run;proc sort data=odm out=ordered; by StudyOID SubjectKey StudyEventOID StudyEventRepeatKey ItemGroupRepeatKey; run;data siblings;  length FAOBJ FADTC $1024;  retain FAOBJ FADTC;  set ordered;  where ItemGroupOID='IG.FA';  by StudyOID SubjectKey StudyEventOID StudyEventRepeatKey ItemGroupRepeatKey;  if first.ItemGroupRepeatKey then do;    FAOBJ=''; FADTC='';  end;    if ItemOID='IT.FA.FAOBJ' then FAOBJ=Value;  if ItemOID='IT.FA.FADTC' then FADTC=Value;  if last.ItemGroupRepeatKey then output;run;proc sql;create table joined as select a.*,b.FAOBJ,b.FADTC from odm a inner join siblings b on a.StudyOID=b.StudyOID and a.SubjectKey=b.SubjectKey and a.StudyEventOID=b.StudyEventOID and a.StudyEventRepeatKey=b.StudyEventRepeatKey and a.ItemGroupRepeatKey=b.ItemGroupRepeatKey where a.ItemOID in ('IT.FA.OCCUR','IT.FA.SEV','IT.FA.LDIAM'); quit;data prepared; set joined; length DOMAIN $2 STUDYID USUBJID FATESTCD FATEST FACAT FAORRES FAORRESU FASTRESC FASTAT FATPT $1024;DOMAIN='FA'; STUDYID=StudyOID; USUBJID=SubjectKey; FACAT='REACTOGENICITY'; FAORRES=Value; FASTRESC=Value; FAORRESU=''; FASTAT=''; if missing(Value) then FASTAT='NOT DONE';if ItemOID='IT.FA.OCCUR' then do; FATESTCD='OCCUR'; FATEST='Occurrence Indicator'; TESTORD=1; end;if ItemOID='IT.FA.SEV' then do; FATESTCD='SEV'; FATEST='Severity/Intensity'; TESTORD=2; end;if ItemOID='IT.FA.LDIAM' then do; FATESTCD='LDIAM'; FATEST='Longest Diameter'; FAORRESU='mm'; TESTORD=3; end;FATPT='END DAY 1'; if StudyEventOID='DAY2' then FATPT='END DAY 2'; DAYORD=input(compress(StudyEventOID,,'kd'),best32.); REP=input(ItemGroupRepeatKey,best32.); run;proc sort data=prepared; by STUDYID USUBJID DAYORD REP TESTORD; run;data result; set prepared; by STUDYID USUBJID; retain FASEQ;  if first.USUBJID then FASEQ=0;  FASEQ+1;run;proc sql;  create table final as select DOMAIN, STUDYID, USUBJID, FASEQ, FATESTCD, FATEST, FAOBJ, FACAT, FAORRES, FAORRESU, FASTRESC, FASTAT, FATPT, FADTC from result;quit;proc export data=final outfile="/app/output/fa.csv" dbms=csv replace; run;