Review ask

As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.

We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.

Summary

Lifecycle: draftEditUnder review
What this benchmark means
README.md

Public entries: Python and R. Passing coverage requires an installed inventory.

Goal: create one FA record for each collected reaction finding from a long-form Operational Data Model (ODM) diary, while carrying the reported reaction name and date from the same diary occurrence.

Input: each diary occurrence contains a free-text reaction name and collection date as sibling items, plus separate occurrence, severity, and optional diameter items. One reaction can therefore produce several FA tests. A form repeat number can be reused on another diary day.

Variables:

  • FASEQ numbers the subject's findings by diary day, reaction repeat, and test order.
  • FATESTCD and FATEST identify occurrence, severity, or longest diameter from the collected result item.
  • FAOBJ is the reaction name exactly as reported in the sibling item.
  • FACAT is REACTOGENICITY for every diary finding.
  • FAORRES and FASTRESC keep the collected result; FAORRESU is mm for diameter only.
  • FASTAT is NOT DONE for a result item that is present but blank, and blank for reported results.
  • FATPT identifies the diary day, and FADTC is its collection date.

Note: a reaction without a severity or diameter item produces no such FA test, while a present item with a blank result still produces its FA record.

Inputs

Under review
1 source file

input/odm.csv

Edit
18 rows
input/odm.csv
StudyOIDMetaDataVersionOIDSubjectKeyStudyEventOIDStudyEventRepeatKeyItemGroupOIDItemGroupRepeatKeyItemOIDValue
YAMAA-01MDV.1YAMAA-01-0001DAY11IG.DM1IT.DM.SEXF
YAMAA-01MDV.1YAMAA-01-0001DAY11IG.FA1IT.FA.FAOBJInjection site redness
YAMAA-01MDV.1YAMAA-01-0001DAY11IG.FA1IT.FA.FADTC2025-01-01
YAMAA-01MDV.1YAMAA-01-0001DAY11IG.FA1IT.FA.OCCURY
YAMAA-01MDV.1YAMAA-01-0001DAY11IG.FA1IT.FA.SEVMODERATE
YAMAA-01MDV.1YAMAA-01-0001DAY11IG.FA1IT.FA.LDIAM25
YAMAA-01MDV.1YAMAA-01-0001DAY11IG.FA2IT.FA.FAOBJFatigue
YAMAA-01MDV.1YAMAA-01-0001DAY11IG.FA2IT.FA.FADTC2025-01-01
YAMAA-01MDV.1YAMAA-01-0001DAY11IG.FA2IT.FA.OCCURN
YAMAA-01MDV.1YAMAA-01-0001DAY21IG.FA1IT.FA.FAOBJInjection site redness
YAMAA-01MDV.1YAMAA-01-0001DAY21IG.FA1IT.FA.FADTC2025-01-02
YAMAA-01MDV.1YAMAA-01-0001DAY21IG.FA1IT.FA.OCCURY
YAMAA-01MDV.1YAMAA-01-0001DAY21IG.FA1IT.FA.SEV—
YAMAA-01MDV.1YAMAA-01-0001DAY21IG.FA1IT.FA.LDIAM10
YAMAA-01MDV.1YAMAA-01-0002DAY11IG.FA1IT.FA.FAOBJFatigue
YAMAA-01MDV.1YAMAA-01-0002DAY11IG.FA1IT.FA.FADTC2025-02-01
YAMAA-01MDV.1YAMAA-01-0002DAY11IG.FA1IT.FA.OCCURY
YAMAA-01MDV.1YAMAA-01-0002DAY11IG.FA1IT.FA.SEVMILD

Expected output

Under review
9 expected rows

expected/fa.csv

Edit
9 rows
expected/fa.csv
DOMAINSTUDYIDUSUBJIDFASEQFATESTCDFATESTFAOBJFACATFAORRESFAORRESUFASTRESCFASTATFATPTFADTC
FAYAMAA-011OCCUROccurrence IndicatorInjection site rednessREACTOGENICITYY—Y—END DAY 12025-01-01
FAYAMAA-012SEVSeverity/IntensityInjection site rednessREACTOGENICITYMODERATE—MODERATE—END DAY 12025-01-01
FAYAMAA-013LDIAMLongest DiameterInjection site rednessREACTOGENICITY25mm25—END DAY 12025-01-01
FAYAMAA-014OCCUROccurrence IndicatorFatigueREACTOGENICITYN—N—END DAY 12025-01-01
FAYAMAA-015OCCUROccurrence IndicatorInjection site rednessREACTOGENICITYY—Y—END DAY 22025-01-02
FAYAMAA-016SEVSeverity/IntensityInjection site rednessREACTOGENICITY———NOT DONEEND DAY 22025-01-02
FAYAMAA-017LDIAMLongest DiameterInjection site rednessREACTOGENICITY10mm10—END DAY 22025-01-02
FAYAMAA-011OCCUROccurrence IndicatorFatigueREACTOGENICITYY—Y—END DAY 12025-02-01
FAYAMAA-012SEVSeverity/IntensityFatigueREACTOGENICITYMILD—MILD—END DAY 12025-02-01
Shaded columns are derived— = missing value

Human review and assessment

Your review

3 questions and your comments · View benchmark discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Variable NameVariable LabelTypeLengthControlled Terms or FormatOriginCoreConversion DefinitionVariable TypeVariable OrderComments for Define
DOMAINDomain AbbreviationCharAssignedSet constant value "FA".SDTM1
STUDYIDStudy IdentifierCharCollectedCopy value from ODM.StudyOID.SDTM2
USUBJIDUnique Subject IdentifierCharCollectedCopy value from ODM.SubjectKey.SDTM3
FATESTCDFindings About Test Short NameCharDerivedSet per constructed row; see the Row construction sheet.SDTM4
FATESTFindings About Test NameCharDerivedSet per constructed row; see the Row construction sheet.SDTM5
FAOBJObject of the ObservationCharDerivedTake Value from the ODM_SIBLING record whose StudyOID equals ODM.StudyOID, SubjectKey equals ODM.SubjectKey, StudyEventOID equals ODM.StudyEventOID, StudyEventRepeatKey equals ODM.StudyEventRepeatKey, ItemGroupOID equals ODM.ItemGroupOID and ItemGroupRepeatKey equals ODM.ItemGroupRepeatKey, where ODM_SIBLING.ItemOID = 'IT.FA.FAOBJ', and a row with no match is an error.SDTM6
FACATCategory for Findings AboutCharAssignedSet constant value "REACTOGENICITY".SDTM7
FAORRESResult or Finding in Original UnitsCharCollectedCopy value from ODM.Value.SDTM8
FAORRESUOriginal UnitsCharDerivedIf FATESTCD = 'LDIAM' then "mm"; otherwise blank.SDTM9
FASTRESCCharacter Result/Finding in Std FormatCharDerivedCopy value from FAORRES.SDTM10
FASTATCompletion StatusCharDerivedIf FAORRES IS NULL then "NOT DONE"; otherwise blank.SDTM11
FATPTPlanned Time Point NameCharCollectedRecode ODM.StudyEventOID: "DAY1" → "END DAY 1", "DAY2" → "END DAY 2"; missing values stay missing; unlisted values are errors.SDTM12
FADTCDate/Time of CollectionCharDerivedTake Value from the ODM_SIBLING record whose StudyOID equals ODM.StudyOID, SubjectKey equals ODM.SubjectKey, StudyEventOID equals ODM.StudyEventOID, StudyEventRepeatKey equals ODM.StudyEventRepeatKey, ItemGroupOID equals ODM.ItemGroupOID and ItemGroupRepeatKey equals ODM.ItemGroupRepeatKey, where ODM_SIBLING.ItemOID = 'IT.FA.FADTC', and a row with no match is an error.SDTM13
FASEQSequence NumberNumDerivedRow number within each group of STUDYID, USUBJID, ordered by EVENTORD then EVENTREPEAT then REPEATKEY then TESTORD.SDTM14
TemplateInclude whenFATESTCDFATESTTESTORD
reported-findingODM.ItemGroupOID = 'IG.FA' AND ODM.ItemOID IN ('IT.FA.OCCUR', 'IT.FA.SEV', 'IT.FA.LDIAM')Recode ODM.ItemOID: "IT.FA.OCCUR" → "OCCUR", "IT.FA.SEV" → "SEV", "IT.FA.LDIAM" → "LDIAM"; missing or unlisted values → "UNMAPPED"Recode ODM.ItemOID: "IT.FA.OCCUR" → "Occurrence Indicator", "IT.FA.SEV" → "Severity/Intensity", "IT.FA.LDIAM" → "Longest Diameter"; missing or unlisted values → "Unmapped"Recode ODM.ItemOID: "IT.FA.OCCUR" → "1", "IT.FA.SEV" → "2", "IT.FA.LDIAM" → "3"; missing or unlisted values → "99"
VersionDateAuthorDescriptionReviewerSign-off
1.72026-10-20yamaa docs buildRead a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default
1.62026-10-17yamaa docs buildDescribe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight
1.52026-10-17yamaa docs buildName collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part
1.42026-10-15yamaa docs buildState datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/sdtm-fa-odm-multitest/full.md

Edit
30 lines

Following CDISC SDTM standards, use the provided ODM dataset to create an FA dataset with one record per collected reaction finding.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, FASEQ, FATESTCD, FATEST, FAOBJ, FACAT, FAORRES, FAORRESU, FASTRESC, FASTAT, FATPT, FADTC

DOMAIN is "FA". STUDYID is the ODM study OID and USUBJID is the ODM subject key, each as given. One record is written for each occurrence, severity, or longest diameter result item collected for a reaction. A reaction with no severity or diameter item has no such record, while a result item that is present with a blank result still gives its record. FATESTCD is "OCCUR" for the occurrence result, "SEV" for the severity result, and "LDIAM" for the longest diameter result, and FATEST is "Occurrence Indicator", "Severity/Intensity", and "Longest Diameter" for the same results. FAOBJ is the reaction name exactly as reported in the reaction name item of the same diary occurrence. FACAT is "REACTOGENICITY" for every record. FAORRES is the result as collected, and FASTRESC holds the same result. FAORRESU is "mm" on a longest diameter record and has no value on an occurrence or severity record. FASTAT is "NOT DONE" when a result item is present but its result is blank, and has no value when a result is reported. FATPT is "END DAY 1" for diary day DAY1 and "END DAY 2" for diary day DAY2. FADTC is the collection date collected for the same diary occurrence. FASEQ numbers a subject's records from 1 by diary day, then by the diary day's repeat, then by reaction repeat, then by test order, occurrence first, then severity, then longest diameter.

Read the source datasets from /app/input and save the completed dataset as /app/output/fa.csv.

prompts/sdtm-fa-odm-multitest/conventions.md

Edit
15 lines

Following CDISC SDTM standards, use the provided ODM dataset to create an FA dataset with one record per collected reaction finding.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, FASEQ, FATESTCD, FATEST, FAOBJ, FACAT, FAORRES, FAORRESU, FASTRESC, FASTAT, FATPT, FADTC

DOMAIN is "FA". FATESTCD is "OCCUR", "SEV", or "LDIAM", and FATEST is "Occurrence Indicator", "Severity/Intensity", and "Longest Diameter". FACAT is "REACTOGENICITY" for every record. FASTAT is "NOT DONE" or has no value. FATPT is "END DAY 1" for diary day DAY1 and "END DAY 2" for diary day DAY2.

Read the source datasets from /app/input and save the completed dataset as /app/output/fa.csv.

prompts/sdtm-fa-odm-multitest/brief.md

Edit
9 lines

Following CDISC SDTM standards, use the provided ODM dataset to create an FA dataset with one record per collected reaction finding.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, FASEQ, FATESTCD, FATEST, FAOBJ, FACAT, FAORRES, FAORRESU, FASTRESC, FASTAT, FATPT, FADTC

Read the source datasets from /app/input and save the completed dataset as /app/output/fa.csv.

yamaa code

Public calls generated from benchmark.yaml; results must match the expected output

R

Edit
3 lines
library(yamaa)yamaa_spec("spec.yaml")$run()$output

Python

Edit
3 lines
import yamaayamaa.spec("spec.yaml").run().output

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/sdtm-fa-odm-multitest/result.R

Edit
93 lines
# Reference solution for the yamaa benchmark sdtm-fa-odm-multitest (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)library(purrr)odm <- read_csv(  "/app/input/odm.csv",  col_types = cols(.default = col_character())) |>  mutate(Value = na_if(Value, ""))forms <- odm |>  filter(ItemGroupOID == "IG.FA") |>  distinct(    StudyOID, SubjectKey, StudyEventOID, StudyEventRepeatKey,    ItemGroupRepeatKey  )one_form <- function(study, subj, event, rep, igrep) {  sub <- odm |>    filter(      SubjectKey == subj,      StudyEventOID == event,      StudyEventRepeatKey == rep,      ItemGroupOID == "IG.FA",      ItemGroupRepeatKey == igrep    )  present <- sub$ItemOID  get <- function(oid) {    hit <- sub |> filter(ItemOID == oid)    if (nrow(hit) == 0) return(list(present = FALSE, value = NA_character_))    list(present = TRUE, value = hit$Value[[1]])  }  obj <- get("IT.FA.FAOBJ")  if (!obj$present || is.na(obj$value)) return(tibble())  fadtc <- get("IT.FA.FADTC")$value  day <- dplyr::recode(event, DAY1 = 1L, DAY2 = 2L, .default = 99L)  fatpt <- dplyr::recode(    event, DAY1 = "END DAY 1", DAY2 = "END DAY 2", .default = NA_character_  )  specs <- list(    list(oid = "IT.FA.OCCUR", code = "OCCUR", name = "Occurrence Indicator"),    list(oid = "IT.FA.SEV", code = "SEV", name = "Severity/Intensity"),    list(oid = "IT.FA.LDIAM", code = "LDIAM", name = "Longest Diameter")  )  imap_dfr(specs, function(spec, i) {    g <- get(spec$oid)    if (!g$present) return(tibble())    blank <- is.na(g$value)    tibble(      DOMAIN = "FA",      STUDYID = study,      USUBJID = subj,      DAY = day,      REP = as.integer(igrep),      ORDER = as.integer(i - 1L),      FATESTCD = spec$code,      FATEST = spec$name,      FAOBJ = obj$value,      FACAT = "REACTOGENICITY",      FAORRES = g$value,      FAORRESU = if_else(spec$oid == "IT.FA.LDIAM", "mm", NA_character_),      FASTRESC = g$value,      FASTAT = if_else(blank, "NOT DONE", NA_character_),      FATPT = fatpt,      FADTC = fadtc    )  })}fa <- pmap_dfr(  list(    forms$StudyOID, forms$SubjectKey, forms$StudyEventOID,    forms$StudyEventRepeatKey, forms$ItemGroupRepeatKey  ),  one_form) |>  arrange(STUDYID, USUBJID, DAY, REP, ORDER) |>  group_by(STUDYID, USUBJID) |>  mutate(FASEQ = row_number()) |>  ungroup() |>  select(    DOMAIN, STUDYID, USUBJID, FASEQ, FATESTCD, FATEST, FAOBJ, FACAT,    FAORRES, FAORRESU, FASTRESC, FASTAT, FATPT, FADTC  )dir.create("/app/output", showWarnings = FALSE)write_csv(fa, "/app/output/fa.csv", na = "")

solutions/sdtm-fa-odm-multitest/result.py

Edit
113 lines
# Reference solution for the yamaa benchmark sdtm-fa-odm-multitest (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1."""Build the SDTM FA (Findings About) dataset from the ODM extract.One record per collected reaction finding: for each diary occurrence(an IG.FA item group repeat within a study event), one record for eachof the occurrence, severity, and longest diameter result items that wascollected. A result item present with a blank result still gives itsrecord, flagged FASTAT = "NOT DONE"."""import csvimport reINPUT_ODM = "/app/input/odm.csv"OUTPUT_FA = "/app/output/fa.csv"COLUMNS = [    "DOMAIN", "STUDYID", "USUBJID", "FASEQ", "FATESTCD", "FATEST",    "FAOBJ", "FACAT", "FAORRES", "FAORRESU", "FASTRESC", "FASTAT",    "FATPT", "FADTC",]# Result items in test order: (item OID suffix, FATESTCD, FATEST).TESTS = [    ("OCCUR", "OCCUR", "Occurrence Indicator"),    ("SEV", "SEV", "Severity/Intensity"),    ("LDIAM", "LDIAM", "Longest Diameter"),]def day_number(study_event_oid):    """Numeric part of a diary day OID such as DAY1 or DAY2."""    match = re.search(r"(\d+)\s*$", study_event_oid)    return int(match.group(1)) if match else 0def main():    with open(INPUT_ODM, newline="", encoding="utf-8") as handle:        rows = list(csv.DictReader(handle))    # Group the IG.FA items by diary occurrence: subject, study event    # (diary day) and its repeat, and the item group (reaction) repeat.    occurrences = {}    subject_order = []    for row in rows:        if row["ItemGroupOID"] != "IG.FA":            continue        key = (            row["SubjectKey"],            row["StudyEventOID"],            row["StudyEventRepeatKey"],            row["ItemGroupRepeatKey"],        )        if key not in occurrences:            occurrences[key] = {"study": row["StudyOID"], "items": {}}            if row["SubjectKey"] not in subject_order:                subject_order.append(row["SubjectKey"])        item = row["ItemOID"].rsplit(".", 1)[-1]        occurrences[key]["items"][item] = row["Value"]    records_by_subject = {}    for (subject, event_oid, event_repeat, group_repeat), occ in occurrences.items():        items = occ["items"]        reaction = items.get("FAOBJ", "")        fadtc = items.get("FADTC", "")        day = day_number(event_oid)        fatpt = "END DAY %d" % day        for test_index, (item_key, testcd, test) in enumerate(TESTS):            if item_key not in items:                # The result item was not collected for this reaction:                # no record.                continue            result = items[item_key]            record = {                "DOMAIN": "FA",                "STUDYID": occ["study"],                "USUBJID": subject,                "FATESTCD": testcd,                "FATEST": test,                "FAOBJ": reaction,                "FACAT": "REACTOGENICITY",                "FAORRES": result,                "FAORRESU": "mm" if testcd == "LDIAM" else "",                "FASTRESC": result,                "FASTAT": "NOT DONE" if result == "" else "",                "FATPT": fatpt,                "FADTC": fadtc,            }            sort_key = (day, int(event_repeat), int(group_repeat), test_index)            records_by_subject.setdefault(subject, []).append((sort_key, record))    out_rows = []    for subject in subject_order:        subject_records = sorted(            records_by_subject.get(subject, []), key=lambda pair: pair[0]        )        for seq, (_, record) in enumerate(subject_records, start=1):            record["FASEQ"] = str(seq)            out_rows.append(record)    with open(OUTPUT_FA, "w", newline="", encoding="utf-8") as handle:        writer = csv.DictWriter(handle, fieldnames=COLUMNS)        writer.writeheader()        writer.writerows(out_rows)if __name__ == "__main__":    main()

solutions/sdtm-fa-odm-multitest/result.sas

Edit
42 lines
/* opensas reference solution for the yamaa benchmark sdtm-fa-odm-multitest.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data odm;  length StudyOID MetaDataVersionOID SubjectKey StudyEventOID StudyEventRepeatKey ItemGroupOID ItemGroupRepeatKey ItemOID Value $1024;  infile "/app/input/odm.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input StudyOID : $char1024. MetaDataVersionOID : $char1024. SubjectKey : $char1024. StudyEventOID : $char1024. StudyEventRepeatKey : $char1024. ItemGroupOID : $char1024. ItemGroupRepeatKey : $char1024. ItemOID : $char1024. Value : $char1024.;run;proc sort data=odm out=ordered; by StudyOID SubjectKey StudyEventOID StudyEventRepeatKey ItemGroupRepeatKey; run;data siblings;  length FAOBJ FADTC $1024;  retain FAOBJ FADTC;  set ordered;  where ItemGroupOID='IG.FA';  by StudyOID SubjectKey StudyEventOID StudyEventRepeatKey ItemGroupRepeatKey;  if first.ItemGroupRepeatKey then do;    FAOBJ=''; FADTC='';  end;    if ItemOID='IT.FA.FAOBJ' then FAOBJ=Value;  if ItemOID='IT.FA.FADTC' then FADTC=Value;  if last.ItemGroupRepeatKey then output;run;proc sql;create table joined as select a.*,b.FAOBJ,b.FADTC from odm a inner join siblings b on a.StudyOID=b.StudyOID and a.SubjectKey=b.SubjectKey and a.StudyEventOID=b.StudyEventOID and a.StudyEventRepeatKey=b.StudyEventRepeatKey and a.ItemGroupRepeatKey=b.ItemGroupRepeatKey where a.ItemOID in ('IT.FA.OCCUR','IT.FA.SEV','IT.FA.LDIAM'); quit;data prepared; set joined; length DOMAIN $2 STUDYID USUBJID FATESTCD FATEST FACAT FAORRES FAORRESU FASTRESC FASTAT FATPT $1024;DOMAIN='FA'; STUDYID=StudyOID; USUBJID=SubjectKey; FACAT='REACTOGENICITY'; FAORRES=Value; FASTRESC=Value; FAORRESU=''; FASTAT=''; if missing(Value) then FASTAT='NOT DONE';if ItemOID='IT.FA.OCCUR' then do; FATESTCD='OCCUR'; FATEST='Occurrence Indicator'; TESTORD=1; end;if ItemOID='IT.FA.SEV' then do; FATESTCD='SEV'; FATEST='Severity/Intensity'; TESTORD=2; end;if ItemOID='IT.FA.LDIAM' then do; FATESTCD='LDIAM'; FATEST='Longest Diameter'; FAORRESU='mm'; TESTORD=3; end;FATPT='END DAY 1'; if StudyEventOID='DAY2' then FATPT='END DAY 2'; DAYORD=input(compress(StudyEventOID,,'kd'),best32.); REP=input(ItemGroupRepeatKey,best32.); run;proc sort data=prepared; by STUDYID USUBJID DAYORD REP TESTORD; run;data result; set prepared; by STUDYID USUBJID; retain FASEQ;  if first.USUBJID then FASEQ=0;  FASEQ+1;run;proc sql;  create table final as select DOMAIN, STUDYID, USUBJID, FASEQ, FATESTCD, FATEST, FAOBJ, FACAT, FAORRES, FAORRESU, FASTRESC, FASTAT, FATPT, FADTC from result;quit;proc export data=final outfile="/app/output/fa.csv" dbms=csv replace; run;