Review ask

As a clinical trial statistician or programmer, challenge anything in this benchmark: describe a problem in Comments, or propose a fix with the Edit button on any source file.

  1. Task: Would a real study need this derivation, specified this way? Does it follow CDISC SDTM: variable names, controlled terms, record structure? Flag any rule that is ambiguous or that a sponsor would handle differently.
  2. Inputs: Do they look like data collected in a real trial, and do they include the situations you would expect to meet in one?
  3. Expected output: Would independent QC programming reproduce every value from these inputs and rules?

The other tabs beside Comments are background for assessment, not under review.

Summary

Lifecycle: draftEditUnder review
What this benchmark means
README.md

Goal: build one Exposure (EX) record per constant-dose interval from collected administration records, carrying EXSEQ, EXTRT, EXDOSE, EXDOSU, EXDOSFRQ, EXSTDTC, EXENDTC, and EXADJ.

Input: collected administration rows, one per dosing day, carrying treatment, dose, dose unit, dosing frequency, administration date, and the adjustment reason recorded when the dose changed.

Variables:

  • EXSEQ numbers the subject's intervals in the order they started, by start date then treatment. With STUDYID and USUBJID it identifies the record.
  • EXDOSE is the collected dose that defines the interval: the subject's administrations of one treatment at one dose, unit, and frequency form one interval.
  • EXSTDTC is the first administration date at the interval's dose level.
  • EXENDTC is the last administration date at the interval's dose level.
  • EXADJ is the adjustment reason recorded on any administration in the interval, for example DOSE REDUCED or DOSE INTERRUPTED (the alphabetically first when they differ); blank when none was recorded.

Note: a dose interruption is its own interval with a zero dose and the reason recorded. A dose level that recurs later, for example one resumed after an interruption, is not split: its one record spans from the level's first to its last administration.

Provenance: one record per constant dosing interval follows the CDISC SDTMIG exposure assumption, with the adjustment reason on the record where the new dose took effect. All subjects, dates, doses, and reasons are invented fixtures.

Inputs

Under review
1 source file

input/ec_raw.csv

Edit
13 rows
input/ec_raw.csv
STUDYIDUSUBJIDECTRTECDOSEECDOSUECDOSFRQECSTDTCECADJ
DOSEINTStudy Drug50mgQD2025-01-02—
DOSEINTStudy Drug50mgQD2025-01-03—
DOSEINTStudy Drug50mgQD2025-01-04—
DOSEINTStudy Drug100mgQD2025-02-01—
DOSEINTStudy Drug100mgQD2025-02-02—
DOSEINTStudy Drug75mgQD2025-02-03DOSE REDUCED
DOSEINTStudy Drug75mgQD2025-02-04—
DOSEINTStudy Drug200mgQD2025-03-01—
DOSEINTStudy Drug200mgQD2025-03-02—
DOSEINTStudy Drug0mgQD2025-03-03DOSE INTERRUPTED
DOSEINTStudy Drug0mgQD2025-03-04—
DOSEINTStudy Drug150mgQD2025-03-05DOSE REDUCED
DOSEINTStudy Drug150mgQD2025-03-06—

Expected output

Under review
6 expected rows

expected/ex.csv

Edit
6 rows
expected/ex.csv
DOMAINSTUDYIDUSUBJIDEXSEQEXTRTEXDOSEEXDOSUEXDOSFRQEXSTDTCEXENDTCEXADJ
EXDOSEINT1Study Drug50mgQD2025-01-022025-01-04—
EXDOSEINT1Study Drug100mgQD2025-02-012025-02-02—
EXDOSEINT2Study Drug75mgQD2025-02-032025-02-04DOSE REDUCED
EXDOSEINT1Study Drug200mgQD2025-03-012025-03-02—
EXDOSEINT2Study Drug0mgQD2025-03-032025-03-04DOSE INTERRUPTED
EXDOSEINT3Study Drug150mgQD2025-03-052025-03-06DOSE REDUCED
Shaded columns are derived— = missing value

Comments and assessment materials

Comments

Signed in with GitHub; saved in repository Discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Variable NameVariable LabelTypeLengthControlled Terms or FormatOriginCoreConversion DefinitionVariable TypeVariable OrderComments for Define
DOMAINDomain AbbreviationCharAssignedSet constant value "EX".SDTM1
STUDYIDStudy IdentifierCharCollectedCopy value from EC_RAW.STUDYID.SDTM2
USUBJIDUnique Subject IdentifierCharCollectedCopy value from EC_RAW.USUBJID.SDTM3
EXTRTName of TreatmentCharCollectedCopy value from EC_RAW.ECTRT.SDTM4
EXDOSEDoseNumCollectedCopy value from EC_RAW.ECDOSE.SDTM5
EXDOSUDose UnitsCharmgCollectedCopy value from EC_RAW.ECDOSU.SDTM6
EXDOSFRQDosing Frequency per IntervalCharQDCollectedCopy value from EC_RAW.ECDOSFRQ.SDTM7
EXSTDTCStart Date/Time of TreatmentCharISO 8601DerivedSet per constructed row; see the Row construction sheet.SDTM8
EXENDTCEnd Date/Time of TreatmentCharISO 8601DerivedSet per constructed row; see the Row construction sheet.SDTM9
EXADJReason for Dose AdjustmentCharDerivedSet per constructed row; see the Row construction sheet.SDTM10
EXSEQSequence NumberNumDerivedRow number within each group of STUDYID, USUBJID, ordered by EXSTDTC then EXTRT.SDTM11
TemplateInclude whenEXSTDTCEXENDTCEXADJ
dose_intervalsAggregate MIN(EC_RAW.ECSTDTC)Aggregate MAX(EC_RAW.ECSTDTC)Aggregate MIN(EC_RAW.ECADJ)
VariablePermitted values
EXDOSUmg
EXDOSFRQQD
VersionDateAuthorDescriptionReviewerSign-off
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/sdtm-ex-intervals/full.md

Edit
20 lines

Following CDISC SDTM standards, use the provided EC_RAW dataset to create an EX dataset with one record per constant-dose interval.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, EXSEQ, EXTRT, EXDOSE, EXDOSU, EXDOSFRQ, EXSTDTC, EXENDTC, EXADJ

The subject's administrations of one treatment at one dose, unit, and frequency form one interval. EXSTDTC is the first administration date at the interval dose level, and EXENDTC is the last. EXADJ is the adjustment reason recorded on any administration in the interval, with no value when none was recorded; when the recorded reasons differ, use the alphabetically first. A dose interruption is its own interval with a zero dose and the recorded reason. A dose level that recurs later is not split: its one record spans from the level's first to its last administration. EXSEQ numbers the subject's intervals in the order they started, by start date then treatment.

Read the source datasets from /app/input and save the completed dataset as /app/output/ex.csv.

prompts/sdtm-ex-intervals/conventions.md

Edit
9 lines

Following CDISC SDTM standards, use the provided EC_RAW dataset to create an EX dataset with one record per constant-dose interval.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, EXSEQ, EXTRT, EXDOSE, EXDOSU, EXDOSFRQ, EXSTDTC, EXENDTC, EXADJ

Read the source datasets from /app/input and save the completed dataset as /app/output/ex.csv.

prompts/sdtm-ex-intervals/brief.md

Edit
9 lines

Following CDISC SDTM standards, use the provided EC_RAW dataset to create an EX dataset with one record per constant-dose interval.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, EXSEQ, EXTRT, EXDOSE, EXDOSU, EXDOSFRQ, EXSTDTC, EXENDTC, EXADJ

Read the source datasets from /app/input and save the completed dataset as /app/output/ex.csv.

yamaa code

Runs the yamaa specification; its result must match the expected output

run.R

Edit
4 lines
library(yamaa)ex <- yamaa_domain("spec.yaml")$outputex

run.py

Edit
4 lines
import yamaaex = yamaa.yamaa_domain("spec.yaml").outputex

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/sdtm-ex-intervals/result.R

Edit
50 lines
# Reference solution for the yamaa benchmark sdtm-ex-intervals (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)ec_raw <- read_csv(  "/app/input/ec_raw.csv",  col_types = cols(.default = col_character()))ex <- ec_raw |>  mutate(    EXDOSE_NUM = suppressWarnings(as.numeric(ECDOSE)),    ECSTDTC_DT = suppressWarnings(as.Date(ECSTDTC)),    ECADJ_CLEAN = na_if(ECADJ, "")  ) |>  group_by(STUDYID, USUBJID, ECTRT, EXDOSE_NUM, ECDOSU, ECDOSFRQ) |>  summarise(    START_DT = min(ECSTDTC_DT, na.rm = TRUE),    END_DT = max(ECSTDTC_DT, na.rm = TRUE),    EXADJ = {      vals <- sort(unique(ECADJ_CLEAN[!is.na(ECADJ_CLEAN)]))      if (length(vals) == 0) NA_character_ else vals[1]    },    .groups = "drop"  ) |>  mutate(    DOMAIN = "EX",    EXTRT = ECTRT,    EXDOSE = EXDOSE_NUM,    EXDOSU = ECDOSU,    EXDOSFRQ = ECDOSFRQ,    EXSTDTC = START_DT,    EXENDTC = END_DT  ) |>  arrange(STUDYID, USUBJID, START_DT, EXTRT) |>  group_by(STUDYID, USUBJID) |>  mutate(EXSEQ = row_number()) |>  ungroup() |>  select(    DOMAIN, STUDYID, USUBJID, EXSEQ, EXTRT, EXDOSE, EXDOSU,    EXDOSFRQ, EXSTDTC, EXENDTC, EXADJ  )dir.create("/app/output", showWarnings = FALSE)write_csv(ex, "/app/output/ex.csv", na = "")

solutions/sdtm-ex-intervals/result.py

Edit
57 lines
# Reference solution for the yamaa benchmark sdtm-ex-intervals (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.from pathlib import Pathimport polars as plec_raw = pl.read_csv("/app/input/ec_raw.csv", infer_schema=False)ec = ec_raw.with_columns(    EXDOSE_NUM=pl.col("ECDOSE").cast(pl.Float64, strict=False),    ECSTDTC_DT=pl.col("ECSTDTC").str.to_date(strict=False),    ECADJ_CLEAN=pl.col("ECADJ").map_elements(        lambda v: None if v is None or v == "" else v, return_dtype=pl.String    ),)grouped = (    ec.group_by("STUDYID", "USUBJID", "ECTRT", "EXDOSE_NUM", "ECDOSU", "ECDOSFRQ")    .agg(        pl.col("ECSTDTC_DT").min().alias("START_DT"),        pl.col("ECSTDTC_DT").max().alias("END_DT"),        pl.col("ECADJ_CLEAN").drop_nulls().unique().sort().first().alias("EXADJ"),        pl.col("ECDOSE").first().alias("EXDOSE_RAW"),    )    .with_columns(        DOMAIN=pl.lit("EX"),        EXTRT=pl.col("ECTRT"),        EXDOSE=pl.col("EXDOSE_NUM"),        EXDOSU=pl.col("ECDOSU"),        EXDOSFRQ=pl.col("ECDOSFRQ"),        EXSTDTC=pl.col("START_DT"),        EXENDTC=pl.col("END_DT"),    )    .sort(["STUDYID", "USUBJID", "START_DT", "EXTRT"])    .with_columns(EXSEQ=pl.col("USUBJID").cum_count().over(["STUDYID", "USUBJID"]))    .with_columns(EXSEQ=pl.col("EXSEQ").cast(pl.Int64))    .select(        "DOMAIN",        "STUDYID",        "USUBJID",        "EXSEQ",        "EXTRT",        "EXDOSE",        "EXDOSU",        "EXDOSFRQ",        "EXSTDTC",        "EXENDTC",        "EXADJ",    ))Path("/app/output").mkdir(exist_ok=True)grouped.write_csv("/app/output/ex.csv")

solutions/sdtm-ex-intervals/result.sas

Edit
24 lines
/* opensas reference solution for the yamaa benchmark sdtm-ex-intervals.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data ec_raw;  length STUDYID USUBJID ECTRT ECDOSU ECDOSFRQ ECSTDTC ECADJ $1024;  infile "/app/input/ec_raw.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. ECTRT : $char1024. ECDOSE ECDOSU : $char1024. ECDOSFRQ : $char1024. ECSTDTC : $char1024. ECADJ : $char1024.;run;proc sql;create table intervals as select STUDYID,USUBJID,ECTRT as EXTRT,ECDOSE as EXDOSE,ECDOSU as EXDOSU,ECDOSFRQ as EXDOSFRQ,min(ECSTDTC) as EXSTDTC,max(ECSTDTC) as EXENDTC from ec_raw group by STUDYID,USUBJID,ECTRT,ECDOSE,ECDOSU,ECDOSFRQ;create table reasons as select STUDYID,USUBJID,ECTRT,ECDOSE,ECDOSU,ECDOSFRQ,min(ECADJ) as EXADJ from ec_raw where not missing(ECADJ) group by STUDYID,USUBJID,ECTRT,ECDOSE,ECDOSU,ECDOSFRQ;create table prepared as select 'EX' as DOMAIN,a.*,b.EXADJ from intervals a left join reasons b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID and a.EXTRT=b.ECTRT and a.EXDOSE=b.ECDOSE and a.EXDOSU=b.ECDOSU and a.EXDOSFRQ=b.ECDOSFRQ; quit;proc sort data=prepared; by STUDYID USUBJID EXSTDTC EXTRT; run;data result; set prepared; by STUDYID USUBJID; retain EXSEQ;  if first.USUBJID then EXSEQ=0;  EXSEQ+1;run;proc sql;  create table final as select DOMAIN, STUDYID, USUBJID, EXSEQ, EXTRT, EXDOSE, EXDOSU, EXDOSFRQ, EXSTDTC, EXENDTC, EXADJ from result;quit;proc export data=final outfile="/app/output/ex.csv" dbms=csv replace; run;