Review ask

As a clinical trial statistician or programmer, challenge anything in this benchmark: describe a problem in Comments, or propose a fix with the Edit button on any source file.

  1. Task: Would a real study need this derivation, specified this way? Does it follow CDISC SDTM: variable names, controlled terms, record structure? Flag any rule that is ambiguous or that a sponsor would handle differently.
  2. Inputs: Do they look like data collected in a real trial, and do they include the situations you would expect to meet in one?
  3. Expected output: Would independent QC programming reproduce every value from these inputs and rules?

The other tabs beside Comments are background for assessment, not under review.

Summary

Lifecycle: reviewedEditUnder review
What this benchmark means
README.md

Goal: carry each administered combination component with its dose, unit, administration interval, and adjustment reason, adding EXSEQ, EXTRT, EXDOSE, EXDOSU, EXSTDTC, EXENDTC, and EXADJ.

Input: collected exposure records with one row per administered component, carrying treatment, dose, unit, administration interval, and adjustment reason.

Variables:

  • EXSEQ is the order of the administration within the subject, numbered by start date and then, on the same date, by treatment name.
  • EXDOSU is the dose unit as collected; milligrams (mg), milligrams per square metre (mg/m2), and area under the curve (AUC) remain distinct.
  • EXADJ is the adjustment reason as collected (for example DOSE INTERRUPTED); blank when none was reported.

Note: a saline placebo component is administered, so it keeps its EXDOSE of 0; a zero dose is not an uncollected dose.

Inputs

Under review
1 source file

input/ex_raw.csv

Edit
9 rows
input/ex_raw.csv
STUDYIDUSUBJIDEXTRTEXDOSEEXDOSUEXSTDTCEXENDTCEXADJ
PILOT7Pembrolizumab200mg2025-01-022025-01-02—
PILOT7Pemetrexed500mg/m22025-01-022025-01-02—
PILOT7Carboplatin5AUC2025-01-022025-01-02—
PILOT7Pembrolizumab200mg2025-01-232025-01-23DOSE INTERRUPTED
PILOT7Pemetrexed500mg/m22025-01-232025-01-23—
PILOT7Carboplatin5AUC2025-01-232025-01-23—
PILOT7Saline placebo0mg2025-01-022025-01-02—
PILOT7Pemetrexed500mg/m22025-01-022025-01-02—
PILOT7Cisplatin75mg/m22025-01-022025-01-02—

Expected output

Under review
9 expected rows

expected/ex.csv

Edit
9 rows
expected/ex.csv
DOMAINSTUDYIDUSUBJIDEXSEQEXTRTEXDOSEEXDOSUEXSTDTCEXENDTCEXADJ
EXPILOT72Pembrolizumab200mg2025-01-022025-01-02—
EXPILOT73Pemetrexed500mg/m22025-01-022025-01-02—
EXPILOT71Carboplatin5AUC2025-01-022025-01-02—
EXPILOT75Pembrolizumab200mg2025-01-232025-01-23DOSE INTERRUPTED
EXPILOT76Pemetrexed500mg/m22025-01-232025-01-23—
EXPILOT74Carboplatin5AUC2025-01-232025-01-23—
EXPILOT73Saline placebo0mg2025-01-022025-01-02—
EXPILOT72Pemetrexed500mg/m22025-01-022025-01-02—
EXPILOT71Cisplatin75mg/m22025-01-022025-01-02—
Shaded columns are derived— = missing value

Comments and assessment materials

Comments

Signed in with GitHub; saved in repository Discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Variable NameVariable LabelTypeLengthControlled Terms or FormatOriginCoreConversion DefinitionVariable TypeVariable OrderComments for Define
DOMAINDomain AbbreviationCharAssignedSet constant value "EX".SDTM1
STUDYIDStudy IdentifierCharCollectedCopy value from EX_RAW.STUDYID.SDTM2
USUBJIDUnique Subject IdentifierCharCollectedCopy value from EX_RAW.USUBJID.SDTM3
EXSEQSequence NumberNumDerivedRow number within each group of STUDYID, USUBJID, ordered by EX_RAW.EXSTDTC then EX_RAW.EXTRT.SDTM4
EXTRTName of TreatmentCharCollectedCopy value from EX_RAW.EXTRT.SDTM5
EXDOSEDoseNumCollectedCopy value from EX_RAW.EXDOSE.SDTM6
EXDOSUDose UnitsCharCollectedCopy value from EX_RAW.EXDOSU.SDTM7
EXSTDTCStart Date/Time of TreatmentCharISO 8601CollectedCopy value from EX_RAW.EXSTDTC.SDTM8
EXENDTCEnd Date/Time of TreatmentCharISO 8601CollectedCopy value from EX_RAW.EXENDTC.SDTM9
EXADJReason for Dose AdjustmentCharCollectedCopy value from EX_RAW.EXADJ.SDTM10
VersionDateAuthorDescriptionReviewerSign-off
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/sdtm-ex-combination/full.md

Edit
18 lines

Following CDISC SDTM standards, use the provided EX_RAW dataset to create an EX dataset with one record per administered combination component.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, EXSEQ, EXTRT, EXDOSE, EXDOSU, EXSTDTC, EXENDTC, EXADJ

EXSEQ numbers the administrations within the subject by start date and then, on the same date, by treatment name. EXDOSU is the dose unit as collected: milligrams (mg), milligrams per square metre (mg/m2), and area under the curve (AUC) remain distinct. EXADJ is the adjustment reason as collected, with no value when none was reported. A saline placebo component is administered, so it keeps its EXDOSE of 0; a zero dose is not an uncollected dose.

Read the source datasets from /app/input and save the completed dataset as /app/output/ex.csv.

prompts/sdtm-ex-combination/conventions.md

Edit
10 lines

Following CDISC SDTM standards, use the provided EX_RAW dataset to create an EX dataset with one record per administered combination component.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, EXSEQ, EXTRT, EXDOSE, EXDOSU, EXSTDTC, EXENDTC, EXADJ

Read the source datasets from /app/input and save the completed dataset as /app/output/ex.csv.

prompts/sdtm-ex-combination/brief.md

Edit
10 lines

Following CDISC SDTM standards, use the provided EX_RAW dataset to create an EX dataset with one record per administered combination component.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, EXSEQ, EXTRT, EXDOSE, EXDOSU, EXSTDTC, EXENDTC, EXADJ

Read the source datasets from /app/input and save the completed dataset as /app/output/ex.csv.

yamaa code

Runs the yamaa specification; its result must match the expected output

run.R

Edit
4 lines
library(yamaa)ex <- yamaa_domain("spec.yaml")$outputex

run.py

Edit
4 lines
import yamaaex = yamaa.yamaa_domain("spec.yaml").outputex

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/sdtm-ex-combination/result.R

Edit
32 lines
# Reference solution for the yamaa benchmark sdtm-ex-combination (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)ex_raw <- read_csv(  "/app/input/ex_raw.csv",  col_types = cols(.default = col_character()))ex <- ex_raw |>  mutate(    DOMAIN = "EX",    EXDOSE = suppressWarnings(as.numeric(EXDOSE)),    EXSTDTC_DATE = suppressWarnings(as.Date(EXSTDTC)),    EXADJ = na_if(EXADJ, "")  ) |>  arrange(STUDYID, USUBJID, EXSTDTC_DATE, EXTRT) |>  group_by(STUDYID, USUBJID) |>  mutate(EXSEQ = row_number()) |>  ungroup() |>  select(    DOMAIN, STUDYID, USUBJID, EXSEQ, EXTRT, EXDOSE, EXDOSU,    EXSTDTC, EXENDTC, EXADJ  )dir.create("/app/output", showWarnings = FALSE)write_csv(ex, "/app/output/ex.csv", na = "")

solutions/sdtm-ex-combination/result.py

Edit
40 lines
# Reference solution for the yamaa benchmark sdtm-ex-combination (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.from pathlib import Pathimport polars as plex_raw = pl.read_csv("/app/input/ex_raw.csv", infer_schema=False)ex = (    ex_raw.with_columns(        DOMAIN=pl.lit("EX"),        EXDOSE=pl.col("EXDOSE").cast(pl.Float64, strict=False),        EXSTDTC_DT=pl.col("EXSTDTC").str.to_date(strict=False),        EXADJ=pl.col("EXADJ").map_elements(            lambda v: None if v == "" or v is None else v, return_dtype=pl.String        ),    )    .sort(["STUDYID", "USUBJID", "EXSTDTC_DT", "EXTRT"])    .with_columns(EXSEQ=pl.col("USUBJID").cum_count().over(["STUDYID", "USUBJID"]))    .with_columns(EXSEQ=pl.col("EXSEQ").cast(pl.Int64))    .select(        "DOMAIN",        "STUDYID",        "USUBJID",        "EXSEQ",        "EXTRT",        "EXDOSE",        "EXDOSU",        "EXSTDTC",        "EXENDTC",        "EXADJ",    ))Path("/app/output").mkdir(exist_ok=True)ex.write_csv("/app/output/ex.csv")

solutions/sdtm-ex-combination/result.sas

Edit
21 lines
/* opensas reference solution for the yamaa benchmark sdtm-ex-combination.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data ex_raw;  length STUDYID USUBJID EXTRT EXDOSU EXSTDTC EXENDTC EXADJ $1024;  infile "/app/input/ex_raw.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. EXTRT : $char1024. EXDOSE EXDOSU : $char1024. EXSTDTC : $char1024. EXENDTC : $char1024. EXADJ : $char1024.;run;data prepared; set ex_raw; length DOMAIN $2; DOMAIN='EX'; run;proc sort data=prepared; by STUDYID USUBJID EXSTDTC EXTRT; run;data result; set prepared; by STUDYID USUBJID; retain EXSEQ;  if first.USUBJID then EXSEQ=0;  EXSEQ+1;run;proc sql;  create table final as select DOMAIN, STUDYID, USUBJID, EXSEQ, EXTRT, EXDOSE, EXDOSU, EXSTDTC, EXENDTC, EXADJ from result;quit;proc export data=final outfile="/app/output/ex.csv" dbms=csv replace; run;