Review ask

As a clinical trial statistician or programmer, challenge anything in this benchmark: describe a problem in Comments, or propose a fix with the Edit button on any source file.

  1. Task: Would a real study need this derivation, specified this way? Does it follow CDISC ADaM: variable names, controlled terms, record structure? Flag any rule that is ambiguous or that a sponsor would handle differently.
  2. Inputs: Do they look like data collected in a real trial, and do they include the situations you would expect to meet in one?
  3. Expected output: Would independent QC programming reproduce every value from these inputs and rules?

The other tabs beside Comments are background for assessment, not under review.

Summary

Lifecycle: reviewedEditUnder review
What this benchmark means
README.md

Goal: one record per subject (STUDYID, USUBJID) recording when randomization happened: the collected date (RANDDT), the study day of randomization (RANDDY), and the collected moment written as text (RANDDTC).

Input: demographics (DM) records carrying the subject reference start date (RFSTDTC), the collected randomization date (RANDDT), and the collected randomization moment (RANDDTTM). The sample covers the study-day edges: a randomization on the reference date (day 1), one the day before it (day -1), and one several days before it; a subject never randomized; a collected date with no collected moment; a missing reference date; and a collected moment with no collected date.

Variables:

  • RANDDT is the randomization date as collected. It stays empty when no randomization date was collected.
  • RANDDY is the study day of randomization measured from RFSTDTC: the reference date itself is day 1, later dates count forward inclusively, and earlier dates count backward with no day 0, so the day before the reference date is day -1. It stays empty when either date is missing.
  • RANDDTC is the collected randomization moment (RANDDTTM) written as text at whole-second precision, so a collected 10:30 becomes 10:30:00. It stays empty when no moment was collected.

Note: the moment is written from what was collected, independently of the date: a subject with a collected moment but no collected date still gets RANDDTC, while RANDDT and RANDDY stay empty.

Inputs

Under review
3 source files

input/dm.parquet

Edit

Binary fixture; inspect it in the benchmark source.

input/dm.schema.yaml

Edit
schema_version: "1.0"
domain: DM
keys: [STUDYID, USUBJID]
input:
  ODM: odm.csv

output:
  path: dm.parquet
  columns: [STUDYID, USUBJID, RFSTDTC, RANDDT, RANDDTTM]

columns:
  - name: STUDYID
    type: str
    label: Study Identifier
    derivation: ODM.StudyOID

  - name: USUBJID
    type: str
    label: Unique Subject Identifier
    derivation: ODM.SubjectKey

  - name: RFSTDTC
    type: date
    label: Subject Reference Start Date

  - name: RANDDT
    type: date
    label: Date of Randomization

  - name: RANDDTTM
    type: datetime
    label: Date and Time of Randomization

rows:
  - id: subject
    group_by: [ODM.StudyOID, ODM.SubjectKey]
    derivations:
      RFSTDTC:
        aggregate:
          filter: "ODM.ItemOID = 'IT.DM.RFSTDTC'"
          expr: "ONLY(ODM.Value)"
      RANDDT:
        aggregate:
          filter: "ODM.ItemOID = 'IT.DM.RANDDT'"
          expr: "ONLY(ODM.Value)"
      RANDDTTM:
        aggregate:
          filter: "ODM.ItemOID = 'IT.DM.RANDDTTM'"
          expr: "ONLY(ODM.Value)"

input/odm.csv

Edit
19 rows
input/odm.csv
StudyOIDMetaDataVersionOIDSubjectKeyStudyEventOIDStudyEventRepeatKeyItemGroupOIDItemGroupRepeatKeyItemOIDValue
YAMAA-01MDV.1YAMAA-01-101SCREENING1IG.DM1IT.DM.RFSTDTC2024-03-01
YAMAA-01MDV.1YAMAA-01-101SCREENING1IG.DM1IT.DM.RANDDT2024-03-04
YAMAA-01MDV.1YAMAA-01-101SCREENING1IG.DM1IT.DM.RANDDTTM2024-03-04T10:30
YAMAA-01MDV.1YAMAA-01-102SCREENING1IG.DM1IT.DM.RFSTDTC2024-05-10
YAMAA-01MDV.1YAMAA-01-102SCREENING1IG.DM1IT.DM.RANDDT2024-05-10
YAMAA-01MDV.1YAMAA-01-102SCREENING1IG.DM1IT.DM.RANDDTTM2024-05-10T08:05:33
YAMAA-01MDV.1YAMAA-01-103SCREENING1IG.DM1IT.DM.RFSTDTC2024-06-20
YAMAA-01MDV.1YAMAA-01-103SCREENING1IG.DM1IT.DM.RANDDT2024-06-19
YAMAA-01MDV.1YAMAA-01-103SCREENING1IG.DM1IT.DM.RANDDTTM2024-06-19T16:45
YAMAA-01MDV.1YAMAA-01-104SCREENING1IG.DM1IT.DM.RFSTDTC2024-07-15
YAMAA-01MDV.1YAMAA-01-104SCREENING1IG.DM1IT.DM.RANDDT2024-07-10
YAMAA-01MDV.1YAMAA-01-104SCREENING1IG.DM1IT.DM.RANDDTTM2024-07-10T23:59:59
YAMAA-01MDV.1YAMAA-01-105SCREENING1IG.DM1IT.DM.RFSTDTC2024-08-01
YAMAA-01MDV.1YAMAA-01-106SCREENING1IG.DM1IT.DM.RFSTDTC2024-09-12
YAMAA-01MDV.1YAMAA-01-106SCREENING1IG.DM1IT.DM.RANDDT2024-09-15
YAMAA-01MDV.1YAMAA-01-107SCREENING1IG.DM1IT.DM.RANDDT2024-10-05
YAMAA-01MDV.1YAMAA-01-107SCREENING1IG.DM1IT.DM.RANDDTTM2024-10-05T12:00
YAMAA-01MDV.1YAMAA-01-108SCREENING1IG.DM1IT.DM.RFSTDTC2024-11-20
YAMAA-01MDV.1YAMAA-01-108SCREENING1IG.DM1IT.DM.RANDDTTM2024-11-22T09:00

Expected output

Under review
8 expected rows

expected/adsl.csv

Edit
8 rows
expected/adsl.csv
STUDYIDUSUBJIDRANDDTRANDDYRANDDTC
YAMAA-012024-03-0442024-03-04T10:30:00
YAMAA-012024-05-1012024-05-10T08:05:33
YAMAA-012024-06-19-12024-06-19T16:45:00
YAMAA-012024-07-10-52024-07-10T23:59:59
YAMAA-01———
YAMAA-012024-09-154—
YAMAA-012024-10-05—2024-10-05T12:00:00
YAMAA-01——2024-11-22T09:00:00
Shaded columns are derived— = missing value

Comments and assessment materials

Comments

Signed in with GitHub; saved in repository Discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Dataset NameVariable NameVariable LabelTypeCodelist/Controlled TermsCoreComputational MethodOrigin
ADSLSTUDYIDStudy IdentifiercharacterCopy value from DM.STUDYID.Predecessor
ADSLUSUBJIDUnique Subject IdentifiercharacterCopy value from DM.USUBJID.Predecessor
ADSLRANDDTDate of RandomizationnumericISO 8601Copy value from DM.RANDDT.Predecessor
ADSLRANDDYRandomization Study DaynumericCDISC study day of DM.RANDDT against DM.RFSTDTC as day 1; there is no day zero.Derived
ADSLRANDDTCDate and Time of RandomizationcharacterCopy value from DM.RANDDTTM.Predecessor
VersionDateAuthorDescriptionReviewerSign-off
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/adam-adsl-randomization/full.md

Edit
19 lines

Following CDISC ADaM standards, use the provided DM dataset to create an ADSL dataset with one record per subject.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, RANDDT, RANDDY, RANDDTC

RANDDT is the randomization date as collected, and has no value when no randomization date was collected.

RANDDY is the study day of randomization relative to the subject reference start date.

RANDDTC is the collected randomization moment written as ISO 8601 text with a T between date and time, at whole-second precision. It has no value when no moment was collected, and it is written from what was collected even when no randomization date was collected.

Read the source datasets from /app/input and save the completed dataset as /app/output/adsl.csv.

prompts/adam-adsl-randomization/conventions.md

Edit
11 lines

Following CDISC ADaM standards, use the provided DM dataset to create an ADSL dataset with one record per subject.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, RANDDT, RANDDY, RANDDTC

RANDDTC is written as ISO 8601 text with a T between date and time, at whole-second precision.

Read the source datasets from /app/input and save the completed dataset as /app/output/adsl.csv.

prompts/adam-adsl-randomization/brief.md

Edit
8 lines

Following CDISC ADaM standards, use the provided DM dataset to create an ADSL dataset with one record per subject.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, RANDDT, RANDDY, RANDDTC

Read the source datasets from /app/input and save the completed dataset as /app/output/adsl.csv.

yamaa code

Runs the yamaa specification; its result must match the expected output

run.R

Edit
4 lines
library(yamaa)adsl <- yamaa_domain("spec.yaml")$outputadsl

run.py

Edit
4 lines
import yamaaadsl = yamaa.yamaa_domain("spec.yaml").outputadsl

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/adam-adsl-randomization/result.R

Edit
31 lines
# Reference solution for the yamaa benchmark adam-adsl-randomization (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)dm <- nanoparquet::read_parquet("/app/input/dm.parquet")# Study day from the reference start: the reference day itself is day 1,# later dates count forward inclusively, earlier dates count backward with# no day 0, so the day before is day -1.adsl <- dm |>  mutate(    .diff = as.integer(RANDDT - RFSTDTC),    RANDDY = case_when(      is.na(.diff) ~ NA_integer_,      .diff >= 0L ~ .diff + 1L,      .default = .diff    ),    # The collected moment as ISO 8601 text with a T, at whole-second    # precision, written from what was collected even when no date was.    RANDDTC = format(RANDDTTM, "%Y-%m-%dT%H:%M:%S", tz = "UTC")  ) |>  select(STUDYID, USUBJID, RANDDT, RANDDY, RANDDTC) |>  arrange(USUBJID)dir.create("/app/output", showWarnings = FALSE)write_csv(adsl, "/app/output/adsl.csv", na = "")

solutions/adam-adsl-randomization/result.py

Edit
33 lines
# Reference solution for the yamaa benchmark adam-adsl-randomization (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.from pathlib import Pathimport polars as pldm = pl.read_parquet("/app/input/dm.parquet")# Study day from the reference start: the reference day itself is day 1,# later dates count forward inclusively, earlier dates count backward with# no day 0, so the day before is day -1.diff = (pl.col("RANDDT") - pl.col("RFSTDTC")).dt.total_days()adsl = (    dm.with_columns(        RANDDY=pl.when(diff.is_null())        .then(None)        .when(diff >= 0)        .then(diff + 1)        .otherwise(diff),        # The collected moment as ISO 8601 text with a T, at whole-second        # precision, written from what was collected even when no date was.        RANDDTC=pl.col("RANDDTTM").dt.strftime("%Y-%m-%dT%H:%M:%S"),    )    .select("STUDYID", "USUBJID", "RANDDT", "RANDDY", "RANDDTC")    .sort("USUBJID"))Path("/app/output").mkdir(parents=True, exist_ok=True)adsl.write_csv("/app/output/adsl.csv", null_value="")

solutions/adam-adsl-randomization/result.sas

Edit
28 lines
/* opensas reference solution for the yamaa benchmark adam-adsl-randomization.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. *//* opensas v0.6.6 has no Parquet reader. The runner must stage dm.parquet   as dm.csv without deriving any values, preserving ISO dates/timestamps. */data dm;  length STUDYID USUBJID $1024 RANDDT RFSTDTC $10 RANDDTTM $32;  infile '/app/input/dm.csv' dsd dlm=',' firstobs=2 truncover;  input STUDYID : $char1024. USUBJID : $char1024. RFSTDTC : $char10. RANDDT : $char10. RANDDTTM : $char32.;run;data result;  length RANDDTC $19;  set dm;  if not missing(RANDDT) and not missing(RFSTDTC) then do;    RANDDY=input(RANDDT,yymmdd10.)-input(RFSTDTC,yymmdd10.);    if RANDDY>=0 then RANDDY=RANDDY+1;  end;  if not missing(RANDDTTM) then do;    RANDDTC=translate(substr(RANDDTTM,1,19),'T',' ');    if lengthn(RANDDTTM)=16 then RANDDTC=cats(RANDDTTM,':00');  end;run;proc sql;  create table final as select STUDYID, USUBJID, RANDDT, RANDDY, RANDDTC from result;quit;proc export data=final outfile="/app/output/adsl.csv" dbms=csv replace; run;