Review ask

As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.

We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.

Summary

Lifecycle: reviewedEditUnder review
What this benchmark means
README.md

Public entries: Python and R. Passing coverage requires an installed inventory.

Goal: one record per subject (STUDYID, USUBJID) recording when randomization happened: the collected date (RANDDT), the study day of randomization (RANDDY), and the collected moment written as text (RANDDTC).

Input: demographics (DM) records carrying the subject reference start date (RFSTDTC), the collected randomization date (RANDDT), and the collected randomization moment (RANDDTTM). The sample covers the study-day edges: a randomization on the reference date (day 1), one the day before it (day -1), and one several days before it; a subject never randomized; a collected date with no collected moment; a missing reference date; and a collected moment with no collected date.

Variables:

  • RANDDT is the randomization date as collected. It stays empty when no randomization date was collected.
  • RANDDY is the study day of randomization measured from RFSTDTC: the reference date itself is day 1, later dates count forward inclusively, and earlier dates count backward with no day 0, so the day before the reference date is day -1. It stays empty when either date is missing.
  • RANDDTC is the collected randomization moment (RANDDTTM) written as text at whole-second precision, so a collected 10:30 becomes 10:30:00. It stays empty when no moment was collected.

Note: the moment is written from what was collected, independently of the date: a subject with a collected moment but no collected date still gets RANDDTC, while RANDDT and RANDDY stay empty.

Inputs

Under review
3 source files

input/dm.parquet

Edit

Binary fixture; inspect it in the benchmark source.

input/dm.schema.yaml

Edit
yamaa: "1.0"
type: spec
domain: DM
keys: [STUDYID, USUBJID]
input:
  ODM: odm.csv

output:
  path: dm.parquet
  columns: [STUDYID, USUBJID, RFSTDTC, RANDDT, RANDDTTM]

columns:
  - name: STUDYID
    type: str
    label: Study Identifier
    derivation: ODM.StudyOID

  - name: USUBJID
    type: str
    label: Unique Subject Identifier
    derivation: ODM.SubjectKey

  - name: RFSTDTC
    type: date
    label: Subject Reference Start Date

  - name: RANDDT
    type: date
    label: Date of Randomization

  - name: RANDDTTM
    type: datetime
    label: Date and Time of Randomization

rows:
  - id: subject
    group_by: [ODM.StudyOID, ODM.SubjectKey]
    derivations:
      RFSTDTC:
        aggregate:
          filter: "ODM.ItemOID = 'IT.DM.RFSTDTC'"
          expr: "ONLY(ODM.Value)"
      RANDDT:
        aggregate:
          filter: "ODM.ItemOID = 'IT.DM.RANDDT'"
          expr: "ONLY(ODM.Value)"
      RANDDTTM:
        aggregate:
          filter: "ODM.ItemOID = 'IT.DM.RANDDTTM'"
          expr: "ONLY(ODM.Value)"

input/odm.csv

Edit
19 rows
input/odm.csv
StudyOIDMetaDataVersionOIDSubjectKeyStudyEventOIDStudyEventRepeatKeyItemGroupOIDItemGroupRepeatKeyItemOIDValue
YAMAA-01MDV.1YAMAA-01-101SCREENING1IG.DM1IT.DM.RFSTDTC2024-03-01
YAMAA-01MDV.1YAMAA-01-101SCREENING1IG.DM1IT.DM.RANDDT2024-03-04
YAMAA-01MDV.1YAMAA-01-101SCREENING1IG.DM1IT.DM.RANDDTTM2024-03-04T10:30
YAMAA-01MDV.1YAMAA-01-102SCREENING1IG.DM1IT.DM.RFSTDTC2024-05-10
YAMAA-01MDV.1YAMAA-01-102SCREENING1IG.DM1IT.DM.RANDDT2024-05-10
YAMAA-01MDV.1YAMAA-01-102SCREENING1IG.DM1IT.DM.RANDDTTM2024-05-10T08:05:33
YAMAA-01MDV.1YAMAA-01-103SCREENING1IG.DM1IT.DM.RFSTDTC2024-06-20
YAMAA-01MDV.1YAMAA-01-103SCREENING1IG.DM1IT.DM.RANDDT2024-06-19
YAMAA-01MDV.1YAMAA-01-103SCREENING1IG.DM1IT.DM.RANDDTTM2024-06-19T16:45
YAMAA-01MDV.1YAMAA-01-104SCREENING1IG.DM1IT.DM.RFSTDTC2024-07-15
YAMAA-01MDV.1YAMAA-01-104SCREENING1IG.DM1IT.DM.RANDDT2024-07-10
YAMAA-01MDV.1YAMAA-01-104SCREENING1IG.DM1IT.DM.RANDDTTM2024-07-10T23:59:59
YAMAA-01MDV.1YAMAA-01-105SCREENING1IG.DM1IT.DM.RFSTDTC2024-08-01
YAMAA-01MDV.1YAMAA-01-106SCREENING1IG.DM1IT.DM.RFSTDTC2024-09-12
YAMAA-01MDV.1YAMAA-01-106SCREENING1IG.DM1IT.DM.RANDDT2024-09-15
YAMAA-01MDV.1YAMAA-01-107SCREENING1IG.DM1IT.DM.RANDDT2024-10-05
YAMAA-01MDV.1YAMAA-01-107SCREENING1IG.DM1IT.DM.RANDDTTM2024-10-05T12:00
YAMAA-01MDV.1YAMAA-01-108SCREENING1IG.DM1IT.DM.RFSTDTC2024-11-20
YAMAA-01MDV.1YAMAA-01-108SCREENING1IG.DM1IT.DM.RANDDTTM2024-11-22T09:00

Expected output

Under review
8 expected rows

expected/adsl.csv

Edit
8 rows
expected/adsl.csv
STUDYIDUSUBJIDRANDDTRANDDYRANDDTC
YAMAA-012024-03-0442024-03-04T10:30:00
YAMAA-012024-05-1012024-05-10T08:05:33
YAMAA-012024-06-19-12024-06-19T16:45:00
YAMAA-012024-07-10-52024-07-10T23:59:59
YAMAA-01———
YAMAA-012024-09-154—
YAMAA-012024-10-05—2024-10-05T12:00:00
YAMAA-01——2024-11-22T09:00:00
Shaded columns are derived— = missing value

Human review and assessment

Your review

3 questions and your comments · View benchmark discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Dataset NameVariable NameVariable LabelTypeCodelist/Controlled TermsCoreComputational MethodOrigin
ADSLSTUDYIDStudy IdentifiercharacterCopy value from DM.STUDYID.Predecessor
ADSLUSUBJIDUnique Subject IdentifiercharacterCopy value from DM.USUBJID.Predecessor
ADSLRANDDTDate of RandomizationnumericISO 8601Copy value from DM.RANDDT.Predecessor
ADSLRANDDYRandomization Study DaynumericCDISC study day of DM.RANDDT against DM.RFSTDTC as day 1; there is no day zero.Derived
ADSLRANDDTCDate and Time of RandomizationcharacterCopy value from DM.RANDDTTM.Predecessor
VersionDateAuthorDescriptionReviewerSign-off
1.72026-10-20yamaa docs buildRead a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default
1.62026-10-17yamaa docs buildDescribe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight
1.52026-10-17yamaa docs buildName collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part
1.42026-10-15yamaa docs buildState datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/adam-adsl-randomization/full.md

Edit
18 lines

Following CDISC ADaM standards, use the provided DM and ODM datasets to create an ADSL dataset with one record per subject.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, RANDDT, RANDDY, RANDDTC

RANDDY is the study day of the randomization date RANDDT, measured from RFSTDTC, the subject's reference start date. RANDDY has no value when RANDDT or RFSTDTC has no value. RANDDTC is the collected randomization moment RANDDTTM written as text at whole-second precision, the date and time joined by T and the seconds always shown, so a moment collected as 10:30 is written 10:30:00. RANDDTC has no value when no moment was collected. RANDDTC is written from the collected moment independently of the collected date: a subject with a collected moment but no collected date still has RANDDTC, while RANDDT and RANDDY have no value.

Read the source datasets from /app/input and save the completed dataset as /app/output/adsl.csv.

prompts/adam-adsl-randomization/conventions.md

Edit
14 lines

Following CDISC ADaM standards, use the provided DM and ODM datasets to create an ADSL dataset with one record per subject.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, RANDDT, RANDDY, RANDDTC

RANDDY is the study day of the randomization date RANDDT, measured from RFSTDTC, the subject's reference start date. RANDDTC is the collected randomization moment RANDDTTM written as text at whole-second precision, the date and time joined by T and the seconds always shown, so a moment collected as 10:30 is written 10:30:00, or has no value.

Read the source datasets from /app/input and save the completed dataset as /app/output/adsl.csv.

prompts/adam-adsl-randomization/brief.md

Edit
8 lines

Following CDISC ADaM standards, use the provided DM and ODM datasets to create an ADSL dataset with one record per subject.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, RANDDT, RANDDY, RANDDTC

Read the source datasets from /app/input and save the completed dataset as /app/output/adsl.csv.

yamaa code

Public calls generated from benchmark.yaml; results must match the expected output

R

Edit
3 lines
library(yamaa)yamaa_spec("spec.yaml")$run()$output

Python

Edit
3 lines
import yamaayamaa.spec("spec.yaml").run().output

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/adam-adsl-randomization/result.R

Edit
31 lines
# Reference solution for the yamaa benchmark adam-adsl-randomization (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)dm <- nanoparquet::read_parquet("/app/input/dm.parquet")# Study day from the reference start: the reference day itself is day 1,# later dates count forward inclusively, earlier dates count backward with# no day 0, so the day before is day -1.adsl <- dm |>  mutate(    .diff = as.integer(RANDDT - RFSTDTC),    RANDDY = case_when(      is.na(.diff) ~ NA_integer_,      .diff >= 0L ~ .diff + 1L,      .default = .diff    ),    # The collected moment as ISO 8601 text with a T, at whole-second    # precision, written from what was collected even when no date was.    RANDDTC = format(RANDDTTM, "%Y-%m-%dT%H:%M:%S", tz = "UTC")  ) |>  select(STUDYID, USUBJID, RANDDT, RANDDY, RANDDTC) |>  arrange(USUBJID)dir.create("/app/output", showWarnings = FALSE)write_csv(adsl, "/app/output/adsl.csv", na = "")

solutions/adam-adsl-randomization/result.py

Edit
94 lines
# Reference solution for the yamaa benchmark adam-adsl-randomization (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1."""Create the ADSL randomization dataset from the DM and ODM datasets.One record per subject with columns:STUDYID, USUBJID, RANDDT, RANDDY, RANDDTC- RANDDT is the subject's randomization date from DM.- RANDDY is the study day of RANDDT measured from RFSTDTC (the subject's  reference start date): there is no study day 0, so a randomization on  RFSTDTC is day 1 and the day before RFSTDTC is day -1. RANDDY has no  value when RANDDT or RFSTDTC has no value.- RANDDTC is the collected randomization moment RANDDTTM written as text  at whole-second precision (date and time joined by T, seconds always  shown). It is written independently of the collected date, so a  subject with a collected moment but no collected date still has  RANDDTC while RANDDT and RANDDY have no value."""import polars as plINPUT_DIR = "/app/input"OUTPUT_DIR = "/app/output"def check_odm_agrees_with_dm(dm: pl.DataFrame) -> None:    """Confirm the ODM collected DM values agree with the DM dataset."""    odm = pl.read_csv(f"{INPUT_DIR}/odm.csv", infer_schema=False)    items = odm.filter(pl.col("ItemGroupOID") == "IG.DM").pivot(        values="Value", index="SubjectKey", on="ItemOID"    )    for item, column, fmt in (        ("IT.DM.RFSTDTC", "RFSTDTC", "%Y-%m-%d"),        ("IT.DM.RANDDT", "RANDDT", "%Y-%m-%d"),        ("IT.DM.RANDDTTM", "RANDDTTM", "%Y-%m-%d %H:%M:%S"),    ):        if item not in items.columns:            if dm[column].is_not_null().any():                raise ValueError(f"ODM has no {item} but DM has {column}")            continue        collected = items.select(            pl.col("SubjectKey").alias("USUBJID"),            pl.col(item).alias("collected"),        )        joined = dm.join(collected, on="USUBJID", how="left")        typed_text = joined[column].dt.strftime(fmt)        collected_text = joined["collected"]        if column == "RANDDTTM":            # Collected moments may carry only minute precision            # ("2024-03-04T10:30"); normalize to whole seconds.            collected_text = (                pl.when(collected_text.str.len_chars() == 16)                .then(collected_text.str.replace("T", " ") + ":00")                .otherwise(collected_text.str.replace("T", " "))            )        mismatch = joined.filter(            typed_text.is_not_null()            & collected_text.is_not_null()            & (typed_text != collected_text)        )        if mismatch.height:            raise ValueError(                f"ODM collected {column} disagrees with DM for "                f"{mismatch['USUBJID'].to_list()}"            )def main() -> None:    dm = pl.read_parquet(f"{INPUT_DIR}/dm.parquet")    check_odm_agrees_with_dm(dm)    diff_days = (pl.col("RANDDT") - pl.col("RFSTDTC")).dt.total_days()    adsl = dm.select(        "STUDYID",        "USUBJID",        "RANDDT",        pl.when(pl.col("RANDDT").is_null() | pl.col("RFSTDTC").is_null())        .then(None)        .when(diff_days >= 0)        .then(diff_days + 1)        .otherwise(diff_days)        .alias("RANDDY"),        pl.col("RANDDTTM").dt.strftime("%Y-%m-%dT%H:%M:%S").alias("RANDDTC"),    )    adsl.write_csv(f"{OUTPUT_DIR}/adsl.csv")if __name__ == "__main__":    main()

solutions/adam-adsl-randomization/result.sas

Edit
28 lines
/* opensas reference solution for the yamaa benchmark adam-adsl-randomization.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. *//* opensas v0.7.1 has no Parquet reader. The runner must stage dm.parquet   as dm.csv without deriving any values, preserving ISO dates/timestamps. */data dm;  length STUDYID USUBJID $1024 RANDDT RFSTDTC $10 RANDDTTM $32;  infile '/app/input/dm.csv' dsd dlm=',' firstobs=2 truncover;  input STUDYID : $char1024. USUBJID : $char1024. RFSTDTC : $char10. RANDDT : $char10. RANDDTTM : $char32.;run;data result;  length RANDDTC $19;  set dm;  if not missing(RANDDT) and not missing(RFSTDTC) then do;    RANDDY=input(RANDDT,yymmdd10.)-input(RFSTDTC,yymmdd10.);    if RANDDY>=0 then RANDDY=RANDDY+1;  end;  if not missing(RANDDTTM) then do;    RANDDTC=translate(substr(RANDDTTM,1,19),'T',' ');    if lengthn(RANDDTTM)=16 then RANDDTC=cats(RANDDTTM,':00');  end;run;proc sql;  create table final as select STUDYID, USUBJID, RANDDT, RANDDY, RANDDTC from result;quit;proc export data=final outfile="/app/output/adsl.csv" dbms=csv replace; run;