Review ask

As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.

We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.

Summary

Lifecycle: draftEditUnder review
What this benchmark means
README.md

Public entries: Python and R. Passing coverage requires an installed inventory.

Goal: give each electrocardiogram (ECG) time-point record of a QT study its time-matched baseline: flag the baseline records (ABLFL), carry the baseline value (BASE), compute the change from baseline (CHG), and name the baseline each record is compared with (BASETYPE).

Input: one record per subject, parameter, analysis period (APERIOD), nominal day within the period (APERDAY) and nominal time point (ATPT, with its number of hours after the reference dose time in ATPTN), each holding the average of the replicate ECGs taken at that time (AVAL, unit AVALU).

Variables:

  • ABLFL: Y on every record of day -1 of a period, that period's baseline day; no value on every other day.
  • BASE: the value recorded on the same period's baseline day at the same nominal time point for the same subject and parameter; on the baseline day, the record's own value. No value when that baseline day did not record the time point or the period has no baseline day; nothing stands in for it, neither another time point, day or period nor the dosing day's pre-dose value.
  • CHG: AVAL minus BASE, not rounded; no value on the baseline day and wherever BASE has no value.
  • BASETYPE: "Period " followed by the period number, " Baseline Day " and the time point, for example "Period 2 Baseline Day 2 h", on every record, including those with no baseline.

Note: every period has its own baseline day, so a subject has one baseline record per period and time point, and each of them has a baseline type of its own.

Inputs

Under review
1 source file

input/adeg_raw.csv

Edit
17 rows
input/adeg_raw.csv
STUDYIDUSUBJIDPARAMCDPARAMAPERIODAPERDAYATPTATPTNAVALAVALU
TQT-01QTCFQTcF Interval1-10 h0398.000msec
TQT-01QTCFQTcF Interval1-12 h2401.333msec
TQT-01QTCFQTcF Interval110 h0399.667msec
TQT-01QTCFQTcF Interval112 h2407.000msec
TQT-01QTCFQTcF Interval120 h0400.333msec
TQT-01QTCFQTcF Interval122 h2409.667msec
TQT-01QTCFQTcF Interval1-12 h2392.667msec
TQT-01QTCFQTcF Interval112 h2398.333msec
TQT-01QTCFQTcF Interval2-12 h2395.000msec
TQT-01QTCFQTcF Interval212 h2404.667msec
TQT-01QTCFQTcF Interval1-10 h0410.333msec
TQT-01QTCFQTcF Interval110 h0412.000msec
TQT-01QTCFQTcF Interval112 h2418.667msec
TQT-01QTCFQTcF Interval122 h2421.333msec
TQT-01QTCFQTcF Interval1-12 h2385.667msec
TQT-01QTCFQTcF Interval112 h2390.333msec
TQT-01QTCFQTcF Interval212 h2393.000msec

Expected output

Under review
17 expected rows

expected/adeg.csv

Edit
17 rows
expected/adeg.csv
STUDYIDUSUBJIDPARAMCDPARAMAPERIODAPERDAYATPTATPTNAVALAVALUABLFLBASECHGBASETYPE
TQT-01QTCFQTcF Interval1-10 h0398msecY398—Period 1 Baseline Day 0 h
TQT-01QTCFQTcF Interval1-12 h2401.333msecY401.333—Period 1 Baseline Day 2 h
TQT-01QTCFQTcF Interval110 h0399.667msec—3981.6669999999999732Period 1 Baseline Day 0 h
TQT-01QTCFQTcF Interval112 h2407msec—401.3335.666999999999973Period 1 Baseline Day 2 h
TQT-01QTCFQTcF Interval120 h0400.333msec—3982.333000000000027Period 1 Baseline Day 0 h
TQT-01QTCFQTcF Interval122 h2409.667msec—401.3338.333999999999946Period 1 Baseline Day 2 h
TQT-01QTCFQTcF Interval1-12 h2392.667msecY392.667—Period 1 Baseline Day 2 h
TQT-01QTCFQTcF Interval112 h2398.333msec—392.6675.666000000000054Period 1 Baseline Day 2 h
TQT-01QTCFQTcF Interval2-12 h2395msecY395—Period 2 Baseline Day 2 h
TQT-01QTCFQTcF Interval212 h2404.667msec—3959.666999999999973Period 2 Baseline Day 2 h
TQT-01QTCFQTcF Interval1-10 h0410.333msecY410.333—Period 1 Baseline Day 0 h
TQT-01QTCFQTcF Interval110 h0412msec—410.3331.6669999999999732Period 1 Baseline Day 0 h
TQT-01QTCFQTcF Interval112 h2418.667msec———Period 1 Baseline Day 2 h
TQT-01QTCFQTcF Interval122 h2421.333msec———Period 1 Baseline Day 2 h
TQT-01QTCFQTcF Interval1-12 h2385.667msecY385.667—Period 1 Baseline Day 2 h
TQT-01QTCFQTcF Interval112 h2390.333msec—385.6674.666000000000054Period 1 Baseline Day 2 h
TQT-01QTCFQTcF Interval212 h2393msec———Period 2 Baseline Day 2 h
Shaded columns are derived— = missing value

Human review and assessment

Your review

3 questions and your comments · View benchmark discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Dataset NameVariable NameVariable LabelTypeCodelist/Controlled TermsCoreComputational MethodOrigin
ADEGSTUDYIDStudy IdentifiercharacterCopy value from ADEG_RAW.STUDYID.Predecessor
ADEGUSUBJIDUnique Subject IdentifiercharacterCopy value from ADEG_RAW.USUBJID.Predecessor
ADEGPARAMCDParameter CodecharacterCopy value from ADEG_RAW.PARAMCD.Predecessor
ADEGPARAMParametercharacterCopy value from ADEG_RAW.PARAM.Predecessor
ADEGAPERIODPeriodnumericCopy value from ADEG_RAW.APERIOD.Predecessor
ADEGAPERDAYAnalysis Nominal Period DaynumericCopy value from ADEG_RAW.APERDAY.Predecessor
ADEGATPTAnalysis Time PointcharacterCopy value from ADEG_RAW.ATPT.Predecessor
ADEGATPTNAnalysis Time Point (N)numericCopy value from ADEG_RAW.ATPTN.Predecessor
ADEGAVALAnalysis ValuenumericCopy value from ADEG_RAW.AVAL.Predecessor
ADEGAVALUAnalysis Value UnitcharacterCopy value from ADEG_RAW.AVALU.Predecessor
ADEGABLFLBaseline Record FlagcharacterYIf APERDAY = -1 then "Y"; otherwise blank.Derived
ADEGBASEBaseline ValuenumericAggregate ONLY(AVAL) over records grouped by STUDYID, USUBJID, PARAMCD, APERIOD, ATPTN, matching on the shared output keys, where ABLFL = 'Y'.Derived
ADEGCHGChange from BaselinenumericIf ABLFL IS NULL then compute AVAL - BASE; otherwise blank.Derived
ADEGBASETYPEBaseline TypecharacterFill the template "Period {APERIODC} Baseline Day {ATPT}".Derived
VariablePermitted values
ABLFLY
VersionDateAuthorDescriptionReviewerSign-off
1.72026-10-20yamaa docs buildRead a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default
1.62026-10-17yamaa docs buildDescribe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight
1.52026-10-17yamaa docs buildName collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part
1.42026-10-15yamaa docs buildState datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/adam-adeg-time-matched-baseline/full.md

Edit
23 lines

Following CDISC ADaM standards, use the provided ADEG_RAW dataset to create an ADEG dataset with one record per subject, parameter, analysis period, nominal day within the period, and nominal time point.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, APERIOD, APERDAY, ATPT, ATPTN, AVAL, AVALU, ABLFL, BASE, CHG, BASETYPE

  • ABLFL is "Y" on every record of day -1 of a period, that period's baseline day; it has no value on every other day.
  • BASE is the AVAL recorded at the same nominal time point on the same period's baseline day for the same subject and parameter; on the baseline day, BASE is the record's own AVAL.
  • BASE has no value when that baseline day did not record the time point or the period has no baseline day; nothing stands in for it, neither another time point, day or period nor the dosing day's pre-dose value.
  • CHG is AVAL minus BASE and is not rounded; CHG has no value on the baseline day or wherever BASE has no value.
  • BASETYPE is "Period " followed by the period number, " Baseline Day " and ATPT on every record, including those with no baseline.

Read the source datasets from /app/input and save the completed dataset as /app/output/adeg.csv.

prompts/adam-adeg-time-matched-baseline/conventions.md

Edit
13 lines

Following CDISC ADaM standards, use the provided ADEG_RAW dataset to create an ADEG dataset with one record per subject, parameter, analysis period, nominal day within the period, and nominal time point.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, APERIOD, APERDAY, ATPT, ATPTN, AVAL, AVALU, ABLFL, BASE, CHG, BASETYPE

ABLFL is "Y" or has no value. CHG is not rounded. BASETYPE is "Period " followed by the period number, " Baseline Day " and ATPT.

Read the source datasets from /app/input and save the completed dataset as /app/output/adeg.csv.

prompts/adam-adeg-time-matched-baseline/brief.md

Edit
10 lines

Following CDISC ADaM standards, use the provided ADEG_RAW dataset to create an ADEG dataset with one record per subject, parameter, analysis period, nominal day within the period, and nominal time point.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, APERIOD, APERDAY, ATPT, ATPTN, AVAL, AVALU, ABLFL, BASE, CHG, BASETYPE

Read the source datasets from /app/input and save the completed dataset as /app/output/adeg.csv.

yamaa code

Public calls generated from benchmark.yaml; results must match the expected output

R

Edit
3 lines
library(yamaa)yamaa_spec("spec.yaml")$run()$output

Python

Edit
3 lines
import yamaayamaa.spec("spec.yaml").run().output

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/adam-adeg-time-matched-baseline/result.R

Edit
44 lines
# Reference solution for the yamaa benchmark adam-adeg-time-matched-baseline (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)adeg <- read_csv(  "/app/input/adeg_raw.csv",  col_types = cols(    APERIOD = col_integer(), APERDAY = col_integer(),    ATPTN = col_double(), AVAL = col_double(),    .default = col_character()  ),  na = "")# Day -1 of each period is that period's baseline day.baseline <- adeg |>  filter(APERDAY == -1) |>  select(STUDYID, USUBJID, PARAMCD, APERIOD, ATPTN, BASE = AVAL)# Time-matched: the same subject, parameter, period and nominal time on the# period's own baseline day; nothing stands in for a missing one.adeg_out <- adeg |>  left_join(    baseline,    by = c("STUDYID", "USUBJID", "PARAMCD", "APERIOD", "ATPTN"),    relationship = "many-to-one"  ) |>  mutate(    ABLFL = if_else(APERDAY == -1, "Y", NA_character_),    CHG = if_else(APERDAY == -1, NA_real_, AVAL - BASE),    BASETYPE = paste0("Period ", APERIOD, " Baseline Day ", ATPT)  ) |>  select(    STUDYID, USUBJID, PARAMCD, PARAM, APERIOD, APERDAY, ATPT, ATPTN, AVAL,    AVALU, ABLFL, BASE, CHG, BASETYPE  )dir.create("/app/output", showWarnings = FALSE)write_csv(adeg_out, "/app/output/adeg.csv", na = "")

solutions/adam-adeg-time-matched-baseline/result.py

Edit
63 lines
# Reference solution for the yamaa benchmark adam-adeg-time-matched-baseline (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1."""Create ADEG from ADEG_RAW with time-matched period baselines.For each subject, parameter and analysis period, the baseline day isAPERDAY -1. BASE is the AVAL recorded at the same nominal time point onthat period's baseline day; CHG is AVAL - BASE (blank on the baselineday itself and wherever no baseline value exists)."""import csvfrom decimal import DecimalINPUT_PATH = "/app/input/adeg_raw.csv"OUTPUT_PATH = "/app/output/adeg.csv"FIELDS = [    "STUDYID", "USUBJID", "PARAMCD", "PARAM", "APERIOD", "APERDAY",    "ATPT", "ATPTN", "AVAL", "AVALU", "ABLFL", "BASE", "CHG", "BASETYPE",]def main():    with open(INPUT_PATH, newline="") as f:        rows = list(csv.DictReader(f))    # AVAL on each period's baseline day, by subject/parameter/period/time point.    baseline = {}    for row in rows:        if row["APERDAY"] == "-1":            key = (row["USUBJID"], row["PARAMCD"], row["APERIOD"], row["ATPT"])            baseline[key] = row["AVAL"]    out_rows = []    for row in rows:        is_baseline_day = row["APERDAY"] == "-1"        key = (row["USUBJID"], row["PARAMCD"], row["APERIOD"], row["ATPT"])        base = baseline.get(key, "")        if is_baseline_day or base == "":            chg = ""        else:            chg = str(Decimal(row["AVAL"]) - Decimal(base))        out = {name: row[name] for name in FIELDS[:10]}        out["ABLFL"] = "Y" if is_baseline_day else ""        out["BASE"] = base        out["CHG"] = chg        out["BASETYPE"] = (            "Period " + row["APERIOD"] + " Baseline Day " + row["ATPT"]        )        out_rows.append(out)    with open(OUTPUT_PATH, "w", newline="") as f:        writer = csv.DictWriter(f, fieldnames=FIELDS)        writer.writeheader()        writer.writerows(out_rows)if __name__ == "__main__":    main()

solutions/adam-adeg-time-matched-baseline/result.sas

Edit
48 lines
/* opensas reference solution for the yamaa benchmark adam-adeg-time-matched-baseline.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data adeg;  length STUDYID USUBJID PARAMCD PARAM ATPT AVALU $1024;  infile "/app/input/adeg_raw.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. PARAMCD : $char1024. PARAM : $char1024. APERIOD APERDAY ATPT : $char1024. ATPTN AVAL AVALU : $char1024.;run;/* Day -1 of each period is that period's baseline day; its records are the   time-matched baselines, one per subject, parameter, period and time. */data baseline;  length B_STUDYID B_USUBJID B_PARAMCD $1024;  set adeg;  if APERDAY = -1;  B_STUDYID = STUDYID; B_USUBJID = USUBJID; B_PARAMCD = PARAMCD;  B_APERIOD = APERIOD; B_ATPTN = ATPTN; BASE = AVAL;  keep B_STUDYID B_USUBJID B_PARAMCD B_APERIOD B_ATPTN BASE;run;proc sql;  create table matched as    select a.*, b.BASE    from adeg as a left join baseline as b      on a.STUDYID = b.B_STUDYID and a.USUBJID = b.B_USUBJID      and a.PARAMCD = b.B_PARAMCD and a.APERIOD = b.B_APERIOD      and a.ATPTN = b.B_ATPTN;quit;/* No change on the baseline day, and none without a baseline. */data final;  length ABLFL $1 BASETYPE $1024;  set matched;  if APERDAY = -1 then ABLFL = 'Y';  else ABLFL = '';  if APERDAY ne -1 and not missing(BASE) and not missing(AVAL) then CHG = AVAL - BASE;  else CHG = .;  BASETYPE = 'Period ' || strip(put(APERIOD, 8.)) || ' Baseline Day ' || strip(ATPT);run;proc sql;  create table out as    select STUDYID, USUBJID, PARAMCD, PARAM, APERIOD, APERDAY, ATPT, ATPTN,           AVAL, AVALU, ABLFL, BASE, CHG, BASETYPE    from final;quit;proc export data=out outfile="/app/output/adeg.csv" dbms=csv replace; run;