Review ask

As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.

We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.

Summary

Lifecycle: draftEditUnder review
What this benchmark means
README.md

Public entries: Python and R. Passing coverage requires an installed inventory.

Goal: one average record (DTYPE AVERAGE) per subject (USUBJID), planned time point (ATPTN) and parameter (PARAMCD) over the replicate electrocardiograms (ECGs) taken at that time point: the QT interval, the RR interval, and the QT interval corrected for heart rate by Fridericia's formula (QTcF).

Input: SDTM ECG test results, one record per ECG test per replicate per planned time point. Each ECG carries an aggregate QT result (QTAG) and an aggregate RR result (RRAG) in milliseconds; a test that was not done has EGSTAT NOT DONE and no result.

Variables:

  • PARAMCD: QTAG, RRAG, or QTCF.
  • PARAM: "QT Interval, Aggregate (msec)", "RR Interval, Aggregate (msec)", or "QTcF Interval, Aggregate (msec)".
  • ATPT and ATPTN: the planned time point name and number of the ECGs averaged (EGTPT, EGTPTNUM).
  • DTYPE: AVERAGE on every record.
  • AVAL: the mean, over the time point's ECGs that have a value for the parameter, of that value, not rounded. For QTCF the value of one ECG is its own QT divided by the cube root of its own RR in seconds, so the average is the mean of the per-ECG corrected values, not the correction of the mean QT and RR; an ECG without both results has no corrected value. A single ECG gives a record of its own values.

Note: the QT and RR results of one ECG share its replicate number (EGREPNUM) within the time point, even when two ECGs carry the same collection date and time. A parameter with no value at a time point has no record there.

Inputs

Under review
1 source file

input/eg.csv

Edit
22 rows
input/eg.csv
STUDYIDDOMAINUSUBJIDEGSEQEGTESTCDEGTESTEGREPNUMEGSTATEGSTRESNEGSTRESUEGTPTEGTPTNUMEGDTC
YAMAA-01EG1QTAGQT Interval, Aggregate1—400msec0.5 HR0.52026-03-02T08:30
YAMAA-01EG2RRAGRR Interval, Aggregate1—1000msec0.5 HR0.52026-03-02T08:30
YAMAA-01EG3QTAGQT Interval, Aggregate2—392msec0.5 HR0.52026-03-02T08:32
YAMAA-01EG4RRAGRR Interval, Aggregate2—900msec0.5 HR0.52026-03-02T08:32
YAMAA-01EG5QTAGQT Interval, Aggregate3—408msec0.5 HR0.52026-03-02T08:34
YAMAA-01EG6RRAGRR Interval, Aggregate3—1100msec0.5 HR0.52026-03-02T08:34
YAMAA-01EG7QTAGQT Interval, Aggregate1—396msec1 HR12026-03-02T09:00
YAMAA-01EG8RRAGRR Interval, Aggregate1—960msec1 HR12026-03-02T09:00
YAMAA-01EG1QTAGQT Interval, Aggregate1—380msec0.5 HR0.52026-03-02T08:30
YAMAA-01EG2RRAGRR Interval, Aggregate1—800msec0.5 HR0.52026-03-02T08:30
YAMAA-01EG3QTAGQT Interval, Aggregate2—390msec0.5 HR0.52026-03-02T08:32
YAMAA-01EG4RRAGRR Interval, Aggregate2NOT DONE——0.5 HR0.52026-03-02T08:32
YAMAA-01EG5QTAGQT Interval, Aggregate3—388msec0.5 HR0.52026-03-02T08:34
YAMAA-01EG6RRAGRR Interval, Aggregate3—840msec0.5 HR0.52026-03-02T08:34
YAMAA-01EG7RRAGRR Interval, Aggregate1—780msec1 HR12026-03-02T09:00
YAMAA-01EG8RRAGRR Interval, Aggregate2—880msec1 HR12026-03-02T09:00
YAMAA-01EG9QTAGQT Interval, Aggregate1—376msec1 HR12026-03-02T09:00
YAMAA-01EG10QTAGQT Interval, Aggregate2—384msec1 HR12026-03-02T09:00
YAMAA-01EG1QTAGQT Interval, Aggregate1NOT DONE——0.5 HR0.52026-03-03T08:30
YAMAA-01EG2RRAGRR Interval, Aggregate1—1000msec0.5 HR0.52026-03-03T08:30
YAMAA-01EG3QTAGQT Interval, Aggregate2NOT DONE——0.5 HR0.52026-03-03T08:32
YAMAA-01EG4RRAGRR Interval, Aggregate2—1040msec0.5 HR0.52026-03-03T08:32

Expected output

Under review
13 expected rows

expected/adeg.csv

Edit
13 rows
expected/adeg.csv
STUDYIDUSUBJIDPARAMCDPARAMATPTATPTNDTYPEAVAL
YAMAA-01QTAGQT Interval, Aggregate (msec)0.5 HR0.5AVERAGE400
YAMAA-01QTAGQT Interval, Aggregate (msec)1 HR1AVERAGE396
YAMAA-01QTCFQTcF Interval, Aggregate (msec)0.5 HR0.5AVERAGE400.4177570070339
YAMAA-01QTCFQTcF Interval, Aggregate (msec)1 HR1AVERAGE401.4253317078444
YAMAA-01RRAGRR Interval, Aggregate (msec)0.5 HR0.5AVERAGE1000
YAMAA-01RRAGRR Interval, Aggregate (msec)1 HR1AVERAGE960
YAMAA-01QTAGQT Interval, Aggregate (msec)0.5 HR0.5AVERAGE386
YAMAA-01QTAGQT Interval, Aggregate (msec)1 HR1AVERAGE380
YAMAA-01QTCFQTcF Interval, Aggregate (msec)0.5 HR0.5AVERAGE410.28022314500424
YAMAA-01QTCFQTcF Interval, Aggregate (msec)1 HR1AVERAGE404.59133343180497
YAMAA-01RRAGRR Interval, Aggregate (msec)0.5 HR0.5AVERAGE820
YAMAA-01RRAGRR Interval, Aggregate (msec)1 HR1AVERAGE830
YAMAA-01RRAGRR Interval, Aggregate (msec)0.5 HR0.5AVERAGE1020
Shaded columns are derived— = missing value

Human review and assessment

Your review

3 questions and your comments · View benchmark discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Dataset NameVariable NameVariable LabelTypeCodelist/Controlled TermsCoreComputational MethodOrigin
ADEGSTUDYIDStudy IdentifiercharacterSet per constructed row; see the Row construction sheet.Derived
ADEGUSUBJIDUnique Subject IdentifiercharacterSet per constructed row; see the Row construction sheet.Derived
ADEGPARAMCDParameter CodecharacterQTAG, RRAG, QTCFSet per constructed row; see the Row construction sheet.Derived
ADEGPARAMParametercharacterSet per constructed row; see the Row construction sheet.Derived
ADEGATPTAnalysis TimepointcharacterSet per constructed row; see the Row construction sheet.Derived
ADEGATPTNAnalysis Timepoint (N)numericSet per constructed row; see the Row construction sheet.Derived
ADEGDTYPEDerivation TypecharacterSet constant value "AVERAGE".Assigned
ADEGAVALAnalysis ValuenumericSet per constructed row; see the Row construction sheet.Derived
TemplateInclude whenSTUDYIDUSUBJIDPARAMCDPARAMATPTATPTNAVAL
qt_averageAVAL IS NOT NULLCopy EG.STUDYIDCopy EG.USUBJIDConstant "QTAG"Constant "QT Interval, Aggregate (msec)"Copy EG.EGTPTCopy EG.EGTPTNUMAggregate MEAN(EG.EGSTRESN) over records matching on the shared output keys, where EG.EGTESTCD = 'QTAG'
rr_averageAVAL IS NOT NULLCopy EG.STUDYIDCopy EG.USUBJIDConstant "RRAG"Constant "RR Interval, Aggregate (msec)"Copy EG.EGTPTCopy EG.EGTPTNUMAggregate MEAN(EG.EGSTRESN) over records matching on the shared output keys, where EG.EGTESTCD = 'RRAG'
qtcf_averageAVAL IS NOT NULLCopy STUDYID from the EG record matching on the shared output keys, where EG.EGTESTCD = 'QTAG', and no match yields blankCopy USUBJID from the EG record matching on the shared output keys, where EG.EGTESTCD = 'QTAG', and no match yields blankConstant "QTCF"Constant "QTcF Interval, Aggregate (msec)"Copy EGTPT from the EG record matching on the shared output keys, where EG.EGTESTCD = 'QTAG', and no match yields blankCopy EGTPTNUM from the EG record matching on the shared output keys, where EG.EGTESTCD = 'QTAG', and no match yields blankAggregate MEAN(EGSTRESN / POWER(NULLIF(RRPAIR, 0) / 1000, 1 / 3)), with EGSTRESN taken from the EG record matching on the shared output keys, where EG.EGTESTCD = 'QTAG', and no match yields blank and RRPAIR taken from the EG record matching on the shared output keys, where EG.EGTESTCD = 'QTAG', where RRPAIR is derived as EGSTRESN from the EG record matching on STUDYID, USUBJID, EGTPTNUM and EGREPNUM, where EG.EGTESTCD = 'RRAG', "0" when unmatched, and no match yields blank over records matching on the shared output keys
VariablePermitted values
PARAMCDQTAG, RRAG, QTCF
VersionDateAuthorDescriptionReviewerSign-off
1.72026-10-20yamaa docs buildRead a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default
1.62026-10-17yamaa docs buildDescribe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight
1.52026-10-17yamaa docs buildName collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part
1.42026-10-15yamaa docs buildState datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/adam-adeg-replicate-qtc-average/full.md

Edit
27 lines

Following CDISC ADaM standards, use the provided EG dataset to create an ADEG dataset with one record per subject, planned time point and parameter.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, ATPT, ATPTN, DTYPE, AVAL

PARAMCD is QTAG, RRAG, or QTCF. PARAM is "QT Interval, Aggregate (msec)" when PARAMCD is QTAG, "RR Interval, Aggregate (msec)" when PARAMCD is RRAG, and "QTcF Interval, Aggregate (msec)" when PARAMCD is QTCF. ATPT and ATPTN are the planned time point name and number (EGTPT and EGTPTNUM) of the ECGs averaged. DTYPE is AVERAGE on every record.

AVAL is the mean, over the time point's ECGs that have a value for the parameter, of that value, and is not rounded. For QTAG the value of one ECG is its QT result, and for RRAG it is its RR result. An ECG whose test was not done has no result and does not enter the mean. For QTCF the value of one ECG is its own QT divided by the cube root of its own RR in seconds, so AVAL is the mean of the per-ECG corrected values, not the correction of the mean QT and RR. The QT and RR results of one ECG are the pair that shares its replicate number within the subject and time point, even when two ECGs carry the same collection date and time. An ECG without both results, or with an RR of zero, has no corrected value. A single ECG at a time point gives a record of its own values. A parameter with no value at a time point has no record there.

Read the source datasets from /app/input and save the completed dataset as /app/output/adeg.csv.

prompts/adam-adeg-replicate-qtc-average/conventions.md

Edit
13 lines

Following CDISC ADaM standards, use the provided EG dataset to create an ADEG dataset with one record per subject, planned time point and parameter.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, ATPT, ATPTN, DTYPE, AVAL

PARAMCD is QTAG, RRAG, or QTCF. PARAM is "QT Interval, Aggregate (msec)" when PARAMCD is QTAG, "RR Interval, Aggregate (msec)" when PARAMCD is RRAG, and "QTcF Interval, Aggregate (msec)" when PARAMCD is QTCF. DTYPE is AVERAGE on every record. AVAL is not rounded.

Read the source datasets from /app/input and save the completed dataset as /app/output/adeg.csv.

prompts/adam-adeg-replicate-qtc-average/brief.md

Edit
8 lines

Following CDISC ADaM standards, use the provided EG dataset to create an ADEG dataset with one record per subject, planned time point and parameter.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, ATPT, ATPTN, DTYPE, AVAL

Read the source datasets from /app/input and save the completed dataset as /app/output/adeg.csv.

yamaa code

Public calls generated from benchmark.yaml; results must match the expected output

R

Edit
3 lines
library(yamaa)yamaa_spec("spec.yaml")$run()$output

Python

Edit
3 lines
import yamaayamaa.spec("spec.yaml").run().output

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/adam-adeg-replicate-qtc-average/result.R

Edit
53 lines
# Reference solution for the yamaa benchmark adam-adeg-replicate-qtc-average (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)eg <- read_csv(  "/app/input/eg.csv",  col_types = cols(    EGSTRESN = col_double(),    EGTPTNUM = col_double(),    .default = col_character()  ))qt <- eg |> filter(EGTESTCD == "QTAG")rr <- eg |> filter(EGTESTCD == "RRAG")# One ECG is one replicate number at a time point: its QT pairs with the RR# of the same replicate number, never by collection time.pairs <- qt |>  select(STUDYID, USUBJID, EGTPTNUM, EGTPT, EGREPNUM, QT = EGSTRESN) |>  left_join(    rr |> select(STUDYID, USUBJID, EGTPTNUM, EGREPNUM, RR = EGSTRESN),    by = c("STUDYID", "USUBJID", "EGTPTNUM", "EGREPNUM")  ) |>  # Fridericia's correction of each ECG: QT over the cube root of RR in  # seconds.  mutate(QTCF = QT / (RR / 1000)^(1 / 3))# The mean over the ECGs that have a value; no value, no record.averages <- function(data, value, paramcd, param) {  data |>    filter(!is.na(.data[[value]])) |>    group_by(STUDYID, USUBJID, EGTPTNUM, EGTPT) |>    summarise(AVAL = mean(.data[[value]]), .groups = "drop") |>    mutate(PARAMCD = paramcd, PARAM = param)}adeg <- bind_rows(  averages(qt, "EGSTRESN", "QTAG", "QT Interval, Aggregate (msec)"),  averages(rr, "EGSTRESN", "RRAG", "RR Interval, Aggregate (msec)"),  averages(pairs, "QTCF", "QTCF", "QTcF Interval, Aggregate (msec)")) |>  mutate(ATPT = EGTPT, ATPTN = EGTPTNUM, DTYPE = "AVERAGE") |>  arrange(STUDYID, USUBJID, PARAMCD, ATPTN) |>  select(STUDYID, USUBJID, PARAMCD, PARAM, ATPT, ATPTN, DTYPE, AVAL)dir.create("/app/output", showWarnings = FALSE)write_csv(adeg, "/app/output/adeg.csv", na = "")

solutions/adam-adeg-replicate-qtc-average/result.py

Edit
82 lines
# Reference solution for the yamaa benchmark adam-adeg-replicate-qtc-average (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.import csvfrom collections import OrderedDictINPUT = "/app/input"OUTPUT = "/app/output"PARAMS = [    ("QTAG", "QT Interval, Aggregate (msec)"),    ("RRAG", "RR Interval, Aggregate (msec)"),    ("QTCF", "QTcF Interval, Aggregate (msec)"),]def main():    with open(f"{INPUT}/eg.csv", newline="") as f:        rows = list(csv.DictReader(f))    # Group rows by subject and planned time point, preserving input order.    groups = OrderedDict()    for row in rows:        key = (row["STUDYID"], row["USUBJID"], row["EGTPT"], row["EGTPTNUM"])        groups.setdefault(key, []).append(row)    out = []    for (studyid, usubjid, atpt, atptn), grows in groups.items():        qt = {}  # EGREPNUM -> QT result (msec)        rr = {}  # EGREPNUM -> RR result (msec)        for row in grows:            value = row["EGSTRESN"].strip()            if value == "":                # A test that was not done has no result.                continue            if row["EGTESTCD"] == "QTAG":                qt[row["EGREPNUM"]] = float(value)            elif row["EGTESTCD"] == "RRAG":                rr[row["EGREPNUM"]] = float(value)        values = {}        if qt:            values["QTAG"] = sum(qt.values()) / len(qt)        if rr:            values["RRAG"] = sum(rr.values()) / len(rr)        corrected = []        for repnum, qtval in qt.items():            if repnum in rr and rr[repnum] != 0:                corrected.append(qtval / (rr[repnum] / 1000.0) ** (1.0 / 3.0))        if corrected:            values["QTCF"] = sum(corrected) / len(corrected)        for paramcd, param in PARAMS:            if paramcd not in values:                # A parameter with no value at a time point has no record.                continue            out.append(                {                    "STUDYID": studyid,                    "USUBJID": usubjid,                    "PARAMCD": paramcd,                    "PARAM": param,                    "ATPT": atpt,                    "ATPTN": atptn,                    "DTYPE": "AVERAGE",                    "AVAL": repr(values[paramcd]),                }            )    columns = ["STUDYID", "USUBJID", "PARAMCD", "PARAM", "ATPT", "ATPTN",               "DTYPE", "AVAL"]    with open(f"{OUTPUT}/adeg.csv", "w", newline="") as f:        writer = csv.DictWriter(f, fieldnames=columns)        writer.writeheader()        writer.writerows(out)if __name__ == "__main__":    main()

solutions/adam-adeg-replicate-qtc-average/result.sas

Edit
82 lines
/* opensas reference solution for the yamaa benchmark adam-adeg-replicate-qtc-average.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data eg;  length STUDYID DOMAIN USUBJID EGTESTCD EGTEST EGSTAT EGSTRESU EGTPT EGDTC $1024;  infile "/app/input/eg.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. DOMAIN : $char1024. USUBJID : $char1024. EGSEQ        EGTESTCD : $char1024. EGTEST : $char1024. EGREPNUM EGSTAT : $char1024.        EGSTRESN EGSTRESU : $char1024. EGTPT : $char1024. EGTPTNUM        EGDTC : $char1024.;run;/* The QT records, and the RR records under their own column names. */data qt;  set eg;  if EGTESTCD='QTAG';  QT=EGSTRESN;  keep STUDYID USUBJID EGTPT EGTPTNUM EGREPNUM QT;run;data rr;  length R_STUDYID R_USUBJID R_TPT $1024;  set eg;  if EGTESTCD='RRAG';  R_STUDYID=STUDYID; R_USUBJID=USUBJID; R_TPT=EGTPT;  R_TPTNUM=EGTPTNUM; R_REPNUM=EGREPNUM;  RR=EGSTRESN;  keep R_STUDYID R_USUBJID R_TPT R_TPTNUM R_REPNUM RR;run;/* One ECG is one replicate number at a time point: its QT pairs with the   RR of the same replicate number, never by collection time. */proc sql;  create table pairs as    select a.STUDYID, a.USUBJID, a.EGTPT, a.EGTPTNUM, a.QT, b.RR    from qt as a left join rr as b      on a.STUDYID=b.R_STUDYID and a.USUBJID=b.R_USUBJID         and a.EGTPTNUM=b.R_TPTNUM and a.EGREPNUM=b.R_REPNUM;quit;/* Fridericia's correction of each ECG: QT over the cube root of RR in   seconds; an ECG without both results has no corrected value. */data pairs;  set pairs;  if not missing(QT) and not missing(RR) then QTCF=QT/(RR/1000)**(1/3);  else QTCF=.;run;/* The mean over the ECGs that have a value; no value, no record. */proc sql;  create table qtavg as    select STUDYID, USUBJID, EGTPT, EGTPTNUM, mean(QT) as AVAL    from qt where QT is not missing    group by STUDYID, USUBJID, EGTPTNUM, EGTPT;  create table rravg as    select R_STUDYID as STUDYID, R_USUBJID as USUBJID, R_TPT as EGTPT,           R_TPTNUM as EGTPTNUM, mean(RR) as AVAL    from rr where RR is not missing    group by R_STUDYID, R_USUBJID, R_TPTNUM, R_TPT;  create table qtcfavg as    select STUDYID, USUBJID, EGTPT, EGTPTNUM, mean(QTCF) as AVAL    from pairs where QTCF is not missing    group by STUDYID, USUBJID, EGTPTNUM, EGTPT;quit;data result;  length PARAMCD PARAM ATPT DTYPE $1024;  set qtavg(in=inqt) rravg(in=inrr) qtcfavg(in=inqtcf);  if inqt then do; PARAMCD='QTAG'; PARAM='QT Interval, Aggregate (msec)'; end;  else if inrr then do; PARAMCD='RRAG'; PARAM='RR Interval, Aggregate (msec)'; end;  else do; PARAMCD='QTCF'; PARAM='QTcF Interval, Aggregate (msec)'; end;  ATPT=EGTPT; ATPTN=EGTPTNUM; DTYPE='AVERAGE';run;proc sort data=result; by STUDYID USUBJID PARAMCD ATPTN; run;proc sql;  create table final as    select STUDYID, USUBJID, PARAMCD, PARAM, ATPT, ATPTN, DTYPE, AVAL    from result;quit;proc export data=final outfile="/app/output/adeg.csv" dbms=csv replace; run;