Review ask

As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.

We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.

Summary

Lifecycle: draftEditUnder review
What this benchmark means
README.md

Public entries: Python and R. Passing coverage requires an installed inventory.

Goal: derive the criteria CRIT1 to CRIT5 and their flags: whether an on-treatment Fridericia-corrected QT interval (QTcF) lies above 450, 480 or 500 milliseconds (ms), and whether its increase from baseline lies above 30 or 60 ms.

Input: electrocardiogram (ECG) analysis records, one per subject, parameter and analysis visit, with the value AVAL in ms, the baseline flag ABLFL, the on-treatment flag ONTRTFL, and the baseline value BASE and change from baseline CHG already in place.

Variables:

  • CRIT1, CRIT2, CRIT3: QTcF > 450 ms, QTcF > 480 ms and QTcF > 500 ms; empty when the record is not assessed or AVAL is missing.
  • CRIT1FL, CRIT2FL, CRIT3FL: Y when AVAL is above the limit and N otherwise, so a value exactly at a limit reads N.
  • CRIT4, CRIT5: QTcF increase from baseline > 30 ms and QTcF increase from baseline > 60 ms; empty when the record is not assessed or has no CHG, as without a baseline value, while its absolute criteria are still assessed.
  • CRIT4FL, CRIT5FL: Y when CHG is above the limit and N otherwise: an increase of exactly 30 reads N, and a decrease never meets them, however large.

Note: only on-treatment QTcF records (ONTRTFL is Y) are assessed; baseline records, records after the on-treatment window and other parameters, such as the uncorrected QT interval, carry no criterion. Values are compared unrounded, and each flag is empty exactly where its criterion is. A baseline record marked on treatment, or a change without a baseline value, stops the run.

Inputs

Under review
1 source file

input/adeg_raw.csv

Edit
16 rows
input/adeg_raw.csv
STUDYIDUSUBJIDPARAMCDPARAMAVISITAVALABLFLONTRTFLBASECHG
YAMAA-01QTCFQTcF Interval (ms)BASELINE420Y—420—
YAMAA-01QTCFQTcF Interval (ms)WEEK 2450—Y42030
YAMAA-01QTCFQTcF Interval (ms)WEEK 4450.4—Y42030.4
YAMAA-01QTCFQTcF Interval (ms)FOLLOW-UP482——42062
YAMAA-01QTCFQTcF Interval (ms)BASELINE385Y—385—
YAMAA-01QTCFQTcF Interval (ms)WEEK 2445—Y38560
YAMAA-01QTCFQTcF Interval (ms)WEEK 4445.4—Y38560.4
YAMAA-01QTCFQTcF Interval (ms)WEEK 8——Y385—
YAMAA-01QTCFQTcF Interval (ms)BASELINE490Y—490—
YAMAA-01QTCFQTcF Interval (ms)WEEK 2480—Y490-10
YAMAA-01QTCFQTcF Interval (ms)WEEK 4480.4—Y490-9.6
YAMAA-01QTCFQTcF Interval (ms)WEEK 8455—Y490-35
YAMAA-01QTQT Interval (ms)BASELINE430Y—430—
YAMAA-01QTQT Interval (ms)WEEK 2470—Y43040
YAMAA-01QTCFQTcF Interval (ms)WEEK 2500—Y——
YAMAA-01QTCFQTcF Interval (ms)WEEK 4500.4—Y——

Expected output

Under review
16 expected rows

expected/adeg.csv

Edit
16 rows
expected/adeg.csv
STUDYIDUSUBJIDPARAMCDPARAMAVISITAVALABLFLONTRTFLBASECHGCRIT1CRIT1FLCRIT2CRIT2FLCRIT3CRIT3FLCRIT4CRIT4FLCRIT5CRIT5FL
YAMAA-01QTCFQTcF Interval (ms)BASELINE420Y—420———————————
YAMAA-01QTCFQTcF Interval (ms)WEEK 2450—Y42030QTcF > 450 msNQTcF > 480 msNQTcF > 500 msNQTcF increase from baseline > 30 msNQTcF increase from baseline > 60 msN
YAMAA-01QTCFQTcF Interval (ms)WEEK 4450.4—Y42030.4QTcF > 450 msYQTcF > 480 msNQTcF > 500 msNQTcF increase from baseline > 30 msYQTcF increase from baseline > 60 msN
YAMAA-01QTCFQTcF Interval (ms)FOLLOW-UP482——42062——————————
YAMAA-01QTCFQTcF Interval (ms)BASELINE385Y—385———————————
YAMAA-01QTCFQTcF Interval (ms)WEEK 2445—Y38560QTcF > 450 msNQTcF > 480 msNQTcF > 500 msNQTcF increase from baseline > 30 msYQTcF increase from baseline > 60 msN
YAMAA-01QTCFQTcF Interval (ms)WEEK 4445.4—Y38560.4QTcF > 450 msNQTcF > 480 msNQTcF > 500 msNQTcF increase from baseline > 30 msYQTcF increase from baseline > 60 msY
YAMAA-01QTCFQTcF Interval (ms)WEEK 8——Y385———————————
YAMAA-01QTCFQTcF Interval (ms)BASELINE490Y—490———————————
YAMAA-01QTCFQTcF Interval (ms)WEEK 2480—Y490-10QTcF > 450 msYQTcF > 480 msNQTcF > 500 msNQTcF increase from baseline > 30 msNQTcF increase from baseline > 60 msN
YAMAA-01QTCFQTcF Interval (ms)WEEK 4480.4—Y490-9.6QTcF > 450 msYQTcF > 480 msYQTcF > 500 msNQTcF increase from baseline > 30 msNQTcF increase from baseline > 60 msN
YAMAA-01QTCFQTcF Interval (ms)WEEK 8455—Y490-35QTcF > 450 msYQTcF > 480 msNQTcF > 500 msNQTcF increase from baseline > 30 msNQTcF increase from baseline > 60 msN
YAMAA-01QTQT Interval (ms)BASELINE430Y—430———————————
YAMAA-01QTQT Interval (ms)WEEK 2470—Y43040——————————
YAMAA-01QTCFQTcF Interval (ms)WEEK 2500—Y——QTcF > 450 msYQTcF > 480 msYQTcF > 500 msN————
YAMAA-01QTCFQTcF Interval (ms)WEEK 4500.4—Y——QTcF > 450 msYQTcF > 480 msYQTcF > 500 msY————
Shaded columns are derived— = missing value

Human review and assessment

Your review

3 questions and your comments · View benchmark discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Dataset NameVariable NameVariable LabelTypeCodelist/Controlled TermsCoreComputational MethodOrigin
ADEGSTUDYIDStudy IdentifiercharacterCopy value from ADEG_RAW.STUDYID.Predecessor
ADEGUSUBJIDUnique Subject IdentifiercharacterCopy value from ADEG_RAW.USUBJID.Predecessor
ADEGPARAMCDParameter CodecharacterCopy value from ADEG_RAW.PARAMCD.Predecessor
ADEGPARAMParametercharacterCopy value from ADEG_RAW.PARAM.Predecessor
ADEGAVISITAnalysis VisitcharacterCopy value from ADEG_RAW.AVISIT.Predecessor
ADEGAVALAnalysis ValuenumericCopy value from ADEG_RAW.AVAL.Predecessor
ADEGABLFLBaseline Record FlagcharacterYCopy value from ADEG_RAW.ABLFL.Predecessor
ADEGONTRTFLOn Treatment Record FlagcharacterYCopy value from ADEG_RAW.ONTRTFL.Predecessor
ADEGBASEBaseline ValuenumericCopy value from ADEG_RAW.BASE.Predecessor
ADEGCHGChange from BaselinenumericCopy value from ADEG_RAW.CHG.Predecessor
ADEGCRIT1Analysis Criterion 1characterIf PARAMCD = 'QTCF' AND ONTRTFL = 'Y' AND AVAL IS NOT NULL then "QTcF > 450 ms"; otherwise blank.Derived
ADEGCRIT1FLCriterion 1 Evaluation Result FlagcharacterY, NIf CRIT1 IS NULL then "None"; If AVAL > 450 then "Y"; otherwise "N".Derived
ADEGCRIT2Analysis Criterion 2characterIf PARAMCD = 'QTCF' AND ONTRTFL = 'Y' AND AVAL IS NOT NULL then "QTcF > 480 ms"; otherwise blank.Derived
ADEGCRIT2FLCriterion 2 Evaluation Result FlagcharacterY, NIf CRIT2 IS NULL then "None"; If AVAL > 480 then "Y"; otherwise "N".Derived
ADEGCRIT3Analysis Criterion 3characterIf PARAMCD = 'QTCF' AND ONTRTFL = 'Y' AND AVAL IS NOT NULL then "QTcF > 500 ms"; otherwise blank.Derived
ADEGCRIT3FLCriterion 3 Evaluation Result FlagcharacterY, NIf CRIT3 IS NULL then "None"; If AVAL > 500 then "Y"; otherwise "N".Derived
ADEGCRIT4Analysis Criterion 4characterIf PARAMCD = 'QTCF' AND ONTRTFL = 'Y' AND CHG IS NOT NULL then "QTcF increase from baseline > 30 ms"; otherwise blank.Derived
ADEGCRIT4FLCriterion 4 Evaluation Result FlagcharacterY, NIf CRIT4 IS NULL then "None"; If CHG > 30 then "Y"; otherwise "N".Derived
ADEGCRIT5Analysis Criterion 5characterIf PARAMCD = 'QTCF' AND ONTRTFL = 'Y' AND CHG IS NOT NULL then "QTcF increase from baseline > 60 ms"; otherwise blank.Derived
ADEGCRIT5FLCriterion 5 Evaluation Result FlagcharacterY, NIf CRIT5 IS NULL then "None"; If CHG > 60 then "Y"; otherwise "N".Derived
VariablePermitted values
ABLFLY
ONTRTFLY
CRIT1FLY, N
CRIT2FLY, N
CRIT3FLY, N
CRIT4FLY, N
CRIT5FLY, N
VersionDateAuthorDescriptionReviewerSign-off
1.72026-10-20yamaa docs buildRead a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default
1.62026-10-17yamaa docs buildDescribe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight
1.52026-10-17yamaa docs buildName collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part
1.42026-10-15yamaa docs buildState datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/adam-adeg-qtc-flags/full.md

Edit
30 lines

Following CDISC ADaM standards, use the provided ADEG_RAW dataset to create an ADEG dataset with one record per subject per parameter per analysis visit.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, AVISIT, AVAL, ABLFL, ONTRTFL, BASE, CHG, CRIT1, CRIT1FL, CRIT2, CRIT2FL, CRIT3, CRIT3FL, CRIT4, CRIT4FL, CRIT5, CRIT5FL

Only records with PARAMCD "QTCF" and ONTRTFL "Y" are assessed. Baseline records, records after the on-treatment window, and records of other parameters, such as the uncorrected QT interval, carry no criterion.

  • CRIT1 is "QTcF > 450 ms", CRIT2 is "QTcF > 480 ms", and CRIT3 is "QTcF > 500 ms" on an assessed record; each has no value when the record is not assessed or AVAL has no value.
  • CRIT1FL is Y when AVAL is above 450, CRIT2FL is Y when AVAL is above 480, and CRIT3FL is Y when AVAL is above 500, each N otherwise, so a value exactly at a limit reads N.
  • CRIT4 is "QTcF increase from baseline > 30 ms" and CRIT5 is "QTcF increase from baseline > 60 ms" on an assessed record; each has no value when the record is not assessed or CHG has no value, as on a record with no baseline value, while its absolute criteria are still assessed.
  • CRIT4FL is Y when CHG is above 30 and CRIT5FL is Y when CHG is above 60, each N otherwise: an increase of exactly 30 or exactly 60 reads N, and a decrease never reads Y, however large.
  • Each flag has no value exactly where its criterion has no value. AVAL and CHG are compared as they stand, without rounding.

Read the source datasets from /app/input and save the completed dataset as /app/output/adeg.csv.

prompts/adam-adeg-qtc-flags/conventions.md

Edit
16 lines

Following CDISC ADaM standards, use the provided ADEG_RAW dataset to create an ADEG dataset with one record per subject per parameter per analysis visit.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, AVISIT, AVAL, ABLFL, ONTRTFL, BASE, CHG, CRIT1, CRIT1FL, CRIT2, CRIT2FL, CRIT3, CRIT3FL, CRIT4, CRIT4FL, CRIT5, CRIT5FL

CRIT1 is "QTcF > 450 ms", CRIT2 is "QTcF > 480 ms", and CRIT3 is "QTcF > 500 ms", or has no value. CRIT4 is "QTcF increase from baseline > 30 ms" and CRIT5 is "QTcF increase from baseline > 60 ms", or has no value. CRIT1FL, CRIT2FL, CRIT3FL, CRIT4FL, and CRIT5FL are Y or N, or have no value. AVAL and CHG are compared as they stand, without rounding.

Read the source datasets from /app/input and save the completed dataset as /app/output/adeg.csv.

prompts/adam-adeg-qtc-flags/brief.md

Edit
9 lines

Following CDISC ADaM standards, use the provided ADEG_RAW dataset to create an ADEG dataset with one record per subject per parameter per analysis visit.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, AVISIT, AVAL, ABLFL, ONTRTFL, BASE, CHG, CRIT1, CRIT1FL, CRIT2, CRIT2FL, CRIT3, CRIT3FL, CRIT4, CRIT4FL, CRIT5, CRIT5FL

Read the source datasets from /app/input and save the completed dataset as /app/output/adeg.csv.

yamaa code

Public calls generated from benchmark.yaml; results must match the expected output

R

Edit
3 lines
library(yamaa)yamaa_spec("spec.yaml")$run()$output

Python

Edit
3 lines
import yamaayamaa.spec("spec.yaml").run().output

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/adam-adeg-qtc-flags/result.R

Edit
73 lines
# Reference solution for the yamaa benchmark adam-adeg-qtc-flags (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)raw <- read_csv(  "/app/input/adeg_raw.csv",  col_types = cols(    AVAL = col_double(),    BASE = col_double(),    CHG = col_double(),    .default = col_character()  ))# Only on-treatment QTcF records are assessed, and values are compared# unrounded, strictly above each limit: exactly at a limit reads N.absolute <- function(data) {  data$PARAMCD %in% "QTCF" & data$ONTRTFL %in% "Y" & !is.na(data$AVAL)}change <- function(data) {  data$PARAMCD %in% "QTCF" & data$ONTRTFL %in% "Y" & !is.na(data$CHG)}adeg <- raw |>  mutate(    CRIT1 = if_else(absolute(raw), "QTcF > 450 ms", NA_character_),    CRIT1FL = case_when(      !absolute(raw) ~ NA_character_,      AVAL > 450 ~ "Y",      .default = "N"    ),    CRIT2 = if_else(absolute(raw), "QTcF > 480 ms", NA_character_),    CRIT2FL = case_when(      !absolute(raw) ~ NA_character_,      AVAL > 480 ~ "Y",      .default = "N"    ),    CRIT3 = if_else(absolute(raw), "QTcF > 500 ms", NA_character_),    CRIT3FL = case_when(      !absolute(raw) ~ NA_character_,      AVAL > 500 ~ "Y",      .default = "N"    ),    CRIT4 = if_else(      change(raw), "QTcF increase from baseline > 30 ms", NA_character_    ),    CRIT4FL = case_when(      !change(raw) ~ NA_character_,      CHG > 30 ~ "Y",      .default = "N"    ),    CRIT5 = if_else(      change(raw), "QTcF increase from baseline > 60 ms", NA_character_    ),    CRIT5FL = case_when(      !change(raw) ~ NA_character_,      CHG > 60 ~ "Y",      .default = "N"    )  ) |>  select(    STUDYID, USUBJID, PARAMCD, PARAM, AVISIT, AVAL, ABLFL, ONTRTFL,    BASE, CHG, CRIT1, CRIT1FL, CRIT2, CRIT2FL, CRIT3, CRIT3FL, CRIT4,    CRIT4FL, CRIT5, CRIT5FL  )dir.create("/app/output", showWarnings = FALSE)write_csv(adeg, "/app/output/adeg.csv", na = "")

solutions/adam-adeg-qtc-flags/result.py

Edit
82 lines
# Reference solution for the yamaa benchmark adam-adeg-qtc-flags (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1."""Create the ADEG dataset with QTcF criterion flags from ADEG_RAW.Only records with PARAMCD "QTCF" and ONTRTFL "Y" are assessed. Theabsolute criteria compare AVAL against 450, 480 and 500 ms; theincrease criteria compare CHG against 30 and 60 ms. A flag is Y onlywhen the value is strictly above its limit, N otherwise, and eachcriterion and flag has no value when the record is not assessed orthe compared value has no value. Values are compared as they stand,without rounding."""import csvINPUT_PATH = "/app/input/adeg_raw.csv"OUTPUT_PATH = "/app/output/adeg.csv"COLUMNS = [    "STUDYID", "USUBJID", "PARAMCD", "PARAM", "AVISIT", "AVAL",    "ABLFL", "ONTRTFL", "BASE", "CHG",    "CRIT1", "CRIT1FL", "CRIT2", "CRIT2FL", "CRIT3", "CRIT3FL",    "CRIT4", "CRIT4FL", "CRIT5", "CRIT5FL",]ABSOLUTE = [    ("CRIT1", "CRIT1FL", "QTcF > 450 ms", 450.0),    ("CRIT2", "CRIT2FL", "QTcF > 480 ms", 480.0),    ("CRIT3", "CRIT3FL", "QTcF > 500 ms", 500.0),]INCREASE = [    ("CRIT4", "CRIT4FL", "QTcF increase from baseline > 30 ms", 30.0),    ("CRIT5", "CRIT5FL", "QTcF increase from baseline > 60 ms", 60.0),]def number(text):    """The numeric value of a cell, or None when it has no value."""    if text is None or text.strip() == "":        return None    return float(text)def main():    with open(INPUT_PATH, newline="") as handle:        rows = list(csv.DictReader(handle))    out_rows = []    for row in rows:        out = {column: row.get(column, "") for column in COLUMNS[:10]}        assessed = row.get("PARAMCD") == "QTCF" and row.get("ONTRTFL") == "Y"        aval = number(row.get("AVAL"))        chg = number(row.get("CHG"))        for crit, flag, label, limit in ABSOLUTE:            if assessed and aval is not None:                out[crit] = label                out[flag] = "Y" if aval > limit else "N"            else:                out[crit] = ""                out[flag] = ""        for crit, flag, label, limit in INCREASE:            if assessed and chg is not None:                out[crit] = label                out[flag] = "Y" if chg > limit else "N"            else:                out[crit] = ""                out[flag] = ""        out_rows.append(out)    with open(OUTPUT_PATH, "w", newline="") as handle:        writer = csv.DictWriter(handle, fieldnames=COLUMNS)        writer.writeheader()        writer.writerows(out_rows)if __name__ == "__main__":    main()

solutions/adam-adeg-qtc-flags/result.sas

Edit
32 lines
/* opensas reference solution for the yamaa benchmark adam-adeg-qtc-flags.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data adeg;  length STUDYID USUBJID PARAMCD PARAM AVISIT ABLFL ONTRTFL $1024;  infile "/app/input/adeg_raw.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. PARAMCD : $char1024. PARAM : $char1024. AVISIT : $char1024. AVAL ABLFL : $char1024. ONTRTFL : $char1024. BASE CHG;run;data result;  length CRIT1 CRIT2 CRIT3 CRIT4 CRIT5 $1024 CRIT1FL CRIT2FL CRIT3FL CRIT4FL CRIT5FL $1;  set adeg;  CRIT1=''; CRIT1FL=''; CRIT2=''; CRIT2FL=''; CRIT3=''; CRIT3FL='';  CRIT4=''; CRIT4FL=''; CRIT5=''; CRIT5FL='';  /* Only on-treatment QTcF records are assessed, and values are compared     unrounded, strictly above each limit: exactly at a limit reads N. */  if PARAMCD='QTCF' and ONTRTFL='Y' and not missing(AVAL) then do;    CRIT1='QTcF > 450 ms'; CRIT1FL=ifc(AVAL>450,'Y','N');    CRIT2='QTcF > 480 ms'; CRIT2FL=ifc(AVAL>480,'Y','N');    CRIT3='QTcF > 500 ms'; CRIT3FL=ifc(AVAL>500,'Y','N');  end;  if PARAMCD='QTCF' and ONTRTFL='Y' and not missing(CHG) then do;    CRIT4='QTcF increase from baseline > 30 ms'; CRIT4FL=ifc(CHG>30,'Y','N');    CRIT5='QTcF increase from baseline > 60 ms'; CRIT5FL=ifc(CHG>60,'Y','N');  end;run;proc sql;  create table final as select STUDYID, USUBJID, PARAMCD, PARAM, AVISIT, AVAL, ABLFL, ONTRTFL, BASE, CHG, CRIT1, CRIT1FL, CRIT2, CRIT2FL, CRIT3, CRIT3FL, CRIT4, CRIT4FL, CRIT5, CRIT5FL from result;quit;proc export data=final outfile="/app/output/adeg.csv" dbms=csv replace; run;