Review ask

As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.

We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.

Summary

Lifecycle: draftEditUnder review
What this benchmark means
README.md

Public entries: Python and R. Passing coverage requires an installed inventory.

Goal: give each electrocardiogram (ECG) time-point record of a QT study the plasma concentration of the parent drug measured at the same nominal time, for the concentration-QT analysis: flag the records with a matching pharmacokinetic (PK) sample (ECGPCFL), carry the concentration (PKCONC), and flag the records the concentration-QT analysis uses (CQTANLFL).

Input: the ECG time-point averages with their actual treatment (TRTA), time-matched baseline and change from baseline, one record per subject, parameter, analysis period (APERIOD), nominal day within the period (APERDAY) and nominal time point (ATPT, hours after the dose time in ATPTN); and the plasma concentrations, one record per subject, period, period day, time point and analyte (the parent drug DRGX and a metabolite), with AVAL empty and LLOQFL Y for a result below the lower limit of quantification.

Variables:

  • TRTA, AVALU, ABLFL, BASE, CHG, BASETYPE: as in the ECG input.
  • ECGPCFL: Y when the concentrations hold a parent-drug sample of the same subject, period, period day and nominal time point, whatever its result; no value otherwise. A sample at another time point, even the nearest one, and a metabolite result never match.
  • PKCONC: that sample's parent-drug concentration in ng/mL, as recorded and not rounded, on records of any treatment. Nothing is imputed: no value for a result below the limit or when no sample matches, except on placebo records (TRTA "Placebo"), which hold 0 whenever no quantified result exists, the baseline day included.
  • CQTANLFL: Y when the record has both a change from baseline and a concentration; no value otherwise.

Note: a PK sample without an ECG record at its time point adds no record, and whether a record is placebo follows its own period's treatment.

Inputs

Under review
2 source files

input/adeg_raw.csv

Edit
16 rows
input/adeg_raw.csv
STUDYIDUSUBJIDTRTAPARAMCDPARAMAPERIODAPERDAYATPTATPTNAVALAVALUABLFLBASECHGBASETYPE
TQT-01PlaceboQTCFQTcF Interval1-12 h2402.333msecY402.333—Period 1 Baseline Day 2 h
TQT-01PlaceboQTCFQTcF Interval112 h2404msec—402.3331.6669999999999732Period 1 Baseline Day 2 h
TQT-01Drug XQTCFQTcF Interval2-12 h2399.667msecY399.667—Period 2 Baseline Day 2 h
TQT-01Drug XQTCFQTcF Interval212 h2411.333msec—399.66711.666000000000054Period 2 Baseline Day 2 h
TQT-01Drug XQTCFQTcF Interval1-12 h2395msecY395—Period 1 Baseline Day 2 h
TQT-01Drug XQTCFQTcF Interval1-112 h12391.667msecY391.667—Period 1 Baseline Day 12 h
TQT-01Drug XQTCFQTcF Interval112 h2408.667msec—39513.666999999999973Period 1 Baseline Day 2 h
TQT-01Drug XQTCFQTcF Interval1112 h12397.333msec—391.6675.666000000000054Period 1 Baseline Day 12 h
TQT-01PlaceboQTCFQTcF Interval2-12 h2396.333msecY396.333—Period 2 Baseline Day 2 h
TQT-01PlaceboQTCFQTcF Interval212 h2398msec—396.3331.6669999999999732Period 2 Baseline Day 2 h
TQT-01Drug XQTCFQTcF Interval1-10 h0410.333msecY410.333—Period 1 Baseline Day 0 h
TQT-01Drug XQTCFQTcF Interval110 h0411.667msec—410.3331.3339999999999463Period 1 Baseline Day 0 h
TQT-01PlaceboQTCFQTcF Interval2-12 h2407msecY407—Period 2 Baseline Day 2 h
TQT-01PlaceboQTCFQTcF Interval212 h2409.333msec—4072.333000000000027Period 2 Baseline Day 2 h
TQT-01Drug XQTCFQTcF Interval2-10 h0412.333msecY412.333—Period 2 Baseline Day 0 h
TQT-01Drug XQTCFQTcF Interval212 h2415.667msec———Period 2 Baseline Day 2 h

input/adpc.csv

Edit
10 rows
input/adpc.csv
STUDYIDUSUBJIDTRTAAPERIODAPERDAYATPTATPTNPARAMCDPARAMAVALAVALULLOQFL
TQT-01Placebo112 h2DRGXDrug X—ng/mLY
TQT-01Drug X212 h2MET1Drug X Metabolite M136.2ng/mL—
TQT-01Drug X212 h2DRGXDrug X148ng/mL—
TQT-01Drug X112 h2DRGXDrug X171ng/mL—
TQT-01Drug X112 h2MET1Drug X Metabolite M141.5ng/mL—
TQT-01Drug X1114 h14DRGXDrug X62.4ng/mL—
TQT-01Drug X1114 h14MET1Drug X Metabolite M122.8ng/mL—
TQT-01Drug X110 h0DRGXDrug X—ng/mLY
TQT-01Placebo212 h2DRGXDrug X1.2ng/mL—
TQT-01Drug X212 h2DRGXDrug X139ng/mL—

Expected output

Under review
16 expected rows

expected/adeg.csv

Edit
16 rows
expected/adeg.csv
STUDYIDUSUBJIDTRTAPARAMCDPARAMAPERIODAPERDAYATPTATPTNAVALAVALUABLFLBASECHGBASETYPEECGPCFLPKCONCCQTANLFL
TQT-01PlaceboQTCFQTcF Interval1-12 h2402.333msecY402.333—Period 1 Baseline Day 2 h—0—
TQT-01PlaceboQTCFQTcF Interval112 h2404msec—402.3331.6669999999999732Period 1 Baseline Day 2 hY0Y
TQT-01Drug XQTCFQTcF Interval2-12 h2399.667msecY399.667—Period 2 Baseline Day 2 h———
TQT-01Drug XQTCFQTcF Interval212 h2411.333msec—399.66711.666000000000054Period 2 Baseline Day 2 hY148Y
TQT-01Drug XQTCFQTcF Interval1-12 h2395msecY395—Period 1 Baseline Day 2 h———
TQT-01Drug XQTCFQTcF Interval1-112 h12391.667msecY391.667—Period 1 Baseline Day 12 h———
TQT-01Drug XQTCFQTcF Interval112 h2408.667msec—39513.666999999999973Period 1 Baseline Day 2 hY171Y
TQT-01Drug XQTCFQTcF Interval1112 h12397.333msec—391.6675.666000000000054Period 1 Baseline Day 12 h———
TQT-01PlaceboQTCFQTcF Interval2-12 h2396.333msecY396.333—Period 2 Baseline Day 2 h—0—
TQT-01PlaceboQTCFQTcF Interval212 h2398msec—396.3331.6669999999999732Period 2 Baseline Day 2 h—0Y
TQT-01Drug XQTCFQTcF Interval1-10 h0410.333msecY410.333—Period 1 Baseline Day 0 h———
TQT-01Drug XQTCFQTcF Interval110 h0411.667msec—410.3331.3339999999999463Period 1 Baseline Day 0 hY——
TQT-01PlaceboQTCFQTcF Interval2-12 h2407msecY407—Period 2 Baseline Day 2 h—0—
TQT-01PlaceboQTCFQTcF Interval212 h2409.333msec—4072.333000000000027Period 2 Baseline Day 2 hY1.2Y
TQT-01Drug XQTCFQTcF Interval2-10 h0412.333msecY412.333—Period 2 Baseline Day 0 h———
TQT-01Drug XQTCFQTcF Interval212 h2415.667msec———Period 2 Baseline Day 2 hY139—
Shaded columns are derived— = missing value

Human review and assessment

Your review

3 questions and your comments · View benchmark discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Dataset NameVariable NameVariable LabelTypeCodelist/Controlled TermsCoreComputational MethodOrigin
ADEGSTUDYIDStudy IdentifiercharacterCopy value from ADEG_RAW.STUDYID.Predecessor
ADEGUSUBJIDUnique Subject IdentifiercharacterCopy value from ADEG_RAW.USUBJID.Predecessor
ADEGTRTAActual TreatmentcharacterCopy value from ADEG_RAW.TRTA.Predecessor
ADEGPARAMCDParameter CodecharacterCopy value from ADEG_RAW.PARAMCD.Predecessor
ADEGPARAMParametercharacterCopy value from ADEG_RAW.PARAM.Predecessor
ADEGAPERIODPeriodnumericCopy value from ADEG_RAW.APERIOD.Predecessor
ADEGAPERDAYAnalysis Nominal Period DaynumericCopy value from ADEG_RAW.APERDAY.Predecessor
ADEGATPTAnalysis Time PointcharacterCopy value from ADEG_RAW.ATPT.Predecessor
ADEGATPTNAnalysis Time Point (N)numericCopy value from ADEG_RAW.ATPTN.Predecessor
ADEGAVALAnalysis ValuenumericCopy value from ADEG_RAW.AVAL.Predecessor
ADEGAVALUAnalysis Value UnitcharacterCopy value from ADEG_RAW.AVALU.Predecessor
ADEGABLFLBaseline Record FlagcharacterCopy value from ADEG_RAW.ABLFL.Predecessor
ADEGBASEBaseline ValuenumericCopy value from ADEG_RAW.BASE.Predecessor
ADEGCHGChange from BaselinenumericCopy value from ADEG_RAW.CHG.Predecessor
ADEGBASETYPEBaseline TypecharacterCopy value from ADEG_RAW.BASETYPE.Predecessor
ADEGECGPCFLECG Matching PK FlagcharacterY"Y" when SAMPLED = 1, with SAMPLED taken from the ADPC record matching on STUDYID, USUBJID, APERIOD, APERDAY and ATPTN, where ADPC.PARAMCD = 'DRGX', where SAMPLED is derived as set constant value "1", and no match yields "0"; blank otherwise.Derived
ADEGPKCONCTime-Matched Plasma ConcentrationnumericIf SAMPLED = 1 AND AVAL IS NOT NULL, with SAMPLED taken from the ADPC record matching on STUDYID, USUBJID, APERIOD, APERDAY and ATPTN, where ADPC.PARAMCD = 'DRGX', where SAMPLED is derived as set constant value "1", and no match yields "0" and AVAL taken from the ADPC record matching on STUDYID, USUBJID, APERIOD, APERDAY and ATPTN, where ADPC.PARAMCD = 'DRGX', and no match yields "0" then "PARENT_PK.AVAL"; If TRTA = 'Placebo' then "0.0"; otherwise blank.Derived
ADEGCQTANLFLConcentration QT Analysis FlagcharacterY"Y" when CHG IS NOT NULL AND PKCONC IS NOT NULL; blank otherwise.Derived
VariablePermitted values
ECGPCFLY
CQTANLFLY
VersionDateAuthorDescriptionReviewerSign-off
1.72026-10-20yamaa docs buildRead a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default
1.62026-10-17yamaa docs buildDescribe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight
1.52026-10-17yamaa docs buildName collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part
1.42026-10-15yamaa docs buildState datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/adam-adeg-time-matched-concentration/full.md

Edit
28 lines

Following CDISC ADaM standards, use the provided ADEG_RAW and ADPC datasets to create an ADEG dataset with one record per subject, parameter, analysis period, nominal day within the period, and nominal time point.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, TRTA, PARAMCD, PARAM, APERIOD, APERDAY, ATPT, ATPTN, AVAL, AVALU, ABLFL, BASE, CHG, BASETYPE, ECGPCFL, PKCONC, CQTANLFL

ECGPCFL is Y when the concentrations hold a sample of the parent drug DRGX for the same subject, period, period day, and nominal time point; it has no value otherwise. A sample whose result is below the lower limit of quantification still matches. A sample at another time point, even the nearest one, never matches, and neither does a metabolite result.

PKCONC is the matching sample's concentration in ng/mL, as recorded and not rounded, on records of any treatment. PKCONC has no value when the matching sample's result is below the lower limit of quantification or when no sample matches. On placebo records, where TRTA is "Placebo", PKCONC is 0 whenever no quantified result exists, on the baseline day as on any other day. Whether a record is a placebo record follows its own period's treatment.

CQTANLFL is Y when the record has both a change from baseline and a concentration; it has no value otherwise. A PK sample with no ECG record at its time point adds no record.

Read the source datasets from /app/input and save the completed dataset as /app/output/adeg.csv.

prompts/adam-adeg-time-matched-concentration/conventions.md

Edit
13 lines

Following CDISC ADaM standards, use the provided ADEG_RAW and ADPC datasets to create an ADEG dataset with one record per subject, parameter, analysis period, nominal day within the period, and nominal time point.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, TRTA, PARAMCD, PARAM, APERIOD, APERDAY, ATPT, ATPTN, AVAL, AVALU, ABLFL, BASE, CHG, BASETYPE, ECGPCFL, PKCONC, CQTANLFL

ECGPCFL is Y or has no value. PKCONC is as recorded and not rounded. CQTANLFL is Y or has no value.

Read the source datasets from /app/input and save the completed dataset as /app/output/adeg.csv.

prompts/adam-adeg-time-matched-concentration/brief.md

Edit
10 lines

Following CDISC ADaM standards, use the provided ADEG_RAW and ADPC datasets to create an ADEG dataset with one record per subject, parameter, analysis period, nominal day within the period, and nominal time point.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, TRTA, PARAMCD, PARAM, APERIOD, APERDAY, ATPT, ATPTN, AVAL, AVALU, ABLFL, BASE, CHG, BASETYPE, ECGPCFL, PKCONC, CQTANLFL

Read the source datasets from /app/input and save the completed dataset as /app/output/adeg.csv.

yamaa code

Public calls generated from benchmark.yaml; results must match the expected output

R

Edit
3 lines
library(yamaa)yamaa_spec("spec.yaml")$run()$output

Python

Edit
3 lines
import yamaayamaa.spec("spec.yaml").run().output

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/adam-adeg-time-matched-concentration/result.R

Edit
55 lines
# Reference solution for the yamaa benchmark adam-adeg-time-matched-concentration (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)numbers <- cols(  APERIOD = col_integer(), APERDAY = col_integer(),  ATPTN = col_double(), AVAL = col_double(),  BASE = col_double(), CHG = col_double(),  .default = col_character())adeg <- read_csv("/app/input/adeg_raw.csv", col_types = numbers, na = "")adpc <- read_csv(  "/app/input/adpc.csv",  col_types = cols(    APERIOD = col_integer(), APERDAY = col_integer(),    ATPTN = col_double(), AVAL = col_double(),    .default = col_character()  ),  na = "")# The parent drug's sample at exactly the same subject, period, period day# and nominal time point; metabolite records are never used.parent <- adpc |>  filter(PARAMCD == "DRGX") |>  transmute(STUDYID, USUBJID, APERIOD, APERDAY, ATPTN,            SAMPLED = TRUE, PKAVAL = AVAL)adeg_out <- adeg |>  left_join(    parent,    by = c("STUDYID", "USUBJID", "APERIOD", "APERDAY", "ATPTN"),    relationship = "many-to-one"  ) |>  mutate(    # A sample whatever its result, also below the limit.    ECGPCFL = if_else(!is.na(SAMPLED), "Y", NA_character_),    # As recorded; nothing imputed, except 0 for placebo without a    # quantified result.    PKCONC = case_when(      !is.na(PKAVAL) ~ PKAVAL,      TRTA == "Placebo" ~ 0,      .default = NA_real_    ),    CQTANLFL = if_else(!is.na(CHG) & !is.na(PKCONC), "Y", NA_character_)  ) |>  select(all_of(c(names(adeg), "ECGPCFL", "PKCONC", "CQTANLFL")))dir.create("/app/output", showWarnings = FALSE)write_csv(adeg_out, "/app/output/adeg.csv", na = "")

solutions/adam-adeg-time-matched-concentration/result.py

Edit
120 lines
# Reference solution for the yamaa benchmark adam-adeg-time-matched-concentration (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1."""Build ADEG from ADEG_RAW and ADPC.Adds time-matched pharmacokinetic concentrations from the parent drugDRGX onto each ECG record. A DRGX sample matches an ECG record only atthe same subject, analysis period, period day, and nominal time point.Values are carried through exactly as recorded."""import csvINPUT_DIR = "/app/input"OUTPUT_DIR = "/app/output"def read_rows(path):    with open(path, newline="") as handle:        return list(csv.DictReader(handle))adeg_rows = read_rows(f"{INPUT_DIR}/adeg_raw.csv")adpc_rows = read_rows(f"{INPUT_DIR}/adpc.csv")# Index the parent-drug samples by subject, period, period day, and# nominal time point. Metabolite results never match.pk_samples = {}for sample in adpc_rows:    if sample["PARAMCD"] != "DRGX":        continue    key = (        sample["USUBJID"],        sample["APERIOD"],        sample["APERDAY"],        sample["ATPTN"],    )    pk_samples[key] = sampleoutput_rows = []for record in adeg_rows:    key = (        record["USUBJID"],        record["APERIOD"],        record["APERDAY"],        record["ATPTN"],    )    sample = pk_samples.get(key)    if sample is None:        flag = ""        concentration = ""    else:        # A matched sample sets the flag even when its result is below        # the lower limit of quantification, but such a result gives        # no concentration.        flag = "Y"        if sample["LLOQFL"] == "Y" or sample["AVAL"] == "":            concentration = ""        else:            concentration = sample["AVAL"]    # On placebo records a missing concentration is zero, on the    # baseline day as on any other day. The record's own treatment    # decides whether it is a placebo record.    if concentration == "" and record["TRTA"] == "Placebo":        concentration = "0"    # The concentration-QTc analysis flag needs both a change from    # baseline and a concentration.    analysis_flag = (        "Y" if record["CHG"] != "" and concentration != "" else ""    )    output_rows.append(        {            "STUDYID": record["STUDYID"],            "USUBJID": record["USUBJID"],            "TRTA": record["TRTA"],            "PARAMCD": record["PARAMCD"],            "PARAM": record["PARAM"],            "APERIOD": record["APERIOD"],            "APERDAY": record["APERDAY"],            "ATPT": record["ATPT"],            "ATPTN": record["ATPTN"],            "AVAL": record["AVAL"],            "AVALU": record["AVALU"],            "ABLFL": record["ABLFL"],            "BASE": record["BASE"],            "CHG": record["CHG"],            "BASETYPE": record["BASETYPE"],            "ECGPCFL": flag,            "PKCONC": concentration,            "CQTANLFL": analysis_flag,        }    )columns = [    "STUDYID",    "USUBJID",    "TRTA",    "PARAMCD",    "PARAM",    "APERIOD",    "APERDAY",    "ATPT",    "ATPTN",    "AVAL",    "AVALU",    "ABLFL",    "BASE",    "CHG",    "BASETYPE",    "ECGPCFL",    "PKCONC",    "CQTANLFL",]with open(f"{OUTPUT_DIR}/adeg.csv", "w", newline="") as handle:    writer = csv.DictWriter(handle, fieldnames=columns)    writer.writeheader()    writer.writerows(output_rows)

solutions/adam-adeg-time-matched-concentration/result.sas

Edit
60 lines
/* opensas reference solution for the yamaa benchmark adam-adeg-time-matched-concentration.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data adeg;  length STUDYID USUBJID TRTA PARAMCD PARAM ATPT AVALU ABLFL BASETYPE $1024;  infile "/app/input/adeg_raw.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. TRTA : $char1024. PARAMCD : $char1024. PARAM : $char1024. APERIOD APERDAY ATPT : $char1024. ATPTN AVAL AVALU : $char1024. ABLFL : $char1024. BASE CHG BASETYPE : $char1024.;  RECNO = _n_;run;data adpc;  length STUDYID USUBJID TRTA ATPT PARAMCD PARAM AVALU LLOQFL $1024;  infile "/app/input/adpc.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. TRTA : $char1024. APERIOD APERDAY ATPT : $char1024. ATPTN PARAMCD : $char1024. PARAM : $char1024. AVAL AVALU : $char1024. LLOQFL : $char1024.;run;/* The parent drug's samples only; metabolite records are never used. */data parent;  length P_STUDYID P_USUBJID $1024;  set adpc;  if PARAMCD = 'DRGX';  P_STUDYID = STUDYID; P_USUBJID = USUBJID; P_APERIOD = APERIOD;  P_APERDAY = APERDAY; P_ATPTN = ATPTN; PKAVAL = AVAL; SAMPLED = 1;  keep P_STUDYID P_USUBJID P_APERIOD P_APERDAY P_ATPTN PKAVAL SAMPLED;run;/* Exact match on subject, period, period day and nominal time point. */proc sql;  create table matched as    select a.*, p.PKAVAL, p.SAMPLED    from adeg as a left join parent as p      on a.STUDYID = p.P_STUDYID and a.USUBJID = p.P_USUBJID      and a.APERIOD = p.P_APERIOD and a.APERDAY = p.P_APERDAY      and a.ATPTN = p.P_ATPTN    order by RECNO;quit;/* As recorded; nothing imputed, except 0 for placebo without a quantified   result. */data final;  length ECGPCFL CQTANLFL $1;  set matched;  if SAMPLED = 1 then ECGPCFL = 'Y';  else ECGPCFL = '';  if not missing(PKAVAL) then PKCONC = PKAVAL;  else if TRTA = 'Placebo' then PKCONC = 0;  else PKCONC = .;  if not missing(CHG) and not missing(PKCONC) then CQTANLFL = 'Y';  else CQTANLFL = '';run;proc sql;  create table out as    select STUDYID, USUBJID, TRTA, PARAMCD, PARAM, APERIOD, APERDAY, ATPT,           ATPTN, AVAL, AVALU, ABLFL, BASE, CHG, BASETYPE, ECGPCFL, PKCONC,           CQTANLFL    from final;quit;proc export data=out outfile="/app/output/adeg.csv" dbms=csv replace; run;