Review ask

As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.

We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.

Summary

Lifecycle: draftEditUnder review
What this benchmark means
README.md

Public entries: Python and R. Passing coverage requires an installed inventory.

Goal: flag each subject's baseline pain score (ABLFL), mark a baseline taken from the screening questionnaire instead of the protocol's baseline visit (SCBLFL), and carry the baseline score (BASE) and the change from it (CHG), alongside each subject's treatment start date (TRTSDT).

Input: collected questionnaire records of a single-item worst-pain rating from 0 to 10, at screening, Cycle 1 Day 1 and Cycle 2 Day 1, each with its visit number (VISITNUM), completion date and numeric result, which is empty when the questionnaire was not done; plus subject-level treatment start dates.

Variables:

  • ADT is the questionnaire record's collection date, kept when the questionnaire was not done; empty when the visit was not attended.
  • TRTSDT is the subject's treatment start date.
  • AVAL is the collected rating; empty when the questionnaire was not done.
  • ABLFL is Y on the Cycle 1 Day 1 record when it has a rating and was completed on or before the treatment start date. When it has no rating or was completed after the first dose, ABLFL is Y on the screening record instead, if that has a rating. Blank on every other record, and on every record of a subject with neither.
  • SCBLFL (Screening Used as Baseline Flag) is Y on the baseline record when it is the screening record; blank on every other record, including a Cycle 1 Day 1 baseline.
  • BASE is the baseline rating, repeated on every record of the subject and parameter; empty when the subject has no baseline.
  • CHG is AVAL minus BASE on each record completed after the treatment start date; empty on the baseline record and every other record on or before that date, and wherever AVAL or BASE is empty.

Note: the protocol schedules the baseline questionnaire on Cycle 1 Day 1, before the first dose, and only it or the screening questionnaire can be the baseline, so a later rating never stands in. The screening record itself carries both flags; no separate baseline record is added.

Inputs

Under review
2 source files

input/adsl.csv

Edit
4 rows
input/adsl.csv
STUDYIDUSUBJIDTRTSDT
YAMAA-012025-02-03
YAMAA-012025-02-10
YAMAA-012025-02-17
YAMAA-012025-02-24

input/qs.csv

Edit
12 rows
input/qs.csv
STUDYIDUSUBJIDQSSEQQSCATQSTESTCDQSTESTQSSTRESNQSSTATVISITNUMVISITQSDTC
YAMAA-011PAIN NUMERIC RATING SCALEPAIN01Worst Pain in the Last 24 Hours6—1SCREENING2025-01-20
YAMAA-012PAIN NUMERIC RATING SCALEPAIN01Worst Pain in the Last 24 Hours5—2CYCLE 1 DAY 12025-02-03
YAMAA-013PAIN NUMERIC RATING SCALEPAIN01Worst Pain in the Last 24 Hours3—3CYCLE 2 DAY 12025-02-24
YAMAA-011PAIN NUMERIC RATING SCALEPAIN01Worst Pain in the Last 24 Hours7—1SCREENING2025-01-27
YAMAA-012PAIN NUMERIC RATING SCALEPAIN01Worst Pain in the Last 24 Hours—NOT DONE2CYCLE 1 DAY 12025-02-10
YAMAA-013PAIN NUMERIC RATING SCALEPAIN01Worst Pain in the Last 24 Hours4—3CYCLE 2 DAY 12025-03-03
YAMAA-011PAIN NUMERIC RATING SCALEPAIN01Worst Pain in the Last 24 Hours—NOT DONE1SCREENING2025-02-03
YAMAA-012PAIN NUMERIC RATING SCALEPAIN01Worst Pain in the Last 24 Hours—NOT DONE2CYCLE 1 DAY 1—
YAMAA-013PAIN NUMERIC RATING SCALEPAIN01Worst Pain in the Last 24 Hours5—3CYCLE 2 DAY 12025-03-10
YAMAA-011PAIN NUMERIC RATING SCALEPAIN01Worst Pain in the Last 24 Hours4—1SCREENING2025-02-10
YAMAA-012PAIN NUMERIC RATING SCALEPAIN01Worst Pain in the Last 24 Hours6—2CYCLE 1 DAY 12025-02-25
YAMAA-013PAIN NUMERIC RATING SCALEPAIN01Worst Pain in the Last 24 Hours5—3CYCLE 2 DAY 12025-03-17

Expected output

Under review
12 expected rows

expected/adqs.csv

Edit
12 rows
expected/adqs.csv
STUDYIDUSUBJIDPARAMCDPARAMVISITNUMVISITADTTRTSDTAVALABLFLSCBLFLBASECHG
YAMAA-01PAIN01Worst Pain in the Last 24 Hours1SCREENING2025-01-202025-02-036——5—
YAMAA-01PAIN01Worst Pain in the Last 24 Hours2CYCLE 1 DAY 12025-02-032025-02-035Y—5—
YAMAA-01PAIN01Worst Pain in the Last 24 Hours3CYCLE 2 DAY 12025-02-242025-02-033——5-2
YAMAA-01PAIN01Worst Pain in the Last 24 Hours1SCREENING2025-01-272025-02-107YY7—
YAMAA-01PAIN01Worst Pain in the Last 24 Hours2CYCLE 1 DAY 12025-02-102025-02-10———7—
YAMAA-01PAIN01Worst Pain in the Last 24 Hours3CYCLE 2 DAY 12025-03-032025-02-104——7-3
YAMAA-01PAIN01Worst Pain in the Last 24 Hours1SCREENING2025-02-032025-02-17—————
YAMAA-01PAIN01Worst Pain in the Last 24 Hours2CYCLE 1 DAY 1—2025-02-17—————
YAMAA-01PAIN01Worst Pain in the Last 24 Hours3CYCLE 2 DAY 12025-03-102025-02-175————
YAMAA-01PAIN01Worst Pain in the Last 24 Hours1SCREENING2025-02-102025-02-244YY4—
YAMAA-01PAIN01Worst Pain in the Last 24 Hours2CYCLE 1 DAY 12025-02-252025-02-246——42
YAMAA-01PAIN01Worst Pain in the Last 24 Hours3CYCLE 2 DAY 12025-03-172025-02-245——41
Shaded columns are derived— = missing value

Human review and assessment

Your review

3 questions and your comments · View benchmark discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Dataset NameVariable NameVariable LabelTypeCodelist/Controlled TermsCoreComputational MethodOrigin
ADQSSTUDYIDStudy IdentifiercharacterCopy value from QS.STUDYID.Predecessor
ADQSUSUBJIDUnique Subject IdentifiercharacterCopy value from QS.USUBJID.Predecessor
ADQSPARAMCDParameter CodecharacterCopy value from QS.QSTESTCD.Predecessor
ADQSPARAMParametercharacterCopy value from QS.QSTEST.Predecessor
ADQSVISITNUMVisit NumbernumericCopy value from QS.VISITNUM.Predecessor
ADQSVISITVisit NamecharacterCopy value from QS.VISIT.Predecessor
ADQSADTAnalysis DatenumericISO 8601Copy value from QS.QSDTC.Predecessor
ADQSTRTSDTDate of First Exposure to TreatmentnumericISO 8601Copy value from ADSL.TRTSDT.Predecessor
ADQSAVALAnalysis ValuenumericCopy value from QS.QSSTRESN.Predecessor
ADQSABLFLBaseline Record Flagcharacter"Y" for the last record with ADT on or before TRTSDT and a non-missing AVAL within each group of STUDYID, USUBJID, PARAMCD, ordered by ADT; blank otherwise.Derived
ADQSSCBLFLScreening Used as Baseline Flagcharacter"Y" when ABLFL = 'Y' AND VISIT = 'SCREENING'; blank otherwise.Derived
ADQSBASEBaseline ValuenumericAggregate ONLY(AVAL) over records grouped by STUDYID, USUBJID, PARAMCD, matching on the shared output keys, where ABLFL = 'Y'.Derived
ADQSCHGChange from BaselinenumericIf ADT > TRTSDT then compute AVAL - BASE; otherwise "None".Derived
VersionDateAuthorDescriptionReviewerSign-off
1.72026-10-20yamaa docs buildRead a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default
1.62026-10-17yamaa docs buildDescribe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight
1.52026-10-17yamaa docs buildName collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part
1.42026-10-15yamaa docs buildState datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/adam-adqs-screening-baseline/full.md

Edit
33 lines

Following CDISC ADaM standards, use the provided ADSL and QS datasets to create an ADQS dataset with one record per subject per parameter per visit.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, VISITNUM, VISIT, ADT, TRTSDT, AVAL, ABLFL, SCBLFL, BASE, CHG

PARAMCD and PARAM are the collected QSTESTCD and QSTEST, unchanged. ADT is the date in QSDTC, also on a record whose questionnaire was not done; it has no value where QSDTC is empty, as when the visit was not attended. TRTSDT is the subject's treatment start date. AVAL is the collected rating, QSSTRESN, and has no value when the questionnaire was not done.

  • ABLFL is "Y" on the "CYCLE 1 DAY 1" record when it has a rating and its ADT is on or before TRTSDT. When that record has no rating or is dated after TRTSDT, ABLFL is "Y" on the "SCREENING" record instead, if that has a rating. When neither applies, the subject has no baseline. ABLFL has no value on every other record.
  • Only those two visits can be the baseline; a "CYCLE 2 DAY 1" rating never stands in.
  • SCBLFL (Screening Used as Baseline Flag) is "Y" on the baseline record when it is the "SCREENING" record, and has no value on every other record, a "CYCLE 1 DAY 1" baseline included. The screening record itself carries both flags; no separate baseline record is added.
  • BASE is the baseline record's AVAL, on every record of the subject and parameter, and has no value for a subject without a baseline.
  • CHG is AVAL minus BASE on each record whose ADT is after TRTSDT, a "CYCLE 1 DAY 1" record included. CHG has no value on the baseline record and every other record dated on or before TRTSDT, and wherever AVAL or BASE has no value.

Read the source datasets from /app/input and save the completed dataset as /app/output/adqs.csv.

prompts/adam-adqs-screening-baseline/conventions.md

Edit
13 lines

Following CDISC ADaM standards, use the provided ADSL and QS datasets to create an ADQS dataset with one record per subject per parameter per visit.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, VISITNUM, VISIT, ADT, TRTSDT, AVAL, ABLFL, SCBLFL, BASE, CHG

PARAMCD and PARAM are the collected QSTESTCD and QSTEST, unchanged. ABLFL is "Y" or has no value. SCBLFL (Screening Used as Baseline Flag) is "Y" or has no value.

Read the source datasets from /app/input and save the completed dataset as /app/output/adqs.csv.

prompts/adam-adqs-screening-baseline/brief.md

Edit
9 lines

Following CDISC ADaM standards, use the provided ADSL and QS datasets to create an ADQS dataset with one record per subject per parameter per visit.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, VISITNUM, VISIT, ADT, TRTSDT, AVAL, ABLFL, SCBLFL, BASE, CHG

Read the source datasets from /app/input and save the completed dataset as /app/output/adqs.csv.

yamaa code

Public calls generated from benchmark.yaml; results must match the expected output

R

Edit
3 lines
library(yamaa)yamaa_spec("spec.yaml")$run()$output

Python

Edit
3 lines
import yamaayamaa.spec("spec.yaml").run().output

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/adam-adqs-screening-baseline/result.R

Edit
87 lines
# Reference solution for the yamaa benchmark adam-adqs-screening-baseline (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.suppressPackageStartupMessages({  library(dplyr)  library(readr)})adsl <- read_csv(  "/app/input/adsl.csv",  col_types = cols(.default = col_character()),  na = character(),  progress = FALSE) |>  mutate(TRTSDT = as.Date(na_if(TRTSDT, ""))) |>  select(STUDYID, USUBJID, TRTSDT)qs <- read_csv(  "/app/input/qs.csv",  col_types = cols(.default = col_character()),  na = character(),  progress = FALSE) |>  mutate(    QSSTRESN = as.numeric(na_if(QSSTRESN, "")),    VISITNUM = as.numeric(VISITNUM),    QSDTC = na_if(QSDTC, "")  )# One record per subject, parameter and visit, with the subject's treatment# start date. The rating is the collected result, and it is missing when the# questionnaire was not done. ADT is the collected date, not-done records# included.adqs <- qs |>  left_join(adsl, by = c("STUDYID", "USUBJID")) |>  transmute(    STUDYID,    USUBJID,    PARAMCD = QSTESTCD,    PARAM = QSTEST,    VISITNUM,    VISIT,    ADT = as.Date(substr(QSDTC, 1, 10)),    TRTSDT,    AVAL = QSSTRESN  )# A record can be the baseline only when it has a rating. Cycle 1 Day 1 counts# when dated on or before the treatment start date; otherwise screening counts.has_rating <- !is.na(adqs$AVAL)c1d1_ok <- adqs$VISIT == "CYCLE 1 DAY 1" & has_rating &  !is.na(adqs$ADT) & !is.na(adqs$TRTSDT) & adqs$ADT <= adqs$TRTSDTscr_ok <- adqs$VISIT == "SCREENING" & has_ratingadqs <- adqs |>  mutate(.c1d1_ok = c1d1_ok, .scr_ok = scr_ok) |>  group_by(USUBJID, PARAMCD) |>  mutate(    .use_c1d1 = any(.c1d1_ok),    ABLFL = case_when(      .use_c1d1 & .c1d1_ok ~ "Y",      !.use_c1d1 & .scr_ok ~ "Y",      TRUE ~ NA_character_    ),    SCBLFL = if_else(!is.na(ABLFL) & VISIT == "SCREENING", "Y", NA_character_),    BASE = if (any(!is.na(ABLFL))) AVAL[which(!is.na(ABLFL))[1]] else NA_real_  ) |>  ungroup() |>  mutate(    CHG = if_else(      !is.na(ADT) & !is.na(TRTSDT) & ADT > TRTSDT &        !is.na(AVAL) & !is.na(BASE),      AVAL - BASE,      NA_real_    )  ) |>  arrange(USUBJID, PARAMCD, VISITNUM) |>  select(    STUDYID, USUBJID, PARAMCD, PARAM, VISITNUM, VISIT, ADT, TRTSDT,    AVAL, ABLFL, SCBLFL, BASE, CHG  )dir.create("/app/output", showWarnings = FALSE, recursive = TRUE)write_csv(adqs, "/app/output/adqs.csv", na = "")

solutions/adam-adqs-screening-baseline/result.py

Edit
103 lines
# Reference solution for the yamaa benchmark adam-adqs-screening-baseline (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.from pathlib import Pathimport polars as plINPUT = Path("/app/input")OUTPUT = Path("/app/output")KEYS = ["STUDYID", "USUBJID", "PARAMCD"]COLUMNS = [    "STUDYID", "USUBJID", "PARAMCD", "PARAM", "VISITNUM", "VISIT",    "ADT", "TRTSDT", "AVAL", "ABLFL", "SCBLFL", "BASE", "CHG",]adsl = pl.read_csv(INPUT / "adsl.csv", infer_schema_length=0).select(    "STUDYID",    "USUBJID",    pl.col("TRTSDT").str.strip_chars().replace("", None).str.to_date("%Y-%m-%d", strict=False),)qs = pl.read_csv(INPUT / "qs.csv", infer_schema_length=0)adqs = (    qs.select(        "STUDYID",        "USUBJID",        pl.col("QSTESTCD").alias("PARAMCD"),        pl.col("QSTEST").alias("PARAM"),        pl.col("VISITNUM").cast(pl.Int64),        "VISIT",        # ISO 8601 date part of QSDTC; empty when the visit date is not collected.        pl.col("QSDTC").str.strip_chars().replace("", None)        .str.slice(0, 10).str.to_date("%Y-%m-%d", strict=False).alias("ADT"),        pl.col("QSSTRESN").str.strip_chars().replace("", None)        .cast(pl.Float64).alias("AVAL"),    )    .join(adsl, on=["STUDYID", "USUBJID"], how="left"))has_rating = pl.col("AVAL").is_not_null()on_or_before_trt = (    pl.col("ADT").is_not_null()    & pl.col("TRTSDT").is_not_null()    & (pl.col("ADT") <= pl.col("TRTSDT")))c1d1_ok = (pl.col("VISIT") == "CYCLE 1 DAY 1") & has_rating & on_or_before_trtscr_ok = (pl.col("VISIT") == "SCREENING") & has_ratingadqs = adqs.with_columns(    c1d1_ok.any().over(KEYS).alias("_C1D1_BASE"),)adqs = adqs.with_columns(    pl.when(c1d1_ok | (~pl.col("_C1D1_BASE") & scr_ok))    .then(pl.lit("Y"))    .otherwise(pl.lit(None, dtype=pl.String))    .alias("ABLFL"),)adqs = adqs.with_columns(    pl.when((pl.col("ABLFL") == "Y") & (pl.col("VISIT") == "SCREENING"))    .then(pl.lit("Y"))    .otherwise(pl.lit(None, dtype=pl.String))    .alias("SCBLFL"),    pl.col("AVAL").filter(pl.col("ABLFL") == "Y").first().over(KEYS).alias("BASE"),)adqs = adqs.with_columns(    pl.when(        pl.col("ADT").is_not_null()        & pl.col("TRTSDT").is_not_null()        & (pl.col("ADT") > pl.col("TRTSDT"))        & pl.col("ABLFL").is_null()    )    .then(pl.col("AVAL") - pl.col("BASE"))    .otherwise(None)    .alias("CHG"),)def as_text(name: str) -> pl.Expr:    """Render a numeric value without a trailing '.0' when it is whole."""    col = pl.col(name)    return (        pl.when(col.is_null())        .then(None)        .when(col == col.round(0))        .then(col.round(0).cast(pl.Int64).cast(pl.String))        .otherwise(col.cast(pl.String))        .alias(name)    )adqs = (    adqs.sort(["STUDYID", "USUBJID", "PARAMCD", "VISITNUM"])    .with_columns(as_text("AVAL"), as_text("BASE"), as_text("CHG"))    .select(COLUMNS))OUTPUT.mkdir(parents=True, exist_ok=True)adqs.write_csv(OUTPUT / "adqs.csv", date_format="%Y-%m-%d", null_value="")

solutions/adam-adqs-screening-baseline/result.sas

Edit
106 lines
/* opensas reference solution for the yamaa benchmark adam-adqs-screening-baseline.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data adsl;  length STUDYID USUBJID TRTSDT $1024;  infile "/app/input/adsl.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. TRTSDT : $char1024.;run;/* ISO dates are kept as text, so they compare in date order. A not-done   questionnaire has no rating. */data qs;  length STUDYID USUBJID QSCAT PARAMCD PARAM QSSTAT VISIT ADT $1024;  infile "/app/input/qs.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. QSSEQ QSCAT : $char1024.        PARAMCD : $char1024. PARAM : $char1024. AVAL QSSTAT : $char1024.        VISITNUM VISIT : $char1024. ADT : $char1024.;  if upcase(QSSTAT) = "NOT DONE" then AVAL = .;run;/* Treatment start date from ADSL on every QS record. */proc sql;  create table qs2 as    select a.STUDYID, a.USUBJID, a.PARAMCD, a.PARAM, a.VISITNUM, a.VISIT,           a.ADT, b.TRTSDT, a.AVAL    from qs as a left join adsl as b    on a.STUDYID = b.STUDYID and a.USUBJID = b.USUBJID;quit;proc sort data=qs2;  by STUDYID USUBJID PARAMCD VISITNUM;run;/* One row per subject and parameter that has a baseline: the CYCLE 1   DAY 1 rating dated on or before TRTSDT, otherwise the SCREENING rating. */data basesel;  set qs2;  by STUDYID USUBJID PARAMCD;  length BLVISIT $20;  retain c1d1ok scrok c1d1val scrval;  if first.PARAMCD then do;    c1d1ok = 0;    scrok = 0;    c1d1val = .;    scrval = .;  end;  if VISIT = "CYCLE 1 DAY 1" and not missing(AVAL) and not missing(ADT)     and not missing(TRTSDT) and ADT <= TRTSDT then do;    c1d1ok = 1;    c1d1val = AVAL;  end;  if VISIT = "SCREENING" and not missing(AVAL) then do;    scrok = 1;    scrval = AVAL;  end;  if last.PARAMCD then do;    if c1d1ok = 1 then do;      BLVISIT = "CYCLE 1 DAY 1";      BASE = c1d1val;      output;    end;    else if scrok = 1 then do;      BLVISIT = "SCREENING";      BASE = scrval;      output;    end;  end;  keep STUDYID USUBJID PARAMCD BLVISIT BASE;run;proc sql;  create table combined as    select a.*, b.BLVISIT, b.BASE    from qs2 as a left join basesel as b    on a.STUDYID = b.STUDYID and a.USUBJID = b.USUBJID       and a.PARAMCD = b.PARAMCD;quit;/* Flags, BASE on every record of the subject and parameter, and CHG on   the non-baseline records dated after TRTSDT. */data adqs1;  set combined;  length ABLFL SCBLFL $1;  ABLFL = "";  SCBLFL = "";  if not missing(BLVISIT) and VISIT = BLVISIT then do;    ABLFL = "Y";    if VISIT = "SCREENING" then SCBLFL = "Y";  end;  CHG = .;  if ABLFL ne "Y" and not missing(AVAL) and not missing(BASE)     and not missing(ADT) and not missing(TRTSDT) and ADT > TRTSDT then    CHG = AVAL - BASE;run;proc sql;  create table adqs as    select STUDYID, USUBJID, PARAMCD, PARAM, VISITNUM, VISIT, ADT, TRTSDT,           AVAL, ABLFL, SCBLFL, BASE, CHG    from adqs1    order by STUDYID, USUBJID, PARAMCD, VISITNUM;quit;proc export data=adqs outfile="/app/output/adqs.csv" dbms=csv replace;run;