Review ask

As a clinical trial statistician or programmer, challenge anything in this benchmark: describe a problem in Comments, or propose a fix with the Edit button on any source file.

  1. Task: Would a real study need this derivation, specified this way? Does it follow CDISC ADaM: variable names, controlled terms, record structure? Flag any rule that is ambiguous or that a sponsor would handle differently.
  2. Inputs: Do they look like data collected in a real trial, and do they include the situations you would expect to meet in one?
  3. Expected output: Would independent QC programming reproduce every value from these inputs and rules?

The other tabs beside Comments are background for assessment, not under review.

Summary

Lifecycle: draftEditUnder review
What this benchmark means
README.md

Goal: prepare each collected overall response assessment for best overall response (BOR) selection, adding BORCAT, BORPRI, and BORSEQ.

Input: overall response assessments carrying the analysis date (ADT), the assessment day relative to randomization (RANDDY), and the collected overall response (AVALC), plus each subject's start date of new anti-cancer therapy, when one was started.

Variables:

  • BORCAT is the response category the record can support in the best overall response decision: complete response as CR, partial response as PR, stable disease as SD, neither complete response nor progressive disease as NON-CR/NON-PD, progressive disease as PD, or not evaluable as NE. Stable disease and neither-complete-nor-progressive disease count only on or after day 42 after randomization; earlier ones, or ones with no day, fall back to not evaluable. An assessment on or after the start of new anti-cancer therapy supports no category. Any other collected value, including a missing one, supports no category, so BORCAT stays empty.
  • BORPRI orders the supported categories as complete response (1), partial response (2), stable disease (3), neither-complete-nor-progressive disease (4), progressive disease (5), not evaluable (6); empty when the record supports no category.
  • BORSEQ numbers the usable records of each study and subject in category order, then by analysis date, then by assessment sequence. The record numbered 1 supplies the study-subject's best overall response and its supporting date.

Note: a record that supports no category takes no priority and no number, so numbering passes over it to the next usable record: it can never be numbered 1, and it never consumes a number a usable record would otherwise take.

Inputs

Under review
2 source files

input/adrs_raw.csv

Edit
21 rows
input/adrs_raw.csv
STUDYIDUSUBJIDASEQADTRANDDYAVALC
PILOT712025-02-1742PR
PILOT722025-04-0791CR
PILOT712025-02-1028SD
PILOT722025-04-1491SD
PILOT732025-05-12119SD
PILOT712025-02-1021SD
PILOT722025-03-0342PD
PILOT712025-02-2428SD
PILOT722025-04-0770NE
PILOT712025-03-3149NON-CR/NON-PD
PILOT712025-02-0321ND
PILOT722025-03-1763—
PILOT732025-04-28105PR
PILOT712025-03-01—SD
PILOT722025-04-1470CR
PILOT712025-04-0150SD
PILOT722025-04-0155SD
PILOT712025-02-1028PR
PILOT722025-04-0791CR
PILOT812025-03-1056SD
PILOT822025-04-2198PD

input/adsl.csv

Edit
10 rows
input/adsl.csv
STUDYIDUSUBJIDNTXSTDT
PILOT7—
PILOT7—
PILOT72025-03-03
PILOT7—
PILOT72025-05-01
PILOT7—
PILOT7—
PILOT7—
PILOT72025-03-01
PILOT8—

Expected output

Under review
21 expected rows

expected/adrs.csv

Edit
21 rows
expected/adrs.csv
STUDYIDUSUBJIDASEQADTRANDDYAVALCBORCATBORPRIBORSEQ
PILOT712025-02-1742PRPR22
PILOT722025-04-0791CRCR11
PILOT712025-02-1028SDNE63
PILOT722025-04-1491SDSD31
PILOT732025-05-12119SDSD32
PILOT712025-02-1021SDNE61
PILOT722025-03-0342PD———
PILOT712025-02-2428SDNE61
PILOT722025-04-0770NENE62
PILOT712025-03-3149NON-CR/NON-PDNON-CR/NON-PD41
PILOT712025-02-0321ND———
PILOT722025-03-1763————
PILOT732025-04-28105PRPR21
PILOT712025-03-01—SDNE62
PILOT722025-04-1470CRCR11
PILOT712025-04-0150SDSD31
PILOT722025-04-0155SDSD32
PILOT712025-02-1028PRPR21
PILOT722025-04-0791CR———
PILOT812025-03-1056SDSD31
PILOT822025-04-2198PDPD52
Shaded columns are derived— = missing value

Comments and assessment materials

Comments

Signed in with GitHub; saved in repository Discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Dataset NameVariable NameVariable LabelTypeCodelist/Controlled TermsCoreComputational MethodOrigin
ADRSSTUDYIDStudy IdentifiercharacterCopy value from ADRS_RAW.STUDYID.Predecessor
ADRSUSUBJIDUnique Subject IdentifiercharacterCopy value from ADRS_RAW.USUBJID.Predecessor
ADRSASEQSequence NumbernumericCopy value from ADRS_RAW.ASEQ.Predecessor
ADRSADTAnalysis DatenumericISO 8601Copy value from ADRS_RAW.ADT.Predecessor
ADRSRANDDYAnalysis Day Relative to RandomizationnumericCopy value from ADRS_RAW.RANDDY.Predecessor
ADRSAVALCAnalysis Value (C)characterCopy value from ADRS_RAW.AVALC.Predecessor
ADRSBORCATBest Overall Response CategorycharacterCR, PR, SD, NON-CR/NON-PD, PD, NEIf NTXSTDT IS NOT NULL AND ADT >= NTXSTDT then "None"; If AVALC = 'CR' then "CR"; If AVALC = 'PR' then "PR"; If AVALC = 'SD' AND RANDDY >= 42 then "SD"; If AVALC = 'NON-CR/NON-PD' AND RANDDY >= 42 then "NON-CR/NON-PD"; If AVALC = 'PD' then "PD"; If AVALC IN ('SD', 'NON-CR/NON-PD', 'NE') then "NE"; otherwise "None".Derived
ADRSBORPRIBest Overall Response PrioritynumericRecode BORCAT: "CR" → "1", "PR" → "2", "SD" → "3", "NON-CR/NON-PD" → "4", "PD" → "5", "NE" → "6"; missing values stay missing; unlisted values are errors.Derived
ADRSBORSEQBest Overall Response SequencenumericRow number within each group of STUDYID, USUBJID, ordered by BORPRI then ADT then ASEQ.Derived
VariablePermitted values
BORCATCR, PR, SD, NON-CR/NON-PD, PD, NE
VersionDateAuthorDescriptionReviewerSign-off
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/adam-adrs-response-prep/full.md

Edit
32 lines

Following CDISC ADaM standards, use the provided ADRS_RAW and ADSL datasets to create an ADRS dataset with one record per collected overall response assessment.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, ASEQ, ADT, RANDDY, AVALC, BORCAT, BORPRI, BORSEQ

BORCAT is the response category the record can support: complete response as CR, partial response as PR, stable disease as SD, neither complete response nor progressive disease as NON-CR/NON-PD, progressive disease as PD, or not evaluable as NE. An assessment on or after the subject's start of new anti-cancer therapy (NTXSTDT from the ADSL dataset) supports no category; a subject with no therapy date has no excluded assessments. Stable disease and neither-complete-nor-progressive disease count only on or after day 42 after randomization; earlier ones, or ones with no day, fall back to not evaluable. Any other collected value, including a missing one, supports no category, so BORCAT has no value.

BORPRI orders the supported categories as 1 (complete response), 2 (partial response), 3 (stable disease), 4 (neither-complete-nor-progressive disease), 5 (progressive disease), 6 (not evaluable); it has no value when the record supports no category.

BORSEQ numbers the usable records of each study and subject from 1, in BORPRI order, then by analysis date, then by assessment sequence. The record numbered 1 supplies the study-subject's best overall response and its supporting date. A record that supports no category has no BORSEQ, and the numbering passes over it without a gap.

Read the source datasets from /app/input and save the completed dataset as /app/output/adrs.csv.

prompts/adam-adrs-response-prep/conventions.md

Edit
19 lines

Following CDISC ADaM standards, use the provided ADRS_RAW and ADSL datasets to create an ADRS dataset with one record per collected overall response assessment.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, ASEQ, ADT, RANDDY, AVALC, BORCAT, BORPRI, BORSEQ

BORCAT is the response category: complete response as CR, partial response as PR, stable disease as SD, neither complete response nor progressive disease as NON-CR/NON-PD, progressive disease as PD, or not evaluable as NE. An assessment on or after the subject's new anti-cancer therapy start (NTXSTDT in ADSL) supports no category.

BORPRI orders the categories as 1 (complete response), 2 (partial response), 3 (stable disease), 4 (neither-complete-nor-progressive disease), 5 (progressive disease), 6 (not evaluable).

Read the source datasets from /app/input and save the completed dataset as /app/output/adrs.csv.

prompts/adam-adrs-response-prep/brief.md

Edit
9 lines

Following CDISC ADaM standards, use the provided ADRS_RAW and ADSL datasets to create an ADRS dataset with one record per collected overall response assessment.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, ASEQ, ADT, RANDDY, AVALC, BORCAT, BORPRI, BORSEQ

Read the source datasets from /app/input and save the completed dataset as /app/output/adrs.csv.

yamaa code

Runs the yamaa specification; its result must match the expected output

run.R

Edit
4 lines
library(yamaa)adrs <- yamaa_domain("spec.yaml")$outputadrs

run.py

Edit
4 lines
import yamaaadrs = yamaa.yamaa_domain("spec.yaml").outputadrs

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/adam-adrs-response-prep/result.R

Edit
72 lines
# Reference solution for the yamaa benchmark adam-adrs-response-prep (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)raw <- read_csv(  "/app/input/adrs_raw.csv",  col_types = cols(    ASEQ = col_integer(),    ADT = col_date(),    RANDDY = col_integer(),    .default = col_character()  ))therapy <- read_csv(  "/app/input/adsl.csv",  col_types = cols(    NTXSTDT = col_date(),    .default = col_character()  ))raw <- raw |>  left_join(therapy, by = c("STUDYID", "USUBJID"))# The response category each record can support; an assessment on or after# the subject's new anti-cancer therapy start supports no category. Stable# and neither-complete-nor-progressive disease count only on or after day# 42, and earlier ones, or ones with no day, fall back to not evaluable.based <- raw |>  mutate(    BORCAT = case_when(      !is.na(NTXSTDT) & ADT >= NTXSTDT ~ NA_character_,      AVALC %in% c("CR", "PR", "PD", "NE") ~ AVALC,      AVALC %in% "SD" & !is.na(RANDDY) & RANDDY >= 42L ~ "SD",      AVALC %in% "SD" ~ "NE",      AVALC %in% "NON-CR/NON-PD" & !is.na(RANDDY) & RANDDY >= 42L ~ "NON-CR/NON-PD",      AVALC %in% "NON-CR/NON-PD" ~ "NE",      .default = NA_character_    ),    BORPRI = case_when(      BORCAT == "CR" ~ 1L,      BORCAT == "PR" ~ 2L,      BORCAT == "SD" ~ 3L,      BORCAT == "NON-CR/NON-PD" ~ 4L,      BORCAT == "PD" ~ 5L,      BORCAT == "NE" ~ 6L    )  )# Usable records numbered from 1 in priority, date, then sequence order;# records supporting no category are passed over without a gap.order <- based |>  filter(!is.na(BORCAT)) |>  arrange(STUDYID, USUBJID, BORPRI, ADT, ASEQ) |>  group_by(STUDYID, USUBJID) |>  mutate(BORSEQ = row_number()) |>  ungroup() |>  select(STUDYID, USUBJID, ASEQ, BORSEQ)adrs <- based |>  left_join(order, by = c("STUDYID", "USUBJID", "ASEQ")) |>  select(STUDYID, USUBJID, ASEQ, ADT, RANDDY, AVALC, BORCAT, BORPRI, BORSEQ) |>  arrange(STUDYID, USUBJID, ASEQ)dir.create("/app/output", showWarnings = FALSE)write_csv(adrs, "/app/output/adrs.csv", na = "")

solutions/adam-adrs-response-prep/result.py

Edit
70 lines
# Reference solution for the yamaa benchmark adam-adrs-response-prep (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.from pathlib import Pathimport polars as plraw = pl.read_csv("/app/input/adrs_raw.csv", infer_schema=False).with_columns(    pl.col("ASEQ").cast(pl.Int64),    pl.col("ADT").str.to_date(),    pl.col("RANDDY").cast(pl.Int64),)therapy = pl.read_csv("/app/input/adsl.csv", infer_schema=False).with_columns(    pl.col("NTXSTDT").str.to_date(),)raw = raw.join(therapy, on=["STUDYID", "USUBJID"], how="left")# The response category each record can support; an assessment on or after# the subject's new anti-cancer therapy start supports no category, and so# do the usual suspects. Stable and neither-complete-nor-progressive# disease count only on or after day 42, and earlier ones, or ones with no# day, fall back to not evaluable.based = raw.with_columns(    BORCAT=pl.when(pl.col("NTXSTDT").is_not_null() & (pl.col("ADT") >= pl.col("NTXSTDT")))    .then(pl.lit(None, dtype=pl.String))    .when(pl.col("AVALC").is_in(["CR", "PR", "PD", "NE"]))    .then(pl.col("AVALC"))    .when((pl.col("AVALC") == "SD") & (pl.col("RANDDY") >= 42))    .then(pl.lit("SD"))    .when(pl.col("AVALC") == "SD")    .then(pl.lit("NE"))    .when((pl.col("AVALC") == "NON-CR/NON-PD") & (pl.col("RANDDY") >= 42))    .then(pl.lit("NON-CR/NON-PD"))    .when(pl.col("AVALC") == "NON-CR/NON-PD")    .then(pl.lit("NE")),).with_columns(    BORPRI=pl.col("BORCAT").replace_strict(        {"CR": 1, "PR": 2, "SD": 3, "NON-CR/NON-PD": 4, "PD": 5, "NE": 6},        default=None,        return_dtype=pl.Int64,    ))# Usable records numbered from 1 in priority, date, then sequence order;# records supporting no category are passed over without a gap.usable = (    based.filter(pl.col("BORCAT").is_not_null())    .sort(["STUDYID", "USUBJID", "BORPRI", "ADT", "ASEQ"])    .with_columns(        BORSEQ=pl.col("ASEQ").cum_count().over(["STUDYID", "USUBJID"]).cast(pl.Int64)    )    .select("STUDYID", "USUBJID", "ASEQ", "BORSEQ"))adrs = (    based.join(usable, on=["STUDYID", "USUBJID", "ASEQ"], how="left")    .select(        "STUDYID", "USUBJID", "ASEQ", "ADT", "RANDDY",        "AVALC", "BORCAT", "BORPRI", "BORSEQ",    )    .sort(["STUDYID", "USUBJID", "ASEQ"]))Path("/app/output").mkdir(exist_ok=True)adrs.write_csv("/app/output/adrs.csv")

solutions/adam-adrs-response-prep/result.sas

Edit
38 lines
/* opensas reference solution for the yamaa benchmark adam-adrs-response-prep.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data adrs_raw;  length STUDYID USUBJID ADT RANDDY AVALC $1024;  infile "/app/input/adrs_raw.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. ASEQ ADT : $char1024. RANDDY : $char1024. AVALC : $char1024.;run;data adsl;  length STUDYID USUBJID NTXSTDT $1024;  infile "/app/input/adsl.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. NTXSTDT : $char1024.;run;proc sql; create table joined as select a.*,b.NTXSTDT from adrs_raw as a left join adsl as b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID; quit;data categories;  length BORCAT $16;  set joined;  if AVALC in ('CR','PR','SD','NON-CR/NON-PD','PD','NE') and (missing(NTXSTDT) or ADT<NTXSTDT) then do;    BORCAT=AVALC;    if AVALC in ('SD','NON-CR/NON-PD') and (missing(RANDDY) or input(RANDDY,best32.)<42) then BORCAT='NE';  end;  select(BORCAT); when('CR') BORPRI=1; when('PR') BORPRI=2; when('SD') BORPRI=3; when('NON-CR/NON-PD') BORPRI=4; when('PD') BORPRI=5; when('NE') BORPRI=6; otherwise BORPRI=.; end;  unusable=missing(BORPRI);run;proc sort data=categories; by STUDYID USUBJID unusable BORPRI ADT ASEQ; run;data result;  set categories; by STUDYID USUBJID;  if first.USUBJID then sequence=0;  if not missing(BORPRI) then do; sequence+1; BORSEQ=sequence; end;run;proc sql;  create table final as select STUDYID, USUBJID, ASEQ, ADT, RANDDY, AVALC, BORCAT, BORPRI, BORSEQ from result;quit;proc export data=final outfile="/app/output/adrs.csv" dbms=csv replace; run;