Review ask

As a clinical trial statistician or programmer, challenge anything in this benchmark: describe a problem in Comments, or propose a fix with the Edit button on any source file.

  1. Task: Would a real study need this derivation, specified this way? Does it follow CDISC ADaM: variable names, controlled terms, record structure? Flag any rule that is ambiguous or that a sponsor would handle differently.
  2. Inputs: Do they look like data collected in a real trial, and do they include the situations you would expect to meet in one?
  3. Expected output: Would independent QC programming reproduce every value from these inputs and rules?

The other tabs beside Comments are background for assessment, not under review.

Summary

Lifecycle: reviewedEditUnder review
What this benchmark means
README.md

Goal: assign each subject a composite responder status (AVALC), its numeric companion (AVAL), and the reason behind the assignment (ARSN), for every analysis visit.

Input: percent change from baseline values (PCHG) under input parameter code EASI (Eczema Area and Severity Index), matched to a subject-level file carrying a serious adverse event flag (SAEFL) and a reason for discontinuation (DCSREAS). PARAMCD is fixed to RESP75 and PARAM to EASI-75 Response.

Variables:

  • AVALC is the responder status: RESPONDER, NON-RESPONDER, or NOT EVALUABLE.
  • ARSN records which check assigned the status: SAFETY OR DISCONTINUATION RULE, COMPONENT MISSING, THRESHOLD MET, or THRESHOLD NOT MET.
  • AVAL is 1 for a responder and 0 for a non-responder; it stays empty when the subject is not evaluable.

Note: the checks apply in a fixed order. A subject with a serious adverse event or any discontinuation reason is a non-responder whatever the efficacy value, even a missing one; otherwise a subject with no efficacy value is not evaluable, one with a percent change of -75 or less (a reduction of at least 75%) is a responder, and everyone else is a non-responder.

Inputs

Under review
2 source files

input/adrs_raw.csv

Edit
10 rows
input/adrs_raw.csv
STUDYIDUSUBJIDPARAMCDAVISITPCHG
YAMAA-01EASIWEEK 4-80
YAMAA-01EASIWEEK 8-88
YAMAA-01EASIWEEK 4-50
YAMAA-01EASIWEEK 4-75
YAMAA-01EASIWEEK 4-74.9
YAMAA-01EASIWEEK 4-90
YAMAA-01EASIWEEK 4—
YAMAA-01EASIWEEK 4—
YAMAA-01EASIWEEK 4-60
YAMAA-01EASIWEEK 4—

input/adsl.csv

Edit
9 rows
input/adsl.csv
STUDYIDUSUBJIDSAEFLDCSREAS
YAMAA-01——
YAMAA-01——
YAMAA-01——
YAMAA-01——
YAMAA-01Y—
YAMAA-01—ADVERSE EVENT
YAMAA-01——
YAMAA-01YLACK OF EFFICACY
YAMAA-01Y—

Expected output

Under review
10 expected rows

expected/adrs.csv

Edit
10 rows
expected/adrs.csv
STUDYIDUSUBJIDPARAMCDPARAMAVISITPCHGSAEFLDCSREASAVALCARSNAVAL
YAMAA-01RESP75EASI-75 ResponseWEEK 4-80——RESPONDERTHRESHOLD MET1
YAMAA-01RESP75EASI-75 ResponseWEEK 8-88——RESPONDERTHRESHOLD MET1
YAMAA-01RESP75EASI-75 ResponseWEEK 4-50——NON-RESPONDERTHRESHOLD NOT MET0
YAMAA-01RESP75EASI-75 ResponseWEEK 4-75——RESPONDERTHRESHOLD MET1
YAMAA-01RESP75EASI-75 ResponseWEEK 4-74.9——NON-RESPONDERTHRESHOLD NOT MET0
YAMAA-01RESP75EASI-75 ResponseWEEK 4-90Y—NON-RESPONDERSAFETY OR DISCONTINUATION RULE0
YAMAA-01RESP75EASI-75 ResponseWEEK 4——ADVERSE EVENTNON-RESPONDERSAFETY OR DISCONTINUATION RULE0
YAMAA-01RESP75EASI-75 ResponseWEEK 4———NOT EVALUABLECOMPONENT MISSING—
YAMAA-01RESP75EASI-75 ResponseWEEK 4-60YLACK OF EFFICACYNON-RESPONDERSAFETY OR DISCONTINUATION RULE0
YAMAA-01RESP75EASI-75 ResponseWEEK 4—Y—NON-RESPONDERSAFETY OR DISCONTINUATION RULE0
Shaded columns are derived— = missing value

Comments and assessment materials

Comments

Signed in with GitHub; saved in repository Discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Dataset NameVariable NameVariable LabelTypeCodelist/Controlled TermsCoreComputational MethodOrigin
ADRSSTUDYIDStudy IdentifiercharacterCopy value from ADRS_RAW.STUDYID.Predecessor
ADRSUSUBJIDUnique Subject IdentifiercharacterCopy value from ADRS_RAW.USUBJID.Predecessor
ADRSPARAMCDParameter CodecharacterSet constant value "RESP75".Assigned
ADRSPARAMParametercharacterSet constant value "EASI-75 Response".Assigned
ADRSAVISITAnalysis VisitcharacterCopy value from ADRS_RAW.AVISIT.Predecessor
ADRSPCHGPercent Change from BaselinenumericCopy value from ADRS_RAW.PCHG.Predecessor
ADRSSAEFLSerious Adverse Event FlagcharacterCopy value from ADSL.SAEFL.Predecessor
ADRSDCSREASReason for DiscontinuationcharacterCopy value from ADSL.DCSREAS.Predecessor
ADRSAVALCAnalysis Value (C)characterRESPONDER, NON-RESPONDER, NOT EVALUABLERecode RULE: "SAFETY" → "NON-RESPONDER", "MISSING" → "NOT EVALUABLE", "THRESHOLD" → "RESPONDER", "OTHERWISE" → "NON-RESPONDER"; missing or unlisted values are errors.Derived
ADRSARSNAnalysis ReasoncharacterRecode RULE: "SAFETY" → "SAFETY OR DISCONTINUATION RULE", "MISSING" → "COMPONENT MISSING", "THRESHOLD" → "THRESHOLD MET", "OTHERWISE" → "THRESHOLD NOT MET"; missing or unlisted values are errors.Derived
ADRSAVALAnalysis ValuenumericIf AVALC = 'RESPONDER' then "1"; If AVALC = 'NON-RESPONDER' then "0"; otherwise blank.Derived
VariablePermitted values
AVALCRESPONDER, NON-RESPONDER, NOT EVALUABLE
VersionDateAuthorDescriptionReviewerSign-off
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/adam-adrs-composite-response/full.md

Edit
25 lines

Following CDISC ADaM standards, use the provided ADRS_RAW and ADSL datasets to create an ADRS dataset with one record per subject per analysis visit.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, AVISIT, PCHG, SAEFL, DCSREAS, AVALC, ARSN, AVAL

Use PARAMCD "RESP75" and PARAM "EASI-75 Response".

The checks apply in a fixed order. A subject with a serious adverse event or any discontinuation reason is a non-responder whatever the efficacy value, even a missing one; otherwise a subject with no efficacy value is not evaluable, one with a percent change of -75 or less (a reduction of at least 75%) is a responder, and everyone else is a non-responder.

AVALC is the responder status: RESPONDER, NON-RESPONDER, or NOT EVALUABLE. ARSN records which check assigned the status: SAFETY OR DISCONTINUATION RULE, COMPONENT MISSING, THRESHOLD MET, or THRESHOLD NOT MET. AVAL is 1 for a responder and 0 for a non-responder; it has no value when the subject is not evaluable.

Read the source datasets from /app/input and save the completed dataset as /app/output/adrs.csv.

prompts/adam-adrs-composite-response/conventions.md

Edit
17 lines

Following CDISC ADaM standards, use the provided ADRS_RAW and ADSL datasets to create an ADRS dataset with one record per subject per analysis visit.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, AVISIT, PCHG, SAEFL, DCSREAS, AVALC, ARSN, AVAL

Use PARAMCD "RESP75" and PARAM "EASI-75 Response".

AVALC is RESPONDER, NON-RESPONDER, or NOT EVALUABLE. ARSN is SAFETY OR DISCONTINUATION RULE, COMPONENT MISSING, THRESHOLD MET, or THRESHOLD NOT MET. AVAL is 1 for a responder and 0 for a non-responder; it has no value when the subject is not evaluable.

Read the source datasets from /app/input and save the completed dataset as /app/output/adrs.csv.

prompts/adam-adrs-composite-response/brief.md

Edit
10 lines

Following CDISC ADaM standards, use the provided ADRS_RAW and ADSL datasets to create an ADRS dataset with one record per subject per analysis visit.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, AVISIT, PCHG, SAEFL, DCSREAS, AVALC, ARSN, AVAL

Read the source datasets from /app/input and save the completed dataset as /app/output/adrs.csv.

yamaa code

Runs the yamaa specification; its result must match the expected output

run.R

Edit
4 lines
library(yamaa)adrs <- yamaa_domain("spec.yaml")$outputadrs

run.py

Edit
4 lines
import yamaaadrs = yamaa.yamaa_domain("spec.yaml").outputadrs

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/adam-adrs-composite-response/result.R

Edit
51 lines
# Reference solution for the yamaa benchmark adam-adrs-composite-response (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)adsl <- read_csv(  "/app/input/adsl.csv",  col_types = cols(.default = col_character()))raw <- read_csv(  "/app/input/adrs_raw.csv",  col_types = cols(PCHG = col_double(), .default = col_character()))# The checks apply in a fixed order: a safety or discontinuation rule# first, then a missing component, then the 75% reduction threshold.adrs <- raw |>  left_join(select(adsl, USUBJID, SAEFL, DCSREAS), by = "USUBJID") |>  mutate(    PARAMCD = "RESP75",    PARAM = "EASI-75 Response",    safety = SAEFL %in% "Y" | (!is.na(DCSREAS) & DCSREAS != ""),    AVALC = case_when(      safety ~ "NON-RESPONDER",      is.na(PCHG) ~ "NOT EVALUABLE",      PCHG <= -75 ~ "RESPONDER",      .default = "NON-RESPONDER"    ),    ARSN = case_when(      safety ~ "SAFETY OR DISCONTINUATION RULE",      is.na(PCHG) ~ "COMPONENT MISSING",      PCHG <= -75 ~ "THRESHOLD MET",      .default = "THRESHOLD NOT MET"    ),    AVAL = case_when(      AVALC == "RESPONDER" ~ 1L,      AVALC == "NON-RESPONDER" ~ 0L    )  ) |>  select(    STUDYID, USUBJID, PARAMCD, PARAM, AVISIT, PCHG, SAEFL, DCSREAS,    AVALC, ARSN, AVAL  ) |>  arrange(USUBJID, AVISIT)dir.create("/app/output", showWarnings = FALSE)write_csv(adrs, "/app/output/adrs.csv", na = "")

solutions/adam-adrs-composite-response/result.py

Edit
65 lines
# Reference solution for the yamaa benchmark adam-adrs-composite-response (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.from pathlib import Pathimport polars as pladsl = pl.read_csv("/app/input/adsl.csv", infer_schema=False)raw = pl.read_csv("/app/input/adrs_raw.csv", infer_schema=False).with_columns(    pl.col("PCHG").cast(pl.Float64))base = raw.join(adsl.select("USUBJID", "SAEFL", "DCSREAS"), on="USUBJID", how="left")# The checks apply in a fixed order: a safety or discontinuation rule# first, then a missing component, then the 75% reduction threshold.safety = (pl.col("SAEFL") == "Y") | (    pl.col("DCSREAS").is_not_null() & (pl.col("DCSREAS") != ""))avalc = (    pl.when(safety)    .then(pl.lit("NON-RESPONDER"))    .when(pl.col("PCHG").is_null())    .then(pl.lit("NOT EVALUABLE"))    .when(pl.col("PCHG") <= -75)    .then(pl.lit("RESPONDER"))    .otherwise(pl.lit("NON-RESPONDER")))arsn = (    pl.when(safety)    .then(pl.lit("SAFETY OR DISCONTINUATION RULE"))    .when(pl.col("PCHG").is_null())    .then(pl.lit("COMPONENT MISSING"))    .when(pl.col("PCHG") <= -75)    .then(pl.lit("THRESHOLD MET"))    .otherwise(pl.lit("THRESHOLD NOT MET")))adrs = (    base.with_columns(        PARAMCD=pl.lit("RESP75"),        PARAM=pl.lit("EASI-75 Response"),        AVALC=avalc,        ARSN=arsn,    )    .with_columns(        AVAL=pl.when(pl.col("AVALC") == "RESPONDER")        .then(1)        .when(pl.col("AVALC") == "NON-RESPONDER")        .then(0)        .otherwise(None)        .cast(pl.Int64)    )    .select(        "STUDYID", "USUBJID", "PARAMCD", "PARAM", "AVISIT", "PCHG",        "SAEFL", "DCSREAS", "AVALC", "ARSN", "AVAL",    )    .sort(["USUBJID", "AVISIT"]))Path("/app/output").mkdir(exist_ok=True)adrs.write_csv("/app/output/adrs.csv")

solutions/adam-adrs-composite-response/result.sas

Edit
30 lines
/* opensas reference solution for the yamaa benchmark adam-adrs-composite-response.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data adrs_raw;  length STUDYID USUBJID PARAMCD AVISIT $1024;  infile "/app/input/adrs_raw.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. PARAMCD : $char1024. AVISIT : $char1024. PCHG;run;data adsl;  length STUDYID USUBJID SAEFL DCSREAS $1024;  infile "/app/input/adsl.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. SAEFL : $char1024. DCSREAS : $char1024.;run;proc sql; create table joined as select a.*,b.SAEFL,b.DCSREAS from adrs_raw as a left join adsl as b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID; quit;data result;  length PARAM $40 AVALC $20 ARSN $40;  set joined; PARAMCD='RESP75'; PARAM='EASI-75 Response';  if SAEFL='Y' or not missing(DCSREAS) then do; AVALC='NON-RESPONDER'; ARSN='SAFETY OR DISCONTINUATION RULE'; AVAL=0; end;  else if missing(PCHG) then do; AVALC='NOT EVALUABLE'; ARSN='COMPONENT MISSING'; end;  else if PCHG<=-75 then do; AVALC='RESPONDER'; ARSN='THRESHOLD MET'; AVAL=1; end;  else do; AVALC='NON-RESPONDER'; ARSN='THRESHOLD NOT MET'; AVAL=0; end;run;proc sql;  create table final as select STUDYID, USUBJID, PARAMCD, PARAM, AVISIT, PCHG, SAEFL, DCSREAS, AVALC, ARSN, AVAL from result;quit;proc export data=final outfile="/app/output/adrs.csv" dbms=csv replace; run;