As a clinical trial statistician or programmer, challenge anything in this benchmark: describe a problem in Comments, or propose a fix with the Edit button on any source file.
- Task: Would a real study need this derivation, specified this way? Does it follow CDISC ADaM: variable names, controlled terms, record structure? Flag any rule that is ambiguous or that a sponsor would handle differently.
- Inputs: Do they look like data collected in a real trial, and do they include the situations you would expect to meet in one?
- Expected output: Would independent QC programming reproduce every value from these inputs and rules?
The other tabs beside Comments are background for assessment, not under review.
Summary
README.mdGoal: assign each subject a composite responder status (AVALC),
its numeric companion (AVAL), and the reason behind the assignment
(ARSN), for every analysis visit.
Input: percent change from baseline values (PCHG) under input
parameter code EASI (Eczema Area and Severity Index), matched to a
subject-level file carrying a serious adverse event flag (SAEFL)
and a reason for discontinuation (DCSREAS). PARAMCD is fixed to
RESP75 and PARAM to EASI-75 Response.
Variables:
AVALCis the responder status:RESPONDER,NON-RESPONDER, orNOT EVALUABLE.ARSNrecords which check assigned the status:SAFETY OR DISCONTINUATION RULE,COMPONENT MISSING,THRESHOLD MET, orTHRESHOLD NOT MET.AVALis 1 for a responder and 0 for a non-responder; it stays empty when the subject is not evaluable.
Note: the checks apply in a fixed order. A subject with a serious adverse event or any discontinuation reason is a non-responder whatever the efficacy value, even a missing one; otherwise a subject with no efficacy value is not evaluable, one with a percent change of -75 or less (a reduction of at least 75%) is a responder, and everyone else is a non-responder.
Inputs
Under review2 source filesinput/adrs_raw.csv
Edit10 rows| STUDYID | USUBJID | PARAMCD | AVISIT | PCHG |
|---|---|---|---|---|
| YAMAA-01 | EASI | WEEK 4 | -80 | |
| YAMAA-01 | EASI | WEEK 8 | -88 | |
| YAMAA-01 | EASI | WEEK 4 | -50 | |
| YAMAA-01 | EASI | WEEK 4 | -75 | |
| YAMAA-01 | EASI | WEEK 4 | -74.9 | |
| YAMAA-01 | EASI | WEEK 4 | -90 | |
| YAMAA-01 | EASI | WEEK 4 | — | |
| YAMAA-01 | EASI | WEEK 4 | — | |
| YAMAA-01 | EASI | WEEK 4 | -60 | |
| YAMAA-01 | EASI | WEEK 4 | — |
input/adsl.csv
Edit9 rows| STUDYID | USUBJID | SAEFL | DCSREAS |
|---|---|---|---|
| YAMAA-01 | — | — | |
| YAMAA-01 | — | — | |
| YAMAA-01 | — | — | |
| YAMAA-01 | — | — | |
| YAMAA-01 | Y | — | |
| YAMAA-01 | — | ADVERSE EVENT | |
| YAMAA-01 | — | — | |
| YAMAA-01 | Y | LACK OF EFFICACY | |
| YAMAA-01 | Y | — |
Expected output
Under review10 expected rowsexpected/adrs.csv
Edit10 rows| STUDYID | USUBJID | PARAMCD | PARAM | AVISIT | PCHG | SAEFL | DCSREAS | AVALC | ARSN | AVAL |
|---|---|---|---|---|---|---|---|---|---|---|
| YAMAA-01 | RESP75 | EASI-75 Response | WEEK 4 | -80 | — | — | RESPONDER | THRESHOLD MET | 1 | |
| YAMAA-01 | RESP75 | EASI-75 Response | WEEK 8 | -88 | — | — | RESPONDER | THRESHOLD MET | 1 | |
| YAMAA-01 | RESP75 | EASI-75 Response | WEEK 4 | -50 | — | — | NON-RESPONDER | THRESHOLD NOT MET | 0 | |
| YAMAA-01 | RESP75 | EASI-75 Response | WEEK 4 | -75 | — | — | RESPONDER | THRESHOLD MET | 1 | |
| YAMAA-01 | RESP75 | EASI-75 Response | WEEK 4 | -74.9 | — | — | NON-RESPONDER | THRESHOLD NOT MET | 0 | |
| YAMAA-01 | RESP75 | EASI-75 Response | WEEK 4 | -90 | Y | — | NON-RESPONDER | SAFETY OR DISCONTINUATION RULE | 0 | |
| YAMAA-01 | RESP75 | EASI-75 Response | WEEK 4 | — | — | ADVERSE EVENT | NON-RESPONDER | SAFETY OR DISCONTINUATION RULE | 0 | |
| YAMAA-01 | RESP75 | EASI-75 Response | WEEK 4 | — | — | — | NOT EVALUABLE | COMPONENT MISSING | — | |
| YAMAA-01 | RESP75 | EASI-75 Response | WEEK 4 | -60 | Y | LACK OF EFFICACY | NON-RESPONDER | SAFETY OR DISCONTINUATION RULE | 0 | |
| YAMAA-01 | RESP75 | EASI-75 Response | WEEK 4 | — | Y | — | NON-RESPONDER | SAFETY OR DISCONTINUATION RULE | 0 |
Comments and assessment materials
| Dataset Name | Variable Name | Variable Label | Type | Codelist/Controlled Terms | Core | Computational Method | Origin |
|---|---|---|---|---|---|---|---|
| ADRS | STUDYID | Study Identifier | character | Copy value from ADRS_RAW.STUDYID. | Predecessor | ||
| ADRS | USUBJID | Unique Subject Identifier | character | Copy value from ADRS_RAW.USUBJID. | Predecessor | ||
| ADRS | PARAMCD | Parameter Code | character | Set constant value "RESP75". | Assigned | ||
| ADRS | PARAM | Parameter | character | Set constant value "EASI-75 Response". | Assigned | ||
| ADRS | AVISIT | Analysis Visit | character | Copy value from ADRS_RAW.AVISIT. | Predecessor | ||
| ADRS | PCHG | Percent Change from Baseline | numeric | Copy value from ADRS_RAW.PCHG. | Predecessor | ||
| ADRS | SAEFL | Serious Adverse Event Flag | character | Copy value from ADSL.SAEFL. | Predecessor | ||
| ADRS | DCSREAS | Reason for Discontinuation | character | Copy value from ADSL.DCSREAS. | Predecessor | ||
| ADRS | AVALC | Analysis Value (C) | character | RESPONDER, NON-RESPONDER, NOT EVALUABLE | Recode RULE: "SAFETY" → "NON-RESPONDER", "MISSING" → "NOT EVALUABLE", "THRESHOLD" → "RESPONDER", "OTHERWISE" → "NON-RESPONDER"; missing or unlisted values are errors. | Derived | |
| ADRS | ARSN | Analysis Reason | character | Recode RULE: "SAFETY" → "SAFETY OR DISCONTINUATION RULE", "MISSING" → "COMPONENT MISSING", "THRESHOLD" → "THRESHOLD MET", "OTHERWISE" → "THRESHOLD NOT MET"; missing or unlisted values are errors. | Derived | ||
| ADRS | AVAL | Analysis Value | numeric | If AVALC = 'RESPONDER' then "1"; If AVALC = 'NON-RESPONDER' then "0"; otherwise blank. | Derived |
| Variable | Permitted values |
|---|---|
| AVALC | RESPONDER, NON-RESPONDER, NOT EVALUABLE |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/adam-adrs-composite-response/full.md
Edit25 linesFollowing CDISC ADaM standards, use the provided ADRS_RAW and ADSL datasets to create an ADRS dataset with one record per subject per analysis visit.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, AVISIT, PCHG, SAEFL, DCSREAS, AVALC, ARSN, AVAL
Use PARAMCD "RESP75" and PARAM "EASI-75 Response".
The checks apply in a fixed order. A subject with a serious adverse event or any discontinuation reason is a non-responder whatever the efficacy value, even a missing one; otherwise a subject with no efficacy value is not evaluable, one with a percent change of -75 or less (a reduction of at least 75%) is a responder, and everyone else is a non-responder.
AVALC is the responder status: RESPONDER, NON-RESPONDER, or NOT EVALUABLE. ARSN records which check assigned the status: SAFETY OR DISCONTINUATION RULE, COMPONENT MISSING, THRESHOLD MET, or THRESHOLD NOT MET. AVAL is 1 for a responder and 0 for a non-responder; it has no value when the subject is not evaluable.
Read the source datasets from /app/input and save the completed dataset as /app/output/adrs.csv.
prompts/adam-adrs-composite-response/conventions.md
Edit17 linesFollowing CDISC ADaM standards, use the provided ADRS_RAW and ADSL datasets to create an ADRS dataset with one record per subject per analysis visit.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, AVISIT, PCHG, SAEFL, DCSREAS, AVALC, ARSN, AVAL
Use PARAMCD "RESP75" and PARAM "EASI-75 Response".
AVALC is RESPONDER, NON-RESPONDER, or NOT EVALUABLE. ARSN is SAFETY OR DISCONTINUATION RULE, COMPONENT MISSING, THRESHOLD MET, or THRESHOLD NOT MET. AVAL is 1 for a responder and 0 for a non-responder; it has no value when the subject is not evaluable.
Read the source datasets from /app/input and save the completed dataset as /app/output/adrs.csv.
prompts/adam-adrs-composite-response/brief.md
Edit10 linesFollowing CDISC ADaM standards, use the provided ADRS_RAW and ADSL datasets to create an ADRS dataset with one record per subject per analysis visit.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, AVISIT, PCHG, SAEFL, DCSREAS, AVALC, ARSN, AVAL
Read the source datasets from /app/input and save the completed dataset as /app/output/adrs.csv.
run.R
Edit4 lineslibrary(yamaa)adrs <- yamaa_domain("spec.yaml")$outputadrsrun.py
Edit4 linesimport yamaaadrs = yamaa.yamaa_domain("spec.yaml").outputadrssolutions/adam-adrs-composite-response/result.R
Edit51 lines# Reference solution for the yamaa benchmark adam-adrs-composite-response (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)adsl <- read_csv( "/app/input/adsl.csv", col_types = cols(.default = col_character()))raw <- read_csv( "/app/input/adrs_raw.csv", col_types = cols(PCHG = col_double(), .default = col_character()))# The checks apply in a fixed order: a safety or discontinuation rule# first, then a missing component, then the 75% reduction threshold.adrs <- raw |> left_join(select(adsl, USUBJID, SAEFL, DCSREAS), by = "USUBJID") |> mutate( PARAMCD = "RESP75", PARAM = "EASI-75 Response", safety = SAEFL %in% "Y" | (!is.na(DCSREAS) & DCSREAS != ""), AVALC = case_when( safety ~ "NON-RESPONDER", is.na(PCHG) ~ "NOT EVALUABLE", PCHG <= -75 ~ "RESPONDER", .default = "NON-RESPONDER" ), ARSN = case_when( safety ~ "SAFETY OR DISCONTINUATION RULE", is.na(PCHG) ~ "COMPONENT MISSING", PCHG <= -75 ~ "THRESHOLD MET", .default = "THRESHOLD NOT MET" ), AVAL = case_when( AVALC == "RESPONDER" ~ 1L, AVALC == "NON-RESPONDER" ~ 0L ) ) |> select( STUDYID, USUBJID, PARAMCD, PARAM, AVISIT, PCHG, SAEFL, DCSREAS, AVALC, ARSN, AVAL ) |> arrange(USUBJID, AVISIT)dir.create("/app/output", showWarnings = FALSE)write_csv(adrs, "/app/output/adrs.csv", na = "")solutions/adam-adrs-composite-response/result.py
Edit65 lines# Reference solution for the yamaa benchmark adam-adrs-composite-response (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.from pathlib import Pathimport polars as pladsl = pl.read_csv("/app/input/adsl.csv", infer_schema=False)raw = pl.read_csv("/app/input/adrs_raw.csv", infer_schema=False).with_columns( pl.col("PCHG").cast(pl.Float64))base = raw.join(adsl.select("USUBJID", "SAEFL", "DCSREAS"), on="USUBJID", how="left")# The checks apply in a fixed order: a safety or discontinuation rule# first, then a missing component, then the 75% reduction threshold.safety = (pl.col("SAEFL") == "Y") | ( pl.col("DCSREAS").is_not_null() & (pl.col("DCSREAS") != ""))avalc = ( pl.when(safety) .then(pl.lit("NON-RESPONDER")) .when(pl.col("PCHG").is_null()) .then(pl.lit("NOT EVALUABLE")) .when(pl.col("PCHG") <= -75) .then(pl.lit("RESPONDER")) .otherwise(pl.lit("NON-RESPONDER")))arsn = ( pl.when(safety) .then(pl.lit("SAFETY OR DISCONTINUATION RULE")) .when(pl.col("PCHG").is_null()) .then(pl.lit("COMPONENT MISSING")) .when(pl.col("PCHG") <= -75) .then(pl.lit("THRESHOLD MET")) .otherwise(pl.lit("THRESHOLD NOT MET")))adrs = ( base.with_columns( PARAMCD=pl.lit("RESP75"), PARAM=pl.lit("EASI-75 Response"), AVALC=avalc, ARSN=arsn, ) .with_columns( AVAL=pl.when(pl.col("AVALC") == "RESPONDER") .then(1) .when(pl.col("AVALC") == "NON-RESPONDER") .then(0) .otherwise(None) .cast(pl.Int64) ) .select( "STUDYID", "USUBJID", "PARAMCD", "PARAM", "AVISIT", "PCHG", "SAEFL", "DCSREAS", "AVALC", "ARSN", "AVAL", ) .sort(["USUBJID", "AVISIT"]))Path("/app/output").mkdir(exist_ok=True)adrs.write_csv("/app/output/adrs.csv")solutions/adam-adrs-composite-response/result.sas
Edit30 lines/* opensas reference solution for the yamaa benchmark adam-adrs-composite-response. Reads the benchmark inputs and derives the requested output. Execute with the existing opensas runtime: opensas /app/output/result.sas. */data adrs_raw; length STUDYID USUBJID PARAMCD AVISIT $1024; infile "/app/input/adrs_raw.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. PARAMCD : $char1024. AVISIT : $char1024. PCHG;run;data adsl; length STUDYID USUBJID SAEFL DCSREAS $1024; infile "/app/input/adsl.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. SAEFL : $char1024. DCSREAS : $char1024.;run;proc sql; create table joined as select a.*,b.SAEFL,b.DCSREAS from adrs_raw as a left join adsl as b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID; quit;data result; length PARAM $40 AVALC $20 ARSN $40; set joined; PARAMCD='RESP75'; PARAM='EASI-75 Response'; if SAEFL='Y' or not missing(DCSREAS) then do; AVALC='NON-RESPONDER'; ARSN='SAFETY OR DISCONTINUATION RULE'; AVAL=0; end; else if missing(PCHG) then do; AVALC='NOT EVALUABLE'; ARSN='COMPONENT MISSING'; end; else if PCHG<=-75 then do; AVALC='RESPONDER'; ARSN='THRESHOLD MET'; AVAL=1; end; else do; AVALC='NON-RESPONDER'; ARSN='THRESHOLD NOT MET'; AVAL=0; end;run;proc sql; create table final as select STUDYID, USUBJID, PARAMCD, PARAM, AVISIT, PCHG, SAEFL, DCSREAS, AVALC, ARSN, AVAL from result;quit;proc export data=final outfile="/app/output/adrs.csv" dbms=csv replace; run;
yamaa
Comments
Signed in with GitHub; saved in repository Discussions