Review ask

As a clinical trial statistician or programmer, challenge anything in this benchmark: describe a problem in Comments, or propose a fix with the Edit button on any source file.

  1. Task: Would a real study need this derivation, specified this way? Does it follow CDISC ADaM: variable names, controlled terms, record structure? Flag any rule that is ambiguous or that a sponsor would handle differently.
  2. Inputs: Do they look like data collected in a real trial, and do they include the situations you would expect to meet in one?
  3. Expected output: Would independent QC programming reproduce every value from these inputs and rules?

The other tabs beside Comments are background for assessment, not under review.

Summary

Lifecycle: reviewedEditUnder review
What this benchmark means
README.md

Goal: derive the combined population flag POPFL from the safety flag SAFFL and the intent-to-treat flag ITTFL, plus the dates and age they read: first exposure date TRTSDT, randomization date RANDDT, collected age AGE, age band AGEGR1, and age rank AGERNK.

Input: one demographic record per subject carrying age and the randomization date, and exposure records carrying the exposure start date.

Variables:

  • POPFL holds Y only when both SAFFL and ITTFL hold Y, and N otherwise, marking subjects in both populations.
  • SAFFL holds Y when the subject has a first exposure date, and N otherwise, marking subjects who started treatment.
  • ITTFL holds Y when the subject was randomized, and N otherwise, marking subjects in the intent-to-treat population.
  • TRTSDT holds the earliest dated exposure start for the subject, and stays empty when no exposure record carries a date.
  • RANDDT holds the collected randomization date, and stays empty when the subject was not randomized.
  • AGEGR1 holds <65 when AGE is below 65 and >=65 when it is 65 or above; a subject with no age stops the run.
  • AGERNK holds the subject rank by AGE within the study, youngest first, with USUBJID breaking ties.

Note: the dates and age are settled before the flags that read them: the safety and intent-to-treat flags come from their dates, and the combined flag comes from both flags.

Inputs

Under review
2 source files

input/dm.csv

Edit
7 rows
input/dm.csv
STUDYIDUSUBJIDAGERANDDT
YAMAA-01342025-01-05
YAMAA-01652025-02-05
YAMAA-0152—
YAMAA-01652025-02-20
YAMAA-0141—
YAMAA-01882025-03-01
YAMAA-01292025-04-01

input/ex.csv

Edit
7 rows
input/ex.csv
STUDYIDUSUBJIDEXSEQEXSTDTC
YAMAA-0112025-01-08
YAMAA-0122025-01-15
YAMAA-013—
YAMAA-0112025-02-08
YAMAA-0112025-02-28
YAMAA-0112025-03-05
YAMAA-011—

Expected output

Under review
7 expected rows

expected/adsl.csv

Edit
7 rows
expected/adsl.csv
POPFLSAFFLITTFLTRTSDTRANDDTAGEGR1AGERNKSTUDYIDUSUBJIDAGE
YYY2025-01-082025-01-05<652YAMAA-0134
YYY2025-02-082025-02-05>=655YAMAA-0165
NNN——<654YAMAA-0152
NNY—2025-02-20>=656YAMAA-0165
NYN2025-02-28—<653YAMAA-0141
YYY2025-03-052025-03-01>=657YAMAA-0188
NNY—2025-04-01<651YAMAA-0129
Shaded columns are derived— = missing value

Comments and assessment materials

Comments

Signed in with GitHub; saved in repository Discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Dataset NameVariable NameVariable LabelTypeCodelist/Controlled TermsCoreComputational MethodOrigin
ADSLTRTSDTDate of First Exposure to TreatmentnumericISO 8601Aggregate MIN(EX.EXSTDTC) over records matching on the shared output keys.Derived
ADSLSAFFLSafety Population Flagcharacter"Y" when TRTSDT IS NOT NULL; "N" otherwise; "N" when the condition is unknown.Derived
ADSLRANDDTRandomization DatenumericISO 8601Copy value from DM.RANDDT.Predecessor
ADSLITTFLIntent-to-Treat Population Flagcharacter"Y" when RANDFL = 'Y'; "N" otherwise; "N" when the condition is unknown.Derived
ADSLPOPFLAnalysis Population Flagcharacter"Y" when SAFFL = 'Y' AND ITTFL = 'Y'; "N" otherwise; "N" when the condition is unknown.Derived
ADSLSTUDYIDStudy IdentifiercharacterCopy value from DM.STUDYID.Predecessor
ADSLUSUBJIDUnique Subject IdentifiercharacterCopy value from DM.USUBJID.Predecessor
ADSLAGEAgenumericCopy value from DM.AGE.Predecessor
ADSLAGEGR1Pooled Age Group 1characterBand AGE at 65 into "<65" and ">=65", intervals closed on the left.Derived
ADSLAGERNKPooled Age Group 1 RanknumericRow number within each group of STUDYID, ordered by AGE then USUBJID.Derived
VersionDateAuthorDescriptionReviewerSign-off
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/adam-adsl-flag-chain/full.md

Edit
23 lines

Following CDISC ADaM standards, use the provided DM and EX datasets to create an ADSL dataset with one record per subject.

The output dataset should contain the following columns in this order: POPFL, SAFFL, ITTFL, TRTSDT, RANDDT, AGEGR1, AGERNK, STUDYID, USUBJID, AGE

TRTSDT is the earliest dated exposure start for the subject, and has no value when no exposure record carries a date. RANDDT is the collected randomization date, and has no value when the subject was not randomized.

SAFFL is Y when the subject has a first exposure date and N otherwise. ITTFL is Y when the subject was randomized and N otherwise. POPFL is Y only when both SAFFL and ITTFL hold Y, and N otherwise.

AGEGR1 holds <65 when AGE is below 65 and >=65 when it is 65 or above. AGERNK holds the subject rank by AGE within the study, youngest first, with USUBJID breaking ties.

Read the source datasets from /app/input and save the completed dataset as /app/output/adsl.csv.

prompts/adam-adsl-flag-chain/conventions.md

Edit
14 lines

Following CDISC ADaM standards, use the provided DM and EX datasets to create an ADSL dataset with one record per subject.

The output dataset should contain the following columns in this order: POPFL, SAFFL, ITTFL, TRTSDT, RANDDT, AGEGR1, AGERNK, STUDYID, USUBJID, AGE

SAFFL, ITTFL, and POPFL are each Y or N.

AGEGR1 holds <65 or >=65. AGERNK holds the subject rank by AGE within the study, youngest first, with USUBJID breaking ties.

Read the source datasets from /app/input and save the completed dataset as /app/output/adsl.csv.

prompts/adam-adsl-flag-chain/brief.md

Edit
9 lines

Following CDISC ADaM standards, use the provided DM and EX datasets to create an ADSL dataset with one record per subject.

The output dataset should contain the following columns in this order: POPFL, SAFFL, ITTFL, TRTSDT, RANDDT, AGEGR1, AGERNK, STUDYID, USUBJID, AGE

Read the source datasets from /app/input and save the completed dataset as /app/output/adsl.csv.

yamaa code

Runs the yamaa specification; its result must match the expected output

run.R

Edit
4 lines
library(yamaa)adsl <- yamaa_domain("spec.yaml")$outputadsl

run.py

Edit
4 lines
import yamaaadsl = yamaa.yamaa_domain("spec.yaml").outputadsl

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/adam-adsl-flag-chain/result.R

Edit
47 lines
# Reference solution for the yamaa benchmark adam-adsl-flag-chain (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)dm <- read_csv(  "/app/input/dm.csv",  col_types = cols(AGE = col_integer(), RANDDT = col_date(), .default = col_character()))ex <- read_csv(  "/app/input/ex.csv",  col_types = cols(EXSEQ = col_integer(), EXSTDTC = col_date(), .default = col_character()))first_exposure <- ex |>  group_by(STUDYID, USUBJID) |>  summarise(    TRTSDT = if (all(is.na(EXSTDTC))) as.Date(NA) else min(EXSTDTC, na.rm = TRUE),    .groups = "drop"  )adsl <- dm |>  left_join(first_exposure, by = c("STUDYID", "USUBJID")) |>  mutate(    SAFFL = if_else(!is.na(TRTSDT), "Y", "N"),    RANDFL = if_else(!is.na(RANDDT), "Y", "N"),    ITTFL = if_else(RANDFL == "Y", "Y", "N"),    POPFL = if_else(SAFFL == "Y" & ITTFL == "Y", "Y", "N"),    AGEGR1 = if_else(AGE < 65, "<65", ">=65")  ) |>  mutate(.idx = row_number()) |>  arrange(STUDYID, AGE, USUBJID) |>  group_by(STUDYID) |>  mutate(AGERNK = row_number()) |>  ungroup() |>  arrange(.idx) |>  select(    POPFL, SAFFL, ITTFL, TRTSDT, RANDDT, AGEGR1, AGERNK,    STUDYID, USUBJID, AGE  )dir.create("/app/output", showWarnings = FALSE)write_csv(adsl, "/app/output/adsl.csv", na = "")

solutions/adam-adsl-flag-chain/result.py

Edit
72 lines
# Reference solution for the yamaa benchmark adam-adsl-flag-chain (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.from pathlib import Pathimport polars as pldm = pl.read_csv("/app/input/dm.csv", infer_schema=False).with_columns(    pl.col("RANDDT").str.to_date(strict=False),    pl.col("AGE").cast(pl.Int64, strict=False),)ex = pl.read_csv("/app/input/ex.csv", infer_schema=False).with_columns(    pl.col("EXSTDTC").str.to_date(strict=False),)first_exposure = ex.group_by(["STUDYID", "USUBJID"]).agg(    TRTSDT=pl.col("EXSTDTC").min())adsl = (    dm.join(first_exposure, on=["STUDYID", "USUBJID"], how="left", maintain_order="left")    .with_columns(        SAFFL=pl.when(pl.col("TRTSDT").is_not_null())        .then(pl.lit("Y"))        .otherwise(pl.lit("N")),        RANDFL=pl.when(pl.col("RANDDT").is_not_null())        .then(pl.lit("Y"))        .otherwise(pl.lit("N")),    )    .with_columns(        ITTFL=pl.when(pl.col("RANDFL") == "Y")        .then(pl.lit("Y"))        .otherwise(pl.lit("N")),    )    .with_columns(        POPFL=pl.when((pl.col("SAFFL") == "Y") & (pl.col("ITTFL") == "Y"))        .then(pl.lit("Y"))        .otherwise(pl.lit("N")),        AGEGR1=pl.when(pl.col("AGE") < 65)        .then(pl.lit("<65"))        .when(pl.col("AGE") >= 65)        .then(pl.lit(">=65"))        .otherwise(None),    ))# Rank by age within the study, youngest first, USUBJID breaking ties.ranked = (    adsl.with_columns(__idx=pl.int_range(pl.len()))    .sort(["STUDYID", "AGE", "USUBJID"], nulls_last=True)    .with_columns(AGERNK=pl.int_range(1, pl.len() + 1).over("STUDYID"))    .sort("__idx")    .drop("__idx")    .select(        "POPFL",        "SAFFL",        "ITTFL",        "TRTSDT",        "RANDDT",        "AGEGR1",        "AGERNK",        "STUDYID",        "USUBJID",        "AGE",    ))Path("/app/output").mkdir(exist_ok=True)ranked.write_csv("/app/output/adsl.csv")

solutions/adam-adsl-flag-chain/result.sas

Edit
39 lines
/* opensas reference solution for the yamaa benchmark adam-adsl-flag-chain.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data dm;  length STUDYID USUBJID RANDDT $1024;  infile "/app/input/dm.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. AGE RANDDT : $char1024.;run;data ex;  length STUDYID USUBJID EXSTDTC $1024;  infile "/app/input/ex.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. EXSEQ EXSTDTC : $char1024.;run;proc sql;  create table firstex as select STUDYID,USUBJID,min(EXSTDTC) as TRTSDT from ex where not missing(EXSTDTC) group by STUDYID,USUBJID;  create table joined as select a.*,b.TRTSDT from dm as a left join firstex as b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID;quit;data ranked; set joined; NOAGE=missing(AGE); run;proc sort data=ranked; by STUDYID NOAGE AGE USUBJID; run;data result;  length POPFL SAFFL ITTFL $1 AGEGR1 $4;  set ranked; by STUDYID;  if first.STUDYID then AGERNK=0; AGERNK+1;  SAFFL='N'; ITTFL='N'; POPFL='N';  if not missing(TRTSDT) then SAFFL='Y';  if not missing(RANDDT) then ITTFL='Y';  if SAFFL='Y' and ITTFL='Y' then POPFL='Y';  if not missing(AGE) then do;    if AGE<65 then AGEGR1='<65'; else AGEGR1='>=65';  end;run;proc sql;  create table final as select POPFL, SAFFL, ITTFL, TRTSDT, RANDDT, AGEGR1, AGERNK, STUDYID, USUBJID, AGE from result;quit;proc export data=final outfile="/app/output/adsl.csv" dbms=csv replace; run;