Review ask

As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.

We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.

Summary

Lifecycle: draftEditUnder review
What this benchmark means
README.md

Public entries: Python and R. Passing coverage requires an installed inventory.

Goal: one row per adverse event (AE), flagging the phase in which the event is treatment-emergent (TREM01FL, TREM02FL, TREM03FL) and whether it is treatment-emergent at all (TRTEMFL), in an oncology study whose phases are neoadjuvant treatment, surgery and adjuvant treatment.

Input: AE records carrying the reported term (AETERM), whether the event is serious (AESER, Y or N) and the onset date (AESTDTC), plus the subject-level analysis dataset (ADSL) with the first and last dates of each phase: neoadjuvant (TR01SDT, TR01EDT), surgery (TR02SDT, TR02EDT) and adjuvant (TR03SDT, TR03EDT), and the end-of-treatment status (EOTSTT: COMPLETED, DISCONTINUED or ONGOING). A phase the subject never entered has empty dates; an untreated subject has no dates and no status.

Variables:

  • ASTDT is the onset date.
  • TREM01FL holds Y for an event that started in the neoadjuvant phase: on or after TR01SDT and before the next phase the subject entered starts, which is surgery or, for a subject who had no surgery, the first adjuvant dose. Empty for any other event.
  • TREM02FL holds Y for an event that started in the surgery phase: on or after the surgery date TR02SDT and before the first adjuvant dose TR03SDT. Empty for any other event.
  • TREM03FL holds Y for an event that started in the adjuvant phase: on or after the first adjuvant dose TR03SDT. Empty for any other event.
  • TRTEMFL holds Y when one of the three phase flags is Y; empty otherwise.

Note: each phase starts on its own first date and ends the day before the next phase the subject entered starts, so an event on the day of surgery is a surgery-phase event and an event between phases belongs to the earlier phase however long the gap. The last phase a subject entered has no next phase to end it: it ends 30 days after its last date (TR01EDT, TR02EDT or TR03EDT) for a non-serious event and 90 days after it for a serious one, both days included, and has no end while the subject's treatment is ongoing (EOTSTT is ONGOING). An event before TR01SDT, an event after the last phase ends and every event of an untreated subject carry no flags. The window lengths and the ongoing rule are this contract's choices; a protocol sets its own.

Inputs

Under review
2 source files

input/adsl.csv

Edit
6 rows
input/adsl.csv
STUDYIDUSUBJIDTR01SDTTR01EDTTR02SDTTR02EDTTR03SDTTR03EDTEOTSTT
YAMAA-012025-01-062025-02-242025-04-072025-04-072025-05-192025-10-13COMPLETED
YAMAA-012025-02-032025-04-14————DISCONTINUED
YAMAA-012025-03-032025-05-122025-06-092025-06-09——DISCONTINUED
YAMAA-012025-01-132025-03-032025-04-142025-04-142025-05-262025-08-18ONGOING
YAMAA-01———————
YAMAA-012025-02-102025-03-31——2025-05-262025-09-15COMPLETED

input/ae.csv

Edit
13 rows
input/ae.csv
STUDYIDUSUBJIDAESEQAETERMAESERAESTDTC
YAMAA-011COUGHN2025-01-03
YAMAA-012FATIGUEN2025-04-06
YAMAA-013HYPOTENSIONN2025-04-07
YAMAA-014WOUND INFECTIONN2025-05-12
YAMAA-015PNEUMONIAY2025-12-12
YAMAA-016RASHN2025-12-12
YAMAA-011DIARRHOEAN2025-05-14
YAMAA-012HEADACHEN2025-05-15
YAMAA-011PULMONARY EMBOLISMY2025-09-07
YAMAA-012SEPSISY2025-09-08
YAMAA-011ARTHRALGIAN2025-11-26
YAMAA-011NAUSEAN2025-03-17
YAMAA-011PERIPHERAL NEUROPATHYN2025-05-10

Expected output

Under review
13 expected rows

expected/adae.csv

Edit
13 rows
expected/adae.csv
STUDYIDUSUBJIDAESEQAETERMAESERASTDTTREM01FLTREM02FLTREM03FLTRTEMFL
YAMAA-011COUGHN2025-01-03————
YAMAA-012FATIGUEN2025-04-06Y——Y
YAMAA-013HYPOTENSIONN2025-04-07—Y—Y
YAMAA-014WOUND INFECTIONN2025-05-12—Y—Y
YAMAA-015PNEUMONIAY2025-12-12——YY
YAMAA-016RASHN2025-12-12————
YAMAA-011DIARRHOEAN2025-05-14Y——Y
YAMAA-012HEADACHEN2025-05-15————
YAMAA-011PULMONARY EMBOLISMY2025-09-07—Y—Y
YAMAA-012SEPSISY2025-09-08————
YAMAA-011ARTHRALGIAN2025-11-26——YY
YAMAA-011NAUSEAN2025-03-17————
YAMAA-011PERIPHERAL NEUROPATHYN2025-05-10Y——Y
Shaded columns are derived— = missing value

Human review and assessment

Your review

3 questions and your comments · View benchmark discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Dataset NameVariable NameVariable LabelTypeCodelist/Controlled TermsCoreComputational MethodOrigin
ADAESTUDYIDStudy IdentifiercharacterCopy value from AE.STUDYID.Predecessor
ADAEUSUBJIDUnique Subject IdentifiercharacterCopy value from AE.USUBJID.Predecessor
ADAEAESEQSequence NumbernumericCopy value from AE.AESEQ.Predecessor
ADAEAETERMReported Term for the Adverse EventcharacterCopy value from AE.AETERM.Predecessor
ADAEAESERSerious EventcharacterY, NCopy value from AE.AESER.Predecessor
ADAEASTDTAnalysis Start DatenumericISO 8601Copy value from AE.AESTDTC.Predecessor
ADAETREM01FLTreatment Emergent Flag for Period 01characterY"Y" when ADSL.TR01SDT IS NOT NULL AND ASTDT >= ADSL.TR01SDT AND ((NEXT01DT IS NOT NULL AND ASTDT < NEXT01DT) OR (NEXT01DT IS NULL AND (ADSL.EOTSTT = 'ONGOING' OR ASTDT <= TEENDDT))); blank otherwise.Derived
ADAETREM02FLTreatment Emergent Flag for Period 02characterY"Y" when ADSL.TR02SDT IS NOT NULL AND ASTDT >= ADSL.TR02SDT AND ((ADSL.TR03SDT IS NOT NULL AND ASTDT < ADSL.TR03SDT) OR (ADSL.TR03SDT IS NULL AND (ADSL.EOTSTT = 'ONGOING' OR ASTDT <= TEENDDT))); blank otherwise.Derived
ADAETREM03FLTreatment Emergent Flag for Period 03characterY"Y" when ADSL.TR03SDT IS NOT NULL AND ASTDT >= ADSL.TR03SDT AND (ADSL.EOTSTT = 'ONGOING' OR ASTDT <= TEENDDT); blank otherwise.Derived
ADAETRTEMFLTreatment Emergent Analysis FlagcharacterY"Y" when TREM01FL IS NOT NULL OR TREM02FL IS NOT NULL OR TREM03FL IS NOT NULL; blank otherwise.Derived
VariablePermitted values
AESERY, N
TREM01FLY
TREM02FLY
TREM03FLY
TRTEMFLY
VersionDateAuthorDescriptionReviewerSign-off
1.72026-10-20yamaa docs buildRead a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default
1.62026-10-17yamaa docs buildDescribe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight
1.52026-10-17yamaa docs buildName collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part
1.42026-10-15yamaa docs buildState datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/adam-adae-phase-emergence/full.md

Edit
36 lines

Following CDISC ADaM standards, use the provided ADSL and AE datasets to create an ADAE dataset with one record per adverse event.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, AESEQ, AETERM, AESER, ASTDT, TREM01FL, TREM02FL, TREM03FL, TRTEMFL

The study has three phases, each with its first and last dates in ADSL: neoadjuvant treatment (TR01SDT, TR01EDT), surgery (TR02SDT, TR02EDT) and adjuvant treatment (TR03SDT, TR03EDT). A phase the subject never entered has no dates, and an untreated subject has no phase dates at all.

  • ASTDT is the onset date, AESTDTC.
  • Each phase starts on its own first date and ends the day before the next phase the subject entered starts. The next phase after neoadjuvant treatment is surgery or, for a subject with no surgery, adjuvant treatment; the next phase after surgery is adjuvant treatment. So an event on the day of surgery is a surgery-phase event, and an event in a gap between two phases belongs to the earlier phase however long the gap.
  • The last phase the subject entered has no next phase. It ends 30 days after its last date (TR01EDT, TR02EDT or TR03EDT) for a non-serious event (AESER "N") and 90 days after it for a serious event (AESER "Y"), the end day included: a non-serious event 30 days after the last date is in the phase, one 31 days after it is not. When EOTSTT is "ONGOING", the last phase has no end.
  • TREM01FL is "Y" for an event that started in the neoadjuvant phase, TREM02FL is "Y" for one that started in the surgery phase, and TREM03FL is "Y" for one that started in the adjuvant phase. Each has no value for any other event.
  • TRTEMFL is "Y" when one of the three phase flags is "Y" and has no value otherwise.
  • An event before TR01SDT, an event after the last phase ends, and every event of an untreated subject have no value in any of the four flags.

Read the source datasets from /app/input and save the completed dataset as /app/output/adae.csv.

prompts/adam-adae-phase-emergence/conventions.md

Edit
18 lines

Following CDISC ADaM standards, use the provided ADSL and AE datasets to create an ADAE dataset with one record per adverse event.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, AESEQ, AETERM, AESER, ASTDT, TREM01FL, TREM02FL, TREM03FL, TRTEMFL

The study has three phases, each with its first and last dates in ADSL: neoadjuvant treatment (TR01SDT, TR01EDT), surgery (TR02SDT, TR02EDT) and adjuvant treatment (TR03SDT, TR03EDT). TREM01FL, TREM02FL and TREM03FL are the flags of the neoadjuvant, surgery and adjuvant phases, and each is "Y" or has no value; TRTEMFL is "Y" or has no value. Days after a phase's last date are counted with the end day included, for both the 30-day and the 90-day limit: an event 30 days after the last date is within 30 days, one 31 days after it is not.

Read the source datasets from /app/input and save the completed dataset as /app/output/adae.csv.

prompts/adam-adae-phase-emergence/brief.md

Edit
9 lines

Following CDISC ADaM standards, use the provided ADSL and AE datasets to create an ADAE dataset with one record per adverse event.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, AESEQ, AETERM, AESER, ASTDT, TREM01FL, TREM02FL, TREM03FL, TRTEMFL

Read the source datasets from /app/input and save the completed dataset as /app/output/adae.csv.

yamaa code

Public calls generated from benchmark.yaml; results must match the expected output

R

Edit
3 lines
library(yamaa)yamaa_spec("spec.yaml")$run()$output

Python

Edit
3 lines
import yamaayamaa.spec("spec.yaml").run().output

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/adam-adae-phase-emergence/result.R

Edit
82 lines
# Reference solution for the yamaa benchmark adam-adae-phase-emergence (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.# Derive ADAE with phase-specific treatment-emergent flags from ADSL and AE.suppressPackageStartupMessages({  library(dplyr)  library(readr)})input_dir <- "/app/input"output_dir <- "/app/output"dir.create(output_dir, showWarnings = FALSE, recursive = TRUE)adsl <- read_csv(file.path(input_dir, "adsl.csv"),                 col_types = cols(.default = col_character()),                 na = c("", "NA"))ae <- read_csv(file.path(input_dir, "ae.csv"),               col_types = cols(.default = col_character()),               na = c("", "NA"))to_date <- function(x) as.Date(x, format = "%Y-%m-%d")adsl <- adsl %>%  mutate(across(matches("^TR0[123][SE]DT$"), to_date))# Determine the phase (1, 2, 3) an event started in, or NA when none.phase_of_event <- function(astdt, aeser, s1, e1, s2, e2, s3, e3, eotstt) {  if (is.na(astdt)) return(NA_integer_)  starts <- c(s1, s2, s3)  ends <- c(e1, e2, e3)  entered <- which(!is.na(starts))  if (length(entered) == 0) return(NA_integer_)  if (astdt < starts[entered[1]]) return(NA_integer_)  for (i in seq_along(entered)) {    k <- entered[i]    if (astdt < starts[k]) next    if (i < length(entered)) {      phase_end <- starts[entered[i + 1]] - 1      if (astdt <= phase_end) return(k)    } else {      if (!is.na(eotstt) && eotstt == "ONGOING") return(k)      if (is.na(ends[k])) return(NA_integer_)      window <- if (!is.na(aeser) && aeser == "Y") 90 else if (!is.na(aeser) && aeser == "N") 30 else NA      if (is.na(window)) return(NA_integer_)      if (astdt <= ends[k] + window) return(k)    }  }  NA_integer_}adae <- ae %>%  left_join(adsl %>% select(-STUDYID), by = "USUBJID") %>%  mutate(    ASTDT = to_date(AESTDTC),    AESEQ = as.integer(AESEQ)  )adae$PHASE <- vapply(  seq_len(nrow(adae)),  function(r) {    with(adae[r, ], phase_of_event(ASTDT, AESER, TR01SDT, TR01EDT, TR02SDT,                                   TR02EDT, TR03SDT, TR03EDT, EOTSTT))  },  integer(1))adae <- adae %>%  mutate(    TREM01FL = if_else(!is.na(PHASE) & PHASE == 1L, "Y", NA_character_),    TREM02FL = if_else(!is.na(PHASE) & PHASE == 2L, "Y", NA_character_),    TREM03FL = if_else(!is.na(PHASE) & PHASE == 3L, "Y", NA_character_),    TRTEMFL = if_else(!is.na(PHASE), "Y", NA_character_)  ) %>%  arrange(STUDYID, USUBJID, AESEQ) %>%  select(STUDYID, USUBJID, AESEQ, AETERM, AESER, ASTDT,         TREM01FL, TREM02FL, TREM03FL, TRTEMFL)write_csv(adae %>% mutate(ASTDT = format(ASTDT, "%Y-%m-%d")),          file.path(output_dir, "adae.csv"), na = "")

solutions/adam-adae-phase-emergence/result.py

Edit
104 lines
# Reference solution for the yamaa benchmark adam-adae-phase-emergence (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1."""Derive ADAE with phase-specific treatment-emergent flags from ADSL and AE."""from datetime import date, timedeltafrom pathlib import Pathimport polars as plINPUT = Path("/app/input")OUTPUT = Path("/app/output")PHASES = [("TR01SDT", "TR01EDT"), ("TR02SDT", "TR02EDT"), ("TR03SDT", "TR03EDT")]FLAGS = ["TREM01FL", "TREM02FL", "TREM03FL"]COLUMNS = ["STUDYID", "USUBJID", "AESEQ", "AETERM", "AESER", "ASTDT",           "TREM01FL", "TREM02FL", "TREM03FL", "TRTEMFL"]WINDOW = {"N": 30, "Y": 90}def to_date(value):    """Return a date for a complete ISO 8601 date, otherwise None."""    if value is None:        return None    text = str(value).strip()    if len(text) < 10:        return None    try:        return date.fromisoformat(text[:10])    except ValueError:        return Nonedef phase_windows(subject, aeser):    """List (phase index, start, end) for the phases a subject entered; end None = open."""    entered = []    for index, (start_col, end_col) in enumerate(PHASES):        start = to_date(subject.get(start_col))        if start is not None:            entered.append((index, start, to_date(subject.get(end_col))))    windows = []    for position, (index, start, last) in enumerate(entered):        if position + 1 < len(entered):            end = entered[position + 1][1] - timedelta(days=1)        elif str(subject.get("EOTSTT") or "").strip().upper() == "ONGOING":            end = None        elif last is None:            end = None        else:            end = last + timedelta(days=WINDOW.get(aeser, 30))        windows.append((index, start, end))    return windowsdef derive_flags(subject, astdt, aeser):    flags = [None, None, None]    if subject is None or astdt is None:        return flags    if to_date(subject.get("TR01SDT")) is None:        return flags    for index, start, end in phase_windows(subject, aeser):        if astdt >= start and (end is None or astdt <= end):            flags[index] = "Y"            break    return flagsdef main():    # Every column is read as text; an empty field is null.    adsl = pl.read_csv(INPUT / "adsl.csv", infer_schema=False)    ae = pl.read_csv(INPUT / "ae.csv", infer_schema=False)    subjects = {row["USUBJID"]: row for row in adsl.to_dicts()}    records = []    for row in ae.to_dicts():        astdt = to_date(row.get("AESTDTC"))        aeser = (row.get("AESER") or "").strip().upper()        flags = derive_flags(subjects.get(row["USUBJID"]), astdt, aeser)        records.append({            "STUDYID": row["STUDYID"],            "USUBJID": row["USUBJID"],            "AESEQ": int(row["AESEQ"]),            "AETERM": row["AETERM"],            "AESER": row["AESER"],            "ASTDT": astdt.isoformat() if astdt else None,            "TREM01FL": flags[0],            "TREM02FL": flags[1],            "TREM03FL": flags[2],            "TRTEMFL": "Y" if "Y" in flags else None,        })    schema = {name: pl.Utf8 for name in COLUMNS} | {"AESEQ": pl.Int64}    adae = pl.DataFrame(records, schema=schema).sort(        ["STUDYID", "USUBJID", "AESEQ"], maintain_order=True    )    OUTPUT.mkdir(parents=True, exist_ok=True)    adae.write_csv(OUTPUT / "adae.csv")if __name__ == "__main__":    main()

solutions/adam-adae-phase-emergence/result.sas

Edit
86 lines
/* opensas reference solution for the yamaa benchmark adam-adae-phase-emergence.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data adsl;  length STUDYID USUBJID TR01SDT TR01EDT TR02SDT TR02EDT TR03SDT TR03EDT EOTSTT $1024;  infile "/app/input/adsl.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. TR01SDT : $char1024. TR01EDT : $char1024.        TR02SDT : $char1024. TR02EDT : $char1024. TR03SDT : $char1024. TR03EDT : $char1024.        EOTSTT : $char1024.;run;data ae;  length STUDYID USUBJID AETERM AESER AESTDTC $1024;  infile "/app/input/ae.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. AESEQ AETERM : $char1024. AESER : $char1024.        AESTDTC : $char1024.;run;proc sql;  create table joined as  select a.*, b.TR01SDT, b.TR01EDT, b.TR02SDT, b.TR02EDT, b.TR03SDT, b.TR03EDT, b.EOTSTT  from ae as a left join adsl as b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID;quit;data events;  length ASTDT $10 TREM01FL TREM02FL TREM03FL TRTEMFL $1;  set joined;  ASTDT=AESTDTC; TREM01FL=''; TREM02FL=''; TREM03FL=''; TRTEMFL='';  onset=.; s1=.; e1=.; s2=.; e2=.; s3=.; e3=.;  if not missing(AESTDTC) then onset=input(AESTDTC, yymmdd10.);  if not missing(TR01SDT) then s1=input(TR01SDT, yymmdd10.);  if not missing(TR01EDT) then e1=input(TR01EDT, yymmdd10.);  if not missing(TR02SDT) then s2=input(TR02SDT, yymmdd10.);  if not missing(TR02EDT) then e2=input(TR02EDT, yymmdd10.);  if not missing(TR03SDT) then s3=input(TR03SDT, yymmdd10.);  if not missing(TR03EDT) then e3=input(TR03EDT, yymmdd10.);  /* Phase 1 ends the day before the next phase the subject entered starts:     surgery, or the first adjuvant dose for a subject who had no surgery. */  next1=s2;  if missing(next1) then next1=s3;  /* The last phase a subject entered ends 30 days after its last date for a     non-serious event and 90 days after it for a serious one, both days     included, and has no end while treatment is ongoing. */  lastend=e1;  if not missing(s3) then lastend=e3;  else if not missing(s2) then lastend=e2;  tail=.;  if AESER='Y' then tail=90;  else if AESER='N' then tail=30;  inlast=0;  if EOTSTT='ONGOING' then inlast=1;  else if not missing(onset) and not missing(lastend) and not missing(tail) then do;    if onset<=lastend+tail then inlast=1;  end;  /* SAS missing dates sort below every date, so each phase start is tested     for presence before the onset is compared with it. An untreated subject     has no phase start and so no flag. */  if not missing(onset) and not missing(s1) then do;    if onset>=s1 then do;      if not missing(next1) then do;        if onset<next1 then TREM01FL='Y';      end;      else if inlast=1 then TREM01FL='Y';    end;  end;  if not missing(onset) and not missing(s2) then do;    if onset>=s2 then do;      if not missing(s3) then do;        if onset<s3 then TREM02FL='Y';      end;      else if inlast=1 then TREM02FL='Y';    end;  end;  if not missing(onset) and not missing(s3) then do;    if onset>=s3 and inlast=1 then TREM03FL='Y';  end;  if TREM01FL='Y' or TREM02FL='Y' or TREM03FL='Y' then TRTEMFL='Y';run;proc sql;  create table final as  select STUDYID, USUBJID, AESEQ, AETERM, AESER, ASTDT, TREM01FL, TREM02FL, TREM03FL, TRTEMFL  from events;quit;proc export data=final outfile="/app/output/adae.csv" dbms=csv replace; run;