Review ask

As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.

We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.

Summary

Lifecycle: reviewedEditUnder review
What this benchmark means
README.md

Public entries: Python and R. Passing coverage requires an installed inventory.

Goal: one row per adverse event (AE) with an analysis start date (ASTDT) completed from the collected start at whatever precision it was collected, a flag (ASTDTF) marking the dates whose day was supplied, and a treatment emergence flag (TRTEMFL).

Input: adverse event records carrying the reported term (AETERM) and the collected start (AESTDTC): a full date, a year and month, a year alone, non-date text, or nothing at all, plus the first-exposure date (TRTSDT) from the subject-level analysis dataset (ADSL), empty when the subject has no ADSL record.

Variables:

  • ASTDT is the analysis start date. A fully collected date is used as it stands. A year and month is completed to the 15th. A year alone, non-date text, and a missing value give no analysis date.
  • ASTDTC is the same analysis date written as text, empty when there is no analysis date.
  • ASTDTF is D when the day was supplied to complete the date. It is empty when the date was collected in full and when no analysis date could be formed, so the flag always matches the date shown. In CDISC terms only the day was imputed, which is the level of imputation the flag records.
  • TRTEMFL is Y when the analysis start falls on or after TRTSDT. It stays empty when there is no analysis date, when the event started before first exposure, or when the subject's first-exposure date is unknown. A supplied day counts exactly as a collected one would, so ASTDTF tells a reader which flagged events rested on a supplied day.

Note: a supplied day is never placed before a first exposure that falls inside the collected month. When the 15th would fall before TRTSDT, the date moves forward to the exposure date, which the collected month still allows. When the whole collected month ends before the exposure date, no day it allows could reach it, so the date stays on the 15th rather than moving into a month nobody recorded, and the event is not treatment emergent. A fully collected date is left exactly as collected even when it falls before TRTSDT, because there is nothing about it to choose.

Inputs

Under review
2 source files

input/adsl.csv

Edit
3 rows
input/adsl.csv
STUDYIDUSUBJIDTRTSDT
CATH2025-01-10
CATH2025-03-20
CATH2025-02-15

input/ae.csv

Edit
13 rows
input/ae.csv
STUDYIDUSUBJIDAESEQAETERMAESTDTC
CATH1HEADACHE2025-01-15
CATH2NAUSEA2025-01
CATH3ECZEMA FLARE2025
CATH4DIZZINESSUNKNOWN
CATH5FATIGUE—
CATH6RASH2025-02-20
CATH1PYREXIA2025-03
CATH2COUGH2025-02
CATH3INSOMNIA2025-04-05
CATH4MIGRAINE2025-01-05
CATH7FLUSHING2025-01-10
CATH1BACK PAIN2025-04
CATH1VOMITING2025-02

Expected output

Under review
13 expected rows

expected/adae.csv

Edit
13 rows
expected/adae.csv
STUDYIDUSUBJIDAESEQAETERMAESTDTCASTDTASTDTCASTDTFTRTSDTTRTEMFL
CATH1HEADACHE2025-01-152025-01-152025-01-15—2025-01-10Y
CATH2NAUSEA2025-012025-01-152025-01-15D2025-01-10Y
CATH3ECZEMA FLARE2025———2025-01-10—
CATH4DIZZINESSUNKNOWN———2025-01-10—
CATH5FATIGUE————2025-01-10—
CATH6RASH2025-02-202025-02-202025-02-20—2025-01-10Y
CATH1PYREXIA2025-032025-03-202025-03-20D2025-03-20Y
CATH2COUGH2025-022025-02-152025-02-15D2025-03-20—
CATH3INSOMNIA2025-04-052025-04-052025-04-05—2025-03-20Y
CATH4MIGRAINE2025-01-052025-01-052025-01-05—2025-03-20—
CATH7FLUSHING2025-01-102025-01-102025-01-10—2025-01-10Y
CATH1BACK PAIN2025-042025-04-152025-04-15D——
CATH1VOMITING2025-022025-02-152025-02-15D2025-02-15Y
Shaded columns are derived— = missing value

Human review and assessment

Your review

3 questions and your comments · View benchmark discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Dataset NameVariable NameVariable LabelTypeCodelist/Controlled TermsCoreComputational MethodOrigin
ADAESTUDYIDStudy IdentifiercharacterCopy value from AE.STUDYID.Predecessor
ADAEUSUBJIDUnique Subject IdentifiercharacterCopy value from AE.USUBJID.Predecessor
ADAEAESEQSequence NumbernumericCopy value from AE.AESEQ.Predecessor
ADAEAETERMReported Term for the Adverse EventcharacterCopy value from AE.AETERM.Predecessor
ADAEAESTDTCStart Date/Time of Adverse EventcharacterCopy value from AE.AESTDTC.Predecessor
ADAETRTSDTDate of First Exposure to TreatmentnumericISO 8601Copy value from ADSL.TRTSDT.Predecessor
ADAEASTDTAnalysis Start DatenumericISO 8601Impute incomplete date from AESTDTC, defaulting day=15, never before TRTSDT; a bound outside the collected period is ignored; a year alone gives blank; invalid text → blank.Derived
ADAEASTDTCAnalysis Start Date (C)characterCopy value from ASTDT.Derived
ADAEASTDTFAnalysis Start Date Imputation FlagcharacterDDate imputation flag for ASTDT: "Y" when the whole date was imputed, "M" when the month and day were, "D" when only the day was; blank when the date was collected in full.Derived
ADAETRTEMFLTreatment Emergent Analysis FlagcharacterY"Y" when ASTDT >= TRTSDT; blank otherwise.Derived
VariablePermitted values
ASTDTFD
TRTEMFLY
VersionDateAuthorDescriptionReviewerSign-off
1.72026-10-20yamaa docs buildRead a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default
1.62026-10-17yamaa docs buildDescribe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight
1.52026-10-17yamaa docs buildName collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part
1.42026-10-15yamaa docs buildState datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/adam-adae-partial-dates/full.md

Edit
27 lines

Following CDISC ADaM standards, use the provided ADSL and AE datasets to create an ADAE dataset with one record per adverse event.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, AESEQ, AETERM, AESTDTC, ASTDT, ASTDTC, ASTDTF, TRTSDT, TRTEMFL

ASTDT is the collected start date as it stands when AESTDTC is a full date. A full date is left exactly as collected even when it falls before TRTSDT. When AESTDTC holds a year and a month only, ASTDT is the 15th of that month. When that 15th falls before TRTSDT and TRTSDT falls in the same month, ASTDT is TRTSDT instead. When the whole of that month falls before TRTSDT, ASTDT stays on the 15th. ASTDT has no value when AESTDTC holds a year only, holds text that is not a date, or has no value. When TRTSDT has no value, a year and a month is still completed to the 15th. ASTDTC is ASTDT written as text in the form "YYYY-MM-DD", and ASTDTC has no value when ASTDT has no value. ASTDTF is "D" when the day of ASTDT was supplied to complete a year and a month, and ASTDTF has no value when AESTDTC is a full date or when ASTDT has no value. TRTEMFL is "Y" when ASTDT falls on or after TRTSDT. A supplied day counts for TRTEMFL exactly as a collected day would. TRTEMFL has no value when ASTDT has no value, when ASTDT falls before TRTSDT, or when TRTSDT has no value.

Read the source datasets from /app/input and save the completed dataset as /app/output/adae.csv.

prompts/adam-adae-partial-dates/conventions.md

Edit
12 lines

Following CDISC ADaM standards, use the provided ADSL and AE datasets to create an ADAE dataset with one record per adverse event.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, AESEQ, AETERM, AESTDTC, ASTDT, ASTDTC, ASTDTF, TRTSDT, TRTEMFL

ASTDTC is ASTDT written as text in the form "YYYY-MM-DD" or has no value. ASTDTF is "D" or has no value. TRTEMFL is "Y" or has no value.

Read the source datasets from /app/input and save the completed dataset as /app/output/adae.csv.

prompts/adam-adae-partial-dates/brief.md

Edit
9 lines

Following CDISC ADaM standards, use the provided ADSL and AE datasets to create an ADAE dataset with one record per adverse event.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, AESEQ, AETERM, AESTDTC, ASTDT, ASTDTC, ASTDTF, TRTSDT, TRTEMFL

Read the source datasets from /app/input and save the completed dataset as /app/output/adae.csv.

yamaa code

Public calls generated from benchmark.yaml; results must match the expected output

R

Edit
3 lines
library(yamaa)yamaa_spec("spec.yaml")$run()$output

Python

Edit
3 lines
import yamaayamaa.spec("spec.yaml").run().output

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/adam-adae-partial-dates/result.R

Edit
64 lines
# Reference solution for the yamaa benchmark adam-adae-partial-dates (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(lubridate, warn.conflicts = FALSE)library(readr)library(stringr)ae <- read_csv(  "/app/input/ae.csv",  col_types = cols(AESEQ = col_integer(), .default = col_character()))adsl <- read_csv(  "/app/input/adsl.csv",  col_types = cols(TRTSDT = col_date(), .default = col_character())) |>  select(USUBJID, TRTSDT)adae <- ae |>  left_join(adsl, by = "USUBJID") |>  mutate(    is_full = str_detect(AESTDTC, "^\\d{4}-\\d{2}-\\d{2}$"),    is_month = str_detect(AESTDTC, "^\\d{4}-\\d{2}$"),    # A year and month is completed to the 15th.    completed = if_else(      is_month,      ymd(paste0(AESTDTC, "-15"), quiet = TRUE),      ymd(AESTDTC, quiet = TRUE)    ),    completed = if_else(is_full | is_month, completed, as.Date(NA)),    # A supplied day is never placed before a first exposure that falls    # inside the collected month. When the whole collected month ends    # before the exposure date, no day it allows could reach it, so the    # date stays on the 15th.    month_end = if_else(      is_month,      completed + days(days_in_month(completed) - 15L),      completed    ),    month_before_trt = is_month &      !is.na(month_end) & !is.na(TRTSDT) & month_end < TRTSDT,    ASTDT = case_when(      month_before_trt ~ completed,      is_month & !is.na(completed) & !is.na(TRTSDT) & completed < TRTSDT ~ TRTSDT,      .default = completed    ),    ASTDTC = if_else(is.na(ASTDT), NA_character_, as.character(ASTDT)),    ASTDTF = if_else(is_month & !is.na(ASTDT), "D", NA_character_),    TRTEMFL = if_else(      !is.na(ASTDT) & !is.na(TRTSDT) & ASTDT >= TRTSDT,      "Y",      NA_character_    )  ) |>  select(    STUDYID, USUBJID, AESEQ, AETERM, AESTDTC, ASTDT, ASTDTC, ASTDTF,    TRTSDT, TRTEMFL  )dir.create("/app/output", showWarnings = FALSE)write_csv(adae, "/app/output/adae.csv", na = "")

solutions/adam-adae-partial-dates/result.py

Edit
110 lines
# Reference solution for the yamaa benchmark adam-adae-partial-dates (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1."""Build ADAE from ADSL and AE, completing partial AE start dates.ASTDT is the adverse event start date derived from the collectedAESTDTC character date:- a full date is kept exactly as collected, even before treatment start;- a year and month only is completed to the 15th of that month, except  that when the 15th falls before TRTSDT within the same month ASTDT is  TRTSDT; when the whole month falls before TRTSDT the 15th stands;- a year only, non-date text, or a missing value gives no ASTDT.ASTDTF is "D" when a day was supplied to complete a year and month.TRTEMFL is "Y" when ASTDT falls on or after TRTSDT."""import csvimport osimport refrom datetime import dateINPUT_DIR = "/app/input"OUTPUT_DIR = "/app/output"FULL_DATE = re.compile(r"^(\d{4})-(\d{2})-(\d{2})$")YEAR_MONTH = re.compile(r"^(\d{4})-(\d{2})$")def parse_full_date(text):    """Return a date for a strict YYYY-MM-DD string, else None."""    match = FULL_DATE.match(text or "")    if not match:        return None    try:        return date(int(match.group(1)), int(match.group(2)), int(match.group(3)))    except ValueError:        return Nonedef derive_astdt(aestdtc, trtsdt):    """Return (ASTDT as a date or None, day-imputed flag)."""    aestdtc = (aestdtc or "").strip()    full = parse_full_date(aestdtc)    if full is not None:        return full, False    match = YEAR_MONTH.match(aestdtc)    if match:        year, month = int(match.group(1)), int(match.group(2))        if not 1 <= month <= 12:            return None, False        candidate = date(year, month, 15)        if trtsdt is None or candidate >= trtsdt:            return candidate, True        if trtsdt.year == year and trtsdt.month == month:            return trtsdt, True        # The whole month falls before TRTSDT, so the 15th stands.        return candidate, True    return None, Falsedef main():    with open(os.path.join(INPUT_DIR, "adsl.csv"), newline="") as handle:        adsl = list(csv.DictReader(handle))    with open(os.path.join(INPUT_DIR, "ae.csv"), newline="") as handle:        ae = list(csv.DictReader(handle))    trtsdt_by_subject = {}    for row in adsl:        trtsdt_by_subject[row["USUBJID"]] = parse_full_date(row.get("TRTSDT"))    columns = [        "STUDYID", "USUBJID", "AESEQ", "AETERM", "AESTDTC",        "ASTDT", "ASTDTC", "ASTDTF", "TRTSDT", "TRTEMFL",    ]    out_rows = []    for row in ae:        trtsdt = trtsdt_by_subject.get(row["USUBJID"])        astdt, day_imputed = derive_astdt(row.get("AESTDTC"), trtsdt)        astdtc = astdt.isoformat() if astdt is not None else ""        astdtf = "D" if (day_imputed and astdt is not None) else ""        trtemfl = ""        if astdt is not None and trtsdt is not None and astdt >= trtsdt:            trtemfl = "Y"        out_rows.append({            "STUDYID": row["STUDYID"],            "USUBJID": row["USUBJID"],            "AESEQ": row["AESEQ"],            "AETERM": row["AETERM"],            "AESTDTC": row["AESTDTC"],            "ASTDT": astdt.isoformat() if astdt is not None else "",            "ASTDTC": astdtc,            "ASTDTF": astdtf,            "TRTSDT": trtsdt.isoformat() if trtsdt is not None else "",            "TRTEMFL": trtemfl,        })    os.makedirs(OUTPUT_DIR, exist_ok=True)    with open(os.path.join(OUTPUT_DIR, "adae.csv"), "w", newline="") as handle:        writer = csv.DictWriter(handle, fieldnames=columns)        writer.writeheader()        writer.writerows(out_rows)if __name__ == "__main__":    main()

solutions/adam-adae-partial-dates/result.sas

Edit
41 lines
/* opensas reference solution for the yamaa benchmark adam-adae-partial-dates.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data adsl;  length STUDYID USUBJID TRTSDT $1024;  infile "/app/input/adsl.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. TRTSDT : $char1024.;run;data ae;  length STUDYID USUBJID AETERM AESTDTC $1024;  infile "/app/input/ae.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. AESEQ AETERM : $char1024. AESTDTC : $char1024.;run;proc sql; create table joined as select a.*,b.TRTSDT from ae as a left join adsl as b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID; quit;data result;  length ASTDT ASTDTC $10 ASTDTF TRTEMFL $1;  set joined;  treatment=input(TRTSDT,yymmdd10.); analysis=.;  if prxmatch('/^[0-9]{4}-[0-9]{2}-[0-9]{2}$/',strip(AESTDTC)) then analysis=input(AESTDTC,yymmdd10.);  else if prxmatch('/^[0-9]{4}-[0-9]{2}$/',strip(AESTDTC)) then do;    analysis=input(cats(AESTDTC,'-15'),yymmdd10.);    if not missing(analysis) then do;      /* First exposure moves the 15th forward only when it falls inside the         collected month; when it falls after the whole month, the 15th stands. */      if missing(treatment) or treatment<=intnx('month',analysis,0,'end') then analysis=max(analysis,treatment);      ASTDTF='D';    end;  end;  if not missing(analysis) then do;    ASTDT=put(analysis,yymmdd10.); ASTDTC=ASTDT;    if not missing(treatment) and analysis>=treatment then TRTEMFL='Y';  end;run;proc sql;  create table final as select STUDYID, USUBJID, AESEQ, AETERM, AESTDTC, ASTDT, ASTDTC, ASTDTF, TRTSDT, TRTEMFL from result;quit;proc export data=final outfile="/app/output/adae.csv" dbms=csv replace; run;