yamaa As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.
We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.
Summary
EditUnder reviewWhat this benchmark meansREADME.mdPublic entries: Python and R. Passing coverage requires an installed inventory.
Goal: identify the single governing intercurrent event in each subject's completed event history for an Analysis Data Model (ADaM) occurrence dataset.
Input: subject identifiers and collected intercurrent event records with event sequence, type and complete event date.
Variables:
ASEQ,ADECOD1andASTDTidentify the source sequence, type and date.ICEPRIOis 1 for ICE04 and ICE05, 2 for ICE01, 3 for ICE02 and 4 for ICE03: death or kidney replacement, prohibited medication, start of other protocol-prohibited therapy, and treatment discontinuation, respectively.ICERANKnumbers each subject's events from 1 by increasing priority, then earliest date, then lowest source sequence.GOVFLis Y on position 1 and empty elsewhere;SUPERFLis Y on every other event and empty on position 1.GOVTYPE,GOVDTandGOVSEQare the governing type, date and source sequence on every event of the subject.
Note: priority outranks chronology across event types. Equal priority uses the earliest date, then lowest sequence. A subject without events has no record. This is a completed-history summary; it does not choose the event applicable at an earlier efficacy assessment.
Inputs
Under review2 source filesinput/adsl.csv
Edit5 rows| STUDYID | USUBJID |
|---|---|
| YAMAA-01 | |
| YAMAA-01 | |
| YAMAA-01 | |
| YAMAA-01 | |
| YAMAA-01 |
input/ice_raw.csv
Edit10 rows| STUDYID | USUBJID | ICESEQ | ICETYPE | ICEDTC |
|---|---|---|---|---|
| YAMAA-01 | 1 | ICE02 | 2025-01-12 | |
| YAMAA-01 | 1 | ICE02 | 2025-01-03 | |
| YAMAA-01 | 2 | ICE01 | 2025-01-09 | |
| YAMAA-01 | 3 | ICE04 | 2025-01-28 | |
| YAMAA-01 | 1 | ICE01 | 2025-01-14 | |
| YAMAA-01 | 2 | ICE01 | 2025-01-07 | |
| YAMAA-01 | 3 | ICE02 | 2025-01-02 | |
| YAMAA-01 | 1 | ICE05 | 2025-01-20 | |
| YAMAA-01 | 2 | ICE04 | 2025-01-20 | |
| YAMAA-01 | 3 | ICE03 | 2025-01-01 |
Expected output
Under review10 expected rowsexpected/adice.csv
Edit10 rows| STUDYID | USUBJID | ASEQ | ADECOD1 | ASTDT | ICEPRIO | ICERANK | GOVFL | SUPERFL | GOVTYPE | GOVDT | GOVSEQ |
|---|---|---|---|---|---|---|---|---|---|---|---|
| YAMAA-01 | 1 | ICE02 | 2025-01-12 | 3 | 1 | Y | — | ICE02 | 2025-01-12 | 1 | |
| YAMAA-01 | 1 | ICE02 | 2025-01-03 | 3 | 3 | — | Y | ICE04 | 2025-01-28 | 3 | |
| YAMAA-01 | 2 | ICE01 | 2025-01-09 | 2 | 2 | — | Y | ICE04 | 2025-01-28 | 3 | |
| YAMAA-01 | 3 | ICE04 | 2025-01-28 | 1 | 1 | Y | — | ICE04 | 2025-01-28 | 3 | |
| YAMAA-01 | 1 | ICE01 | 2025-01-14 | 2 | 2 | — | Y | ICE01 | 2025-01-07 | 2 | |
| YAMAA-01 | 2 | ICE01 | 2025-01-07 | 2 | 1 | Y | — | ICE01 | 2025-01-07 | 2 | |
| YAMAA-01 | 3 | ICE02 | 2025-01-02 | 3 | 3 | — | Y | ICE01 | 2025-01-07 | 2 | |
| YAMAA-01 | 1 | ICE05 | 2025-01-20 | 1 | 1 | Y | — | ICE05 | 2025-01-20 | 1 | |
| YAMAA-01 | 2 | ICE04 | 2025-01-20 | 1 | 2 | — | Y | ICE05 | 2025-01-20 | 1 | |
| YAMAA-01 | 3 | ICE03 | 2025-01-01 | 4 | 3 | — | Y | ICE05 | 2025-01-20 | 1 |
Human review and assessment
| Dataset Name | Variable Name | Variable Label | Type | Codelist/Controlled Terms | Core | Computational Method | Origin |
|---|---|---|---|---|---|---|---|
| ADICE | STUDYID | Study Identifier | character | Copy value from ICE_RAW.STUDYID. | Predecessor | ||
| ADICE | USUBJID | Unique Subject Identifier | character | Copy value from ICE_RAW.USUBJID. | Predecessor | ||
| ADICE | ASEQ | Analysis Sequence Number | numeric | Copy value from ICE_RAW.ICESEQ. | Predecessor | ||
| ADICE | ADECOD1 | Analysis Dictionary Derived Term | character | Copy value from ICE_RAW.ICETYPE. | Predecessor | ||
| ADICE | ASTDT | Analysis Start Date | numeric | ISO 8601 | Copy value from ICE_RAW.ICEDTC. | Predecessor | |
| ADICE | ICEPRIO | Intercurrent Event Type Priority | numeric | Recode ADECOD1: "ICE04" → "1", "ICE05" → "1", "ICE01" → "2", "ICE02" → "3", "ICE03" → "4"; missing values stay missing; unlisted values are errors. | Derived | ||
| ADICE | ICERANK | Intercurrent Event Priority Position | numeric | Row number using window ICE_ORDER. | Derived | ||
| ADICE | GOVFL | Governing Intercurrent Event Flag | character | "Y" when ICERANK = 1; blank otherwise. | Derived | ||
| ADICE | SUPERFL | Superseded Intercurrent Event Flag | character | "Y" when ICERANK > 1; blank otherwise. | Derived | ||
| ADICE | GOVTYPE | Governing Intercurrent Event Type | character | Take ICETYPE from the first ICE_RAW record matching on STUDYID and USUBJID, ordered by ICE_RAW.PRIORITY then ICE_RAW.ICEDTC then ICE_RAW.ICESEQ, and a row with no match is an error. | Derived | ||
| ADICE | GOVDT | Governing Intercurrent Event Date | numeric | ISO 8601 | Take ICEDTC from the first ICE_RAW record matching on STUDYID and USUBJID, ordered by ICE_RAW.PRIORITY then ICE_RAW.ICEDTC then ICE_RAW.ICESEQ, and a row with no match is an error. | Derived | |
| ADICE | GOVSEQ | Governing Intercurrent Event Sequence | numeric | Take ICESEQ from the first ICE_RAW record matching on STUDYID and USUBJID, ordered by ICE_RAW.PRIORITY then ICE_RAW.ICEDTC then ICE_RAW.ICESEQ, and a row with no match is an error. | Derived |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.7 | 2026-10-20 | yamaa docs build | Read a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default | ||
| 1.6 | 2026-10-17 | yamaa docs build | Describe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight | ||
| 1.5 | 2026-10-17 | yamaa docs build | Name collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part | ||
| 1.4 | 2026-10-15 | yamaa docs build | State datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks | ||
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/adam-adice-governing-event/full.md
Edit23 linesFollowing CDISC ADaM standards, use the provided ADSL and ICE_RAW datasets to create an ADICE dataset with one record per intercurrent event.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, ASEQ, ADECOD1, ASTDT, ICEPRIO, ICERANK, GOVFL, SUPERFL, GOVTYPE, GOVDT, GOVSEQ
ASEQ, ADECOD1, and ASTDT give the event's source sequence, type, and complete date. ICEPRIO is 1 for ICE04 and ICE05 (death or kidney replacement), 2 for ICE01 (prohibited medication), 3 for ICE02 (start of other protocol-prohibited therapy), and 4 for ICE03 (treatment discontinuation). ICERANK numbers each subject's events from 1 by increasing priority, then earliest date, then lowest source sequence; priority outranks chronology across event types. GOVFL is Y on position 1 and has no value elsewhere. SUPERFL is Y on every other event and has no value on position 1. GOVTYPE, GOVDT, and GOVSEQ are the governing event's type, date, and source sequence, on every event of the subject. A subject without events has no record. The governing event is chosen over the subject's completed event history; it is not the event applicable at an earlier efficacy assessment.
Read the source datasets from /app/input and save the completed dataset as /app/output/adice.csv.
prompts/adam-adice-governing-event/conventions.md
Edit16 linesFollowing CDISC ADaM standards, use the provided ADSL and ICE_RAW datasets to create an ADICE dataset with one record per intercurrent event.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, ASEQ, ADECOD1, ASTDT, ICEPRIO, ICERANK, GOVFL, SUPERFL, GOVTYPE, GOVDT, GOVSEQ
ICEPRIO is 1 for ICE04 and ICE05 (death or kidney replacement), 2 for ICE01 (prohibited medication), 3 for ICE02 (start of other protocol-prohibited therapy), and 4 for ICE03 (treatment discontinuation). ICERANK numbers each subject's events from 1 by increasing priority, then earliest date, then lowest source sequence; priority outranks chronology across event types. GOVFL is Y or has no value. SUPERFL is Y or has no value.
Read the source datasets from /app/input and save the completed dataset as /app/output/adice.csv.
prompts/adam-adice-governing-event/brief.md
Edit9 linesFollowing CDISC ADaM standards, use the provided ADSL and ICE_RAW datasets to create an ADICE dataset with one record per intercurrent event.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, ASEQ, ADECOD1, ASTDT, ICEPRIO, ICERANK, GOVFL, SUPERFL, GOVTYPE, GOVDT, GOVSEQ
Read the source datasets from /app/input and save the completed dataset as /app/output/adice.csv.
R
Edit3 lineslibrary(yamaa)yamaa_spec("spec.yaml")$run()$outputPython
Edit3 linesimport yamaayamaa.spec("spec.yaml").run().outputsolutions/adam-adice-governing-event/result.R
Edit23 linesevents <- read.csv("/app/input/ice_raw.csv", stringsAsFactors = FALSE)priority <- c(ICE04 = 1L, ICE05 = 1L, ICE01 = 2L, ICE02 = 3L, ICE03 = 4L)events$ICEPRIO <- unname(priority[events$ICETYPE])events <- events[order(events$STUDYID, events$USUBJID, events$ICEPRIO, events$ICEDTC, events$ICESEQ), ]groups <- split(events, interaction(events$STUDYID, events$USUBJID, drop = TRUE))result <- do.call(rbind, lapply(groups, function(group) { group$ICERANK <- seq_len(nrow(group)) group$GOVFL <- ifelse(group$ICERANK == 1L, "Y", NA_character_) group$SUPERFL <- ifelse(group$ICERANK > 1L, "Y", NA_character_) group$GOVTYPE <- group$ICETYPE[1] group$GOVDT <- group$ICEDTC[1] group$GOVSEQ <- group$ICESEQ[1] group}))names(result)[names(result) == "ICESEQ"] <- "ASEQ"names(result)[names(result) == "ICETYPE"] <- "ADECOD1"names(result)[names(result) == "ICEDTC"] <- "ASTDT"columns <- c("STUDYID", "USUBJID", "ASEQ", "ADECOD1", "ASTDT", "ICEPRIO", "ICERANK", "GOVFL", "SUPERFL", "GOVTYPE", "GOVDT", "GOVSEQ")result <- result[order(result$STUDYID, result$USUBJID, result$ASEQ), columns]write.csv(result, "/app/output/adice.csv", row.names = FALSE, na = "")solutions/adam-adice-governing-event/result.py
Edit83 lines# Reference solution for the yamaa benchmark adam-adice-governing-event (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.import csvINPUT_DIR = "/app/input"OUTPUT_PATH = "/app/output/adice.csv"PRIORITY = { "ICE04": 1, "ICE05": 1, "ICE01": 2, "ICE02": 3, "ICE03": 4,}COLUMNS = [ "STUDYID", "USUBJID", "ASEQ", "ADECOD1", "ASTDT", "ICEPRIO", "ICERANK", "GOVFL", "SUPERFL", "GOVTYPE", "GOVDT", "GOVSEQ",]def main(): with open(f"{INPUT_DIR}/adsl.csv", newline="") as f: adsl = list(csv.DictReader(f)) subjects = {(r["STUDYID"], r["USUBJID"]) for r in adsl} with open(f"{INPUT_DIR}/ice_raw.csv", newline="") as f: ice = list(csv.DictReader(f)) events = {} order = [] for row in ice: key = (row["STUDYID"], row["USUBJID"]) if key not in subjects: continue event = { "STUDYID": row["STUDYID"], "USUBJID": row["USUBJID"], "ASEQ": int(row["ICESEQ"]), "ADECOD1": row["ICETYPE"], "ASTDT": row["ICEDTC"], "ICEPRIO": PRIORITY[row["ICETYPE"]], } events.setdefault(key, []).append(event) order.append(event) for key, subject_events in events.items(): ranked = sorted( subject_events, key=lambda e: (e["ICEPRIO"], e["ASTDT"], e["ASEQ"]), ) governing = ranked[0] for position, event in enumerate(ranked, start=1): event["ICERANK"] = position event["GOVFL"] = "Y" if position == 1 else "" event["SUPERFL"] = "" if position == 1 else "Y" event["GOVTYPE"] = governing["ADECOD1"] event["GOVDT"] = governing["ASTDT"] event["GOVSEQ"] = governing["ASEQ"] with open(OUTPUT_PATH, "w", newline="") as f: writer = csv.DictWriter(f, fieldnames=COLUMNS) writer.writeheader() for event in order: writer.writerow({c: event[c] for c in COLUMNS})if __name__ == "__main__": main()solutions/adam-adice-governing-event/result.sas
Edit37 lines/* opensas reference: select priority before date and source sequence. */data events; length STUDYID USUBJID $80 ADECOD1 $8 date_text $10; infile "/app/input/ice_raw.csv" dsd dlm="," firstobs=2 truncover; input STUDYID : $80. USUBJID : $80. ASEQ ADECOD1 : $8. date_text : $10.; ASTDT=input(date_text,yymmdd10.); if ADECOD1 in ('ICE04','ICE05') then ICEPRIO=1; else if ADECOD1='ICE01' then ICEPRIO=2; else if ADECOD1='ICE02' then ICEPRIO=3; else if ADECOD1='ICE03' then ICEPRIO=4;run;proc sort data=events; by STUDYID USUBJID ICEPRIO ASTDT ASEQ;run;data ranked; length GOVFL SUPERFL $1 GOVTYPE $8; set events; by STUDYID USUBJID; retain ICERANK GOVTYPE GOVDT GOVSEQ; if first.USUBJID then do; ICERANK=0; GOVTYPE=ADECOD1; GOVDT=ASTDT; GOVSEQ=ASEQ; end; ICERANK+1; if ICERANK=1 then GOVFL='Y'; else SUPERFL='Y'; format ASTDT GOVDT yymmdd10.;run;proc sql; create table result as select STUDYID, USUBJID, ASEQ, ADECOD1, put(ASTDT,yymmdd10.) as ASTDT, ICEPRIO, ICERANK, GOVFL, SUPERFL, GOVTYPE, put(GOVDT,yymmdd10.) as GOVDT, GOVSEQ from ranked order by STUDYID, USUBJID, ASEQ;quit;proc export data=result outfile="/app/output/adice.csv" dbms=csv replace;run;