As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.
We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.
Summary
README.mdPublic entries: Python and R. Passing coverage requires an installed inventory.
Goal: carry on each concomitant medication record, named by CMTRT and
identified by its sequence number CMSEQ, the adverse events it was given
for: their sequence numbers (AESEQS), their distinct reported terms
(AETERMS) and how many distinct terms there are (AETERMN).
Input: collected concomitant medications (CM), adverse events (AE) and related records (RELREC), which tie a medication to the adverse events it was given for through a shared relationship identifier.
Variables:
- In RELAE, one record per adverse event a relationship names:
RELIDis the relationship, andAETERMthe event's reported term, read from the subject's adverse event record with that sequence number (AESEQ). AESEQSlists the sequence numbers of the medication's related adverse events in increasing numeric order, separated by a comma and a space; blank when the medication is related to no adverse event.AETERMSlists the reported terms of those events in alphabetical order, each term once, separated by a comma and a space; blank when there is none.AETERMNis the number of different termsAETERMSlists; blank, not zero, when the medication is related to no adverse event.
Note: a relationship may also tie the medication to records other than adverse events, such as medical history, and no list includes those. Sequence numbers sort as numbers, so 2 comes before 10.
Inputs
Under review3 source filesinput/ae.csv
Edit6 rows| STUDYID | USUBJID | AESEQ | AETERM |
|---|---|---|---|
| YAMAA-01 | 1 | NAUSEA | |
| YAMAA-01 | 2 | HEADACHE | |
| YAMAA-01 | 3 | BACK PAIN | |
| YAMAA-01 | 10 | HEADACHE | |
| YAMAA-01 | 1 | HEADACHE | |
| YAMAA-01 | 4 | FEVER |
input/cm.csv
Edit4 rows| STUDYID | USUBJID | CMSEQ | CMTRT |
|---|---|---|---|
| YAMAA-01 | 1 | PARACETAMOL | |
| YAMAA-01 | 2 | IBUPROFEN | |
| YAMAA-01 | 3 | LORATADINE | |
| YAMAA-01 | 1 | PARACETAMOL |
input/relrec.csv
Edit9 rows| STUDYID | RDOMAIN | USUBJID | IDVAR | IDVARVAL | RELTYPE | RELID |
|---|---|---|---|---|---|---|
| YAMAA-01 | CM | CMSEQ | 1 | — | 1 | |
| YAMAA-01 | AE | AESEQ | 10 | — | 1 | |
| YAMAA-01 | AE | AESEQ | 2 | — | 1 | |
| YAMAA-01 | CM | CMSEQ | 2 | — | 2 | |
| YAMAA-01 | AE | AESEQ | 3 | — | 2 | |
| YAMAA-01 | MH | MHSEQ | 1 | — | 2 | |
| YAMAA-01 | CM | CMSEQ | 1 | — | 1 | |
| YAMAA-01 | AE | AESEQ | 4 | — | 1 | |
| YAMAA-01 | AE | AESEQ | 1 | — | 1 |
Expected output
Under review9 expected rowsexpected/adcm.csv
Edit4 rows| STUDYID | USUBJID | CMSEQ | CMTRT | AESEQS | AETERMS | AETERMN |
|---|---|---|---|---|---|---|
| YAMAA-01 | 1 | PARACETAMOL | 2, 10 | HEADACHE | 1 | |
| YAMAA-01 | 2 | IBUPROFEN | 3 | BACK PAIN | 1 | |
| YAMAA-01 | 3 | LORATADINE | — | — | — | |
| YAMAA-01 | 1 | PARACETAMOL | 1, 4 | FEVER, HEADACHE | 2 |
expected/relae.csv
Edit5 rows| STUDYID | USUBJID | RELID | AESEQ | AETERM |
|---|---|---|---|---|
| YAMAA-01 | 1 | 2 | HEADACHE | |
| YAMAA-01 | 1 | 10 | HEADACHE | |
| YAMAA-01 | 2 | 3 | BACK PAIN | |
| YAMAA-01 | 1 | 1 | HEADACHE | |
| YAMAA-01 | 1 | 4 | FEVER |
Human review and assessment
| Dataset Name | Variable Name | Variable Label | Type | Codelist/Controlled Terms | Core | Computational Method | Origin |
|---|---|---|---|---|---|---|---|
| ADCM | STUDYID | Study Identifier | character | Copy value from CM.STUDYID. | Predecessor | ||
| ADCM | USUBJID | Unique Subject Identifier | character | Copy value from CM.USUBJID. | Predecessor | ||
| ADCM | CMSEQ | Sequence Number | numeric | Copy value from CM.CMSEQ. | Predecessor | ||
| ADCM | CMTRT | Reported Name of Drug, Med, or Therapy | character | Copy value from CM.CMTRT. | Predecessor | ||
| ADCM | AESEQS | Related Adverse Event Sequence Numbers | character | Aggregate STR_JOIN(SEQN) over records whose STUDYID equals STUDYID, USUBJID equals USUBJID and RELID equals RELID, where RELREC.RDOMAIN = 'AE' AND RELREC.IDVAR = 'AESEQ', with SEQN as RELREC.IDVARVAL. | Derived | ||
| ADCM | AETERMS | Related Adverse Event Terms | character | Aggregate STR_JOIN(RELAE.AETERM) over records whose STUDYID equals STUDYID, USUBJID equals USUBJID and RELID equals RELID. | Derived | ||
| ADCM | AETERMN | Number of Distinct Related Adverse Event Terms | numeric | Aggregate COUNT(RELAE.AETERM) over records whose STUDYID equals STUDYID, USUBJID equals USUBJID and RELID equals RELID. | Derived |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.7 | 2026-10-20 | yamaa docs build | Read a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default | ||
| 1.6 | 2026-10-17 | yamaa docs build | Describe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight | ||
| 1.5 | 2026-10-17 | yamaa docs build | Name collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part | ||
| 1.4 | 2026-10-15 | yamaa docs build | State datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks | ||
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/adam-adcm-related-events/full.md
Edit38 linesFollowing CDISC ADaM standards, use the provided AE, CM, and RELREC datasets to create a RELAE dataset with one record per adverse event a relationship names and an ADCM dataset with one record per concomitant medication record.
The RELAE dataset should contain the following columns in this order: STUDYID, USUBJID, RELID, AESEQ, AETERM The ADCM dataset should contain the following columns in this order: STUDYID, USUBJID, CMSEQ, CMTRT, AESEQS, AETERMS, AETERMN
RELREC ties a medication to the adverse events it was given for through a shared relationship identifier, RELID. A relationship belongs to one subject: the same RELID in two subjects names two different relationships. A RELREC record names a medication when RDOMAIN is "CM" and IDVAR is "CMSEQ", and an adverse event when RDOMAIN is "AE" and IDVAR is "AESEQ", with the sequence number in IDVARVAL.
RELAE:
- RELAE holds one record for each adverse event a relationship names, and none for a related record of any other domain, such as medical history.
- RELID is the relationship, AESEQ the event's sequence number, and AETERM the reported term of the subject's adverse event record with that AESEQ.
ADCM:
- A medication's related adverse events are the adverse events named by the subject's relationship that names the medication's CMSEQ; no list includes related records of other domains.
- AESEQS lists the sequence numbers of the related adverse events in increasing numeric order, written as whole numbers and separated by a comma and a space (", "). Sequence numbers sort as numbers, so 2 comes before 10.
- AETERMS lists the reported terms of those events in alphabetical order, each term once, separated by a comma and a space (", ").
- AETERMN is the number of different terms AETERMS lists.
- When a medication is related to no adverse event, AESEQS, AETERMS and AETERMN have no value; AETERMN is blank, not zero.
Read the source datasets from /app/input and save the completed RELAE dataset as /app/output/relae.csv and the completed ADCM dataset as /app/output/adcm.csv.
prompts/adam-adcm-related-events/conventions.md
Edit17 linesFollowing CDISC ADaM standards, use the provided AE, CM, and RELREC datasets to create a RELAE dataset with one record per adverse event a relationship names and an ADCM dataset with one record per concomitant medication record.
The RELAE dataset should contain the following columns in this order: STUDYID, USUBJID, RELID, AESEQ, AETERM The ADCM dataset should contain the following columns in this order: STUDYID, USUBJID, CMSEQ, CMTRT, AESEQS, AETERMS, AETERMN
AESEQS lists sequence numbers in increasing numeric order, written as whole numbers, and AETERMS lists terms in alphabetical order, each term once, both separated by a comma and a space (", "). AESEQS, AETERMS and AETERMN may have no value; AETERMN is then blank, not zero.
Read the source datasets from /app/input and save the completed RELAE dataset as /app/output/relae.csv and the completed ADCM dataset as /app/output/adcm.csv.
prompts/adam-adcm-related-events/brief.md
Edit12 linesFollowing CDISC ADaM standards, use the provided AE, CM, and RELREC datasets to create a RELAE dataset with one record per adverse event a relationship names and an ADCM dataset with one record per concomitant medication record.
The RELAE dataset should contain the following columns in this order: STUDYID, USUBJID, RELID, AESEQ, AETERM The ADCM dataset should contain the following columns in this order: STUDYID, USUBJID, CMSEQ, CMTRT, AESEQS, AETERMS, AETERMN
Read the source datasets from /app/input and save the completed RELAE dataset as /app/output/relae.csv and the completed ADCM dataset as /app/output/adcm.csv.
R
Edit4 lineslibrary(yamaa)yamaa_spec("spec_relae.yaml")$run()$save()yamaa_spec("spec_adcm.yaml")$run()$outputPython
Edit4 linesimport yamaayamaa.spec("spec_relae.yaml").run().save()yamaa.spec("spec_adcm.yaml").run().outputsolutions/adam-adcm-related-events/result.R
Edit78 lines# Reference solution for the yamaa benchmark adam-adcm-related-events (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.read_text <- function(name) { read.csv( file.path("/app/input", name), colClasses = "character", na.strings = character(0), check.names = FALSE )}ae <- read_text("ae.csv")cm <- read_text("cm.csv")relrec <- read_text("relrec.csv")# Each adverse event a relationship names, with the event's reported term.events <- relrec[relrec$RDOMAIN == "AE" & relrec$IDVAR == "AESEQ", ]relae <- data.frame( STUDYID = events$STUDYID, USUBJID = events$USUBJID, RELID = events$RELID, AESEQ = as.integer(events$IDVARVAL), stringsAsFactors = FALSE)term_key <- paste(ae$STUDYID, ae$USUBJID, as.integer(ae$AESEQ), sep = "\r")relae$AETERM <- ae$AETERM[match( paste(relae$STUDYID, relae$USUBJID, relae$AESEQ, sep = "\r"), term_key)]relae <- relae[order(relae$STUDYID, relae$USUBJID, relae$RELID, relae$AESEQ, method = "radix"), ]# The relationship each medication belongs to.links <- relrec[relrec$RDOMAIN == "CM" & relrec$IDVAR == "CMSEQ", ]relid <- links$RELID[match( paste(cm$STUDYID, cm$USUBJID, cm$CMSEQ, sep = "\r"), paste(links$STUDYID, links$USUBJID, links$IDVARVAL, sep = "\r"))]aeseqs <- character(nrow(cm))aeterms <- character(nrow(cm))aetermn <- integer(nrow(cm))for (i in seq_len(nrow(cm))) { related <- relae[ !is.na(relid[i]) & relae$STUDYID == cm$STUDYID[i] & relae$USUBJID == cm$USUBJID[i] & relae$RELID == relid[i], ] if (nrow(related) == 0) { aeseqs[i] <- NA_character_ aeterms[i] <- NA_character_ aetermn[i] <- NA_integer_ } else { names <- sort(unique(related$AETERM), method = "radix") aeseqs[i] <- paste(sort(related$AESEQ), collapse = ", ") aeterms[i] <- paste(names, collapse = ", ") aetermn[i] <- length(names) }}adcm <- data.frame( STUDYID = cm$STUDYID, USUBJID = cm$USUBJID, CMSEQ = as.integer(cm$CMSEQ), CMTRT = cm$CMTRT, AESEQS = aeseqs, AETERMS = aeterms, AETERMN = aetermn, stringsAsFactors = FALSE)adcm <- adcm[order(adcm$STUDYID, adcm$USUBJID, adcm$CMSEQ, method = "radix"), ]dir.create("/app/output", showWarnings = FALSE, recursive = TRUE)write.csv(relae, "/app/output/relae.csv", row.names = FALSE, na = "")write.csv(adcm, "/app/output/adcm.csv", row.names = FALSE, na = "")solutions/adam-adcm-related-events/result.py
Edit97 lines# Reference solution for the yamaa benchmark adam-adcm-related-events (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.import csvfrom pathlib import PathINPUT_DIR = Path("/app/input")OUTPUT_DIR = Path("/app/output")RELAE_COLUMNS = ["STUDYID", "USUBJID", "RELID", "AESEQ", "AETERM"]ADCM_COLUMNS = [ "STUDYID", "USUBJID", "CMSEQ", "CMTRT", "AESEQS", "AETERMS", "AETERMN",]def read(name: str) -> list[dict[str, str]]: with open(INPUT_DIR / name, newline="", encoding="utf-8") as handle: return list(csv.DictReader(handle))def write(name: str, columns: list[str], rows: list[dict]) -> None: OUTPUT_DIR.mkdir(parents=True, exist_ok=True) with open(OUTPUT_DIR / name, "w", newline="", encoding="utf-8") as handle: writer = csv.writer(handle, lineterminator="\n") writer.writerow(columns) for row in rows: writer.writerow(["" if row[c] is None else row[c] for c in columns])def main() -> None: ae = read("ae.csv") cm = read("cm.csv") relrec = read("relrec.csv") # Each adverse event a relationship names, with the event's reported term. terms = {(r["STUDYID"], r["USUBJID"], int(r["AESEQ"])): r["AETERM"] for r in ae} relae = [] for r in relrec: if r["RDOMAIN"] == "AE" and r["IDVAR"] == "AESEQ": seq = int(r["IDVARVAL"]) relae.append( { "STUDYID": r["STUDYID"], "USUBJID": r["USUBJID"], "RELID": r["RELID"], "AESEQ": seq, "AETERM": terms[(r["STUDYID"], r["USUBJID"], seq)], } ) relae.sort(key=lambda r: (r["STUDYID"], r["USUBJID"], r["RELID"], r["AESEQ"])) # The relationship each medication belongs to. relationship = { (r["STUDYID"], r["USUBJID"], r["IDVARVAL"]): r["RELID"] for r in relrec if r["RDOMAIN"] == "CM" and r["IDVAR"] == "CMSEQ" } adcm = [] for r in cm: relid = relationship.get((r["STUDYID"], r["USUBJID"], r["CMSEQ"])) related = [ event for event in relae if relid is not None and (event["STUDYID"], event["USUBJID"], event["RELID"]) == (r["STUDYID"], r["USUBJID"], relid) ] sequences = sorted(event["AESEQ"] for event in related) names = sorted({event["AETERM"] for event in related}) adcm.append( { "STUDYID": r["STUDYID"], "USUBJID": r["USUBJID"], "CMSEQ": int(r["CMSEQ"]), "CMTRT": r["CMTRT"], "AESEQS": ", ".join(str(s) for s in sequences) or None, "AETERMS": ", ".join(names) or None, "AETERMN": len(names) or None, } ) adcm.sort(key=lambda r: (r["STUDYID"], r["USUBJID"], r["CMSEQ"])) write("relae.csv", RELAE_COLUMNS, relae) write("adcm.csv", ADCM_COLUMNS, adcm)if __name__ == "__main__": main()solutions/adam-adcm-related-events/result.sas
Edit119 lines/* opensas reference solution for the yamaa benchmark adam-adcm-related-events. Reads the benchmark inputs and derives the requested outputs. Execute with the existing opensas runtime: opensas /app/output/result.sas. */data ae; length STUDYID USUBJID $40 AETERM $200; infile "/app/input/ae.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char40. USUBJID : $char40. AESEQ AETERM : $char200.;run;data cm; length STUDYID USUBJID $40 CMTRT $200; infile "/app/input/cm.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char40. USUBJID : $char40. CMSEQ CMTRT : $char200.;run;data relrec; length STUDYID RDOMAIN USUBJID IDVAR IDVARVAL RELTYPE RELID $40; infile "/app/input/relrec.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char40. RDOMAIN : $char40. USUBJID : $char40. IDVAR : $char40. IDVARVAL : $char40. RELTYPE : $char40. RELID : $char40.;run;/* The medications and the adverse events each relationship names, with their numeric sequence numbers. Function calls in join predicates were rejected by the release, so the keys are calculated before joining. */data relcm; set relrec; if RDOMAIN = 'CM' and IDVAR = 'CMSEQ'; CMSEQ = input(IDVARVAL, best32.); keep STUDYID USUBJID RELID CMSEQ;run;data relaeseq; set relrec; if RDOMAIN = 'AE' and IDVAR = 'AESEQ'; AESEQ = input(IDVARVAL, best32.); keep STUDYID USUBJID RELID AESEQ;run;proc sql; /* Each adverse event a relationship names, with its reported term. */ create table relae as select r.STUDYID, r.USUBJID, r.RELID, r.AESEQ, a.AETERM from relaeseq as r left join ae as a on a.STUDYID = r.STUDYID and a.USUBJID = r.USUBJID and a.AESEQ = r.AESEQ order by r.STUDYID, r.USUBJID, r.RELID, r.AESEQ; /* The relationship each medication belongs to. */ create table cmrel as select c.STUDYID, c.USUBJID, c.CMSEQ, c.CMTRT, r.RELID from cm as c left join relcm as r on r.STUDYID = c.STUDYID and r.USUBJID = c.USUBJID and r.CMSEQ = c.CMSEQ; create table pairs as select m.STUDYID, m.USUBJID, m.CMSEQ, e.AESEQ, e.AETERM from cmrel as m inner join relae as e on e.STUDYID = m.STUDYID and e.USUBJID = m.USUBJID and e.RELID = m.RELID;quit;/* The related sequence numbers in numeric order. */proc sort data=pairs out=byseq; by STUDYID USUBJID CMSEQ AESEQ;run;data seqs; length AESEQS $1000; retain AESEQS; set byseq; by STUDYID USUBJID CMSEQ; if first.CMSEQ then AESEQS = ''; AESEQS = catx(', ', AESEQS, strip(put(AESEQ, best32.))); if last.CMSEQ then output; keep STUDYID USUBJID CMSEQ AESEQS;run;/* The different related terms in alphabetical order, and their number. */data termpairs; set pairs; keep STUDYID USUBJID CMSEQ AETERM;run;proc sort data=termpairs out=byterm nodupkey; by STUDYID USUBJID CMSEQ AETERM;run;data terms; length AETERMS $1000; retain AETERMS AETERMN; set byterm; by STUDYID USUBJID CMSEQ; if first.CMSEQ then do; AETERMS = ''; AETERMN = 0; end; AETERMS = catx(', ', AETERMS, AETERM); AETERMN = AETERMN + 1; if last.CMSEQ then output; keep STUDYID USUBJID CMSEQ AETERMS AETERMN;run;proc sql; create table adcm as select m.STUDYID, m.USUBJID, m.CMSEQ, m.CMTRT, s.AESEQS, t.AETERMS, t.AETERMN from cmrel as m left join seqs as s on s.STUDYID = m.STUDYID and s.USUBJID = m.USUBJID and s.CMSEQ = m.CMSEQ left join terms as t on t.STUDYID = m.STUDYID and t.USUBJID = m.USUBJID and t.CMSEQ = m.CMSEQ order by m.STUDYID, m.USUBJID, m.CMSEQ;quit;proc export data=relae outfile='/app/output/relae.csv' dbms=csv replace; run;proc export data=adcm outfile='/app/output/adcm.csv' dbms=csv replace; run;
yamaa