yamaa As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.
We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.
Summary
EditUnder reviewWhat this benchmark meansREADME.mdPublic entries: Python and R. Passing coverage requires an installed inventory.
Goal: relate each medication to the adverse events it was given
for, carrying IDVARVAL, RELTYPE, and RELID.
Input: adverse event (AE) records with their assigned sequence
number (AESEQ) and the event number collected on the form
(AESPID), and concomitant medication (CM) records with their
assigned sequence number (CMSEQ) and up to two event numbers
entered on the medication form as the events it treated (CMAENO1,
CMAENO2). The link is collected on the medication form only.
Variables:
IDVARVALis the sequence number of the related record as text:CMSEQon a medication row and, on an event row, theAESEQof the same subject's adverse event whose collected event number matches, never the collected number itself.RELTYPEis blank throughout, because each row points at one record rather than a whole dataset.RELIDnumbers the subject's medication and event pairs 1, 2, 3 and so on, in adverse event sequence order and then medication sequence order; the two rows of one pair share it.
Note: each event number on a medication record gives one relationship of two rows, one for the medication and one for the event, so a medication given for two events takes part in two relationships, as does an event treated by two medications. Event numbers match within the same subject only. A medication without an event number, and an event no medication names, contribute no row. An event number that names no adverse event of that subject stops the run, and so does a medication repeating one event number in both fields, which would state the same relationship twice.
Inputs
Under review2 source filesExpected output
Under review10 expected rowsexpected/relrec.csv
Edit10 rows| STUDYID | USUBJID | RDOMAIN | IDVAR | IDVARVAL | RELTYPE | RELID |
|---|---|---|---|---|---|---|
| CATH | AE | AESEQ | 1 | — | 1 | |
| CATH | CM | CMSEQ | 1 | — | 1 | |
| CATH | AE | AESEQ | 1 | — | 2 | |
| CATH | CM | CMSEQ | 3 | — | 2 | |
| CATH | AE | AESEQ | 2 | — | 3 | |
| CATH | CM | CMSEQ | 2 | — | 3 | |
| CATH | AE | AESEQ | 2 | — | 4 | |
| CATH | CM | CMSEQ | 3 | — | 4 | |
| CATH | AE | AESEQ | 1 | — | 1 | |
| CATH | CM | CMSEQ | 1 | — | 1 |
Human review and assessment
| Variable Name | Variable Label | Type | Length | Controlled Terms or Format | Origin | Core | Conversion Definition | Variable Type | Variable Order | Comments for Define |
|---|---|---|---|---|---|---|---|---|---|---|
| STUDYID | Study Identifier | Char | Collected | Copy value from CM.STUDYID. | SDTM | 1 | ||||
| USUBJID | Unique Subject Identifier | Char | Collected | Copy value from CM.USUBJID. | SDTM | 2 | ||||
| RDOMAIN | Related Domain Abbreviation | Char | Derived | Set per constructed row; see the Row construction sheet. | SDTM | 3 | ||||
| IDVAR | Identifying Variable | Char | Derived | Set per constructed row; see the Row construction sheet. | SDTM | 4 | ||||
| RELTYPE | Relationship Type | Char | Assigned | Set constant value "None". | SDTM | 5 | ||||
| IDVARVAL | Identifying Variable Value | Char | Derived | Set per constructed row; see the Row construction sheet. | SDTM | 6 | ||||
| RELID | Relationship Identifier | Char | Derived | Copy value from RELNUM. | SDTM | 7 |
| Template | Include when | RDOMAIN | IDVAR | IDVARVAL | PAIRAE |
|---|---|---|---|---|---|
| cm_reference_1 | CM.CMAENO1 IS NOT NULL | Constant "CM" | Constant "CMSEQ" | Copy CM.CMSEQ | If CM.CMAENO1 IS NOT NULL then "AE_REF1.AESEQ"; otherwise blank |
| ae_reference_1 | CM.CMAENO1 IS NOT NULL | Constant "AE" | Constant "AESEQ" | Copy PAIRAE | If CM.CMAENO1 IS NOT NULL then "AE_REF1.AESEQ"; otherwise blank |
| cm_reference_2 | CM.CMAENO2 IS NOT NULL | Constant "CM" | Constant "CMSEQ" | Copy CM.CMSEQ | If CM.CMAENO2 IS NOT NULL then "AE_REF2.AESEQ"; otherwise blank |
| ae_reference_2 | CM.CMAENO2 IS NOT NULL | Constant "AE" | Constant "AESEQ" | Copy PAIRAE | If CM.CMAENO2 IS NOT NULL then "AE_REF2.AESEQ"; otherwise blank |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.7 | 2026-10-20 | yamaa docs build | Read a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default | ||
| 1.6 | 2026-10-17 | yamaa docs build | Describe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight | ||
| 1.5 | 2026-10-17 | yamaa docs build | Name collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part | ||
| 1.4 | 2026-10-15 | yamaa docs build | State datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks | ||
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/sdtm-relrec-links/full.md
Edit23 linesFollowing CDISC SDTM standards, use the provided AE and CM datasets to create a RELREC dataset with one record per relationship participation.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, RDOMAIN, IDVAR, IDVARVAL, RELTYPE, RELID
The link between a medication and an adverse event is collected on the medication form only: CMAENO1 and CMAENO2 hold up to two adverse event numbers, each naming the AE record of the same subject whose AESPID equals it. Each event number on a CM record gives one relationship of two rows: a row for the CM record, with RDOMAIN "CM", IDVAR "CMSEQ" and IDVARVAL its CMSEQ, and a row for the AE record it names, with RDOMAIN "AE", IDVAR "AESEQ" and IDVARVAL that AE record's AESEQ, not the event number. IDVARVAL is the sequence number as text. A medication given for two events takes part in two relationships, and an event treated by two medications likewise. A CM record with no event number, and an AE record no medication names, give no row. RELTYPE has no value on any row, because each row points at one record rather than a whole dataset. RELID numbers the relationships within each subject 1, 2, 3 and so on, ordered by the AESEQ of the event and then the CMSEQ of the medication; the two rows of one relationship share it.
Read the source datasets from /app/input and save the completed dataset as /app/output/relrec.csv.
prompts/sdtm-relrec-links/conventions.md
Edit13 linesFollowing CDISC SDTM standards, use the provided AE and CM datasets to create a RELREC dataset with one record per relationship participation.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, RDOMAIN, IDVAR, IDVARVAL, RELTYPE, RELID
IDVAR is "AESEQ" or "CMSEQ". RELTYPE has no value on any row. RELID numbers the relationships within each subject 1, 2, 3 and so on, ordered by the AESEQ of the event and then the CMSEQ of the medication; the two rows of one relationship share it.
Read the source datasets from /app/input and save the completed dataset as /app/output/relrec.csv.
prompts/sdtm-relrec-links/brief.md
Edit8 linesFollowing CDISC SDTM standards, use the provided AE and CM datasets to create a RELREC dataset with one record per relationship participation.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, RDOMAIN, IDVAR, IDVARVAL, RELTYPE, RELID
Read the source datasets from /app/input and save the completed dataset as /app/output/relrec.csv.
R
Edit3 lineslibrary(yamaa)yamaa_spec("spec.yaml")$run()$outputPython
Edit3 linesimport yamaayamaa.spec("spec.yaml").run().outputsolutions/sdtm-relrec-links/result.R
Edit66 lines# Reference solution for the yamaa benchmark sdtm-relrec-links (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)library(tidyr)ae <- read_csv( "/app/input/ae.csv", col_types = cols(.default = col_character()))cm <- read_csv( "/app/input/cm.csv", col_types = cols(.default = col_character()))ae_numbers <- ae |> transmute( STUDYID, USUBJID, AENO = trimws(AESPID), AESEQ = as.integer(AESEQ) )# Each event number on a medication record names the same subject's AE# record with that AESPID; every number must name one.pairs <- cm |> pivot_longer( cols = c(CMAENO1, CMAENO2), values_to = "AENO" ) |> mutate(AENO = trimws(AENO)) |> filter(!is.na(AENO) & AENO != "") |> inner_join( ae_numbers, by = c("STUDYID", "USUBJID", "AENO"), unmatched = c("error", "drop"), relationship = "many-to-one" ) |> mutate(CMSEQ = as.integer(CMSEQ)) |> arrange(STUDYID, USUBJID, AESEQ, CMSEQ) |> group_by(STUDYID, USUBJID) |> mutate(RELID = as.character(row_number())) |> ungroup()ae_rows <- pairs |> transmute( STUDYID, USUBJID, RDOMAIN = "AE", IDVAR = "AESEQ", IDVARVAL = as.character(AESEQ), RELTYPE = NA_character_, RELID )cm_rows <- pairs |> transmute( STUDYID, USUBJID, RDOMAIN = "CM", IDVAR = "CMSEQ", IDVARVAL = as.character(CMSEQ), RELTYPE = NA_character_, RELID )relrec <- bind_rows(ae_rows, cm_rows) |> arrange(USUBJID, as.integer(RELID), RDOMAIN) |> select(STUDYID, USUBJID, RDOMAIN, IDVAR, IDVARVAL, RELTYPE, RELID)dir.create("/app/output", showWarnings = FALSE)write_csv(relrec, "/app/output/relrec.csv", na = "")solutions/sdtm-relrec-links/result.py
Edit80 lines# Reference solution for the yamaa benchmark sdtm-relrec-links (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1."""Build a RELREC dataset from the AE numbers entered on CM records.Each event number in CMAENO1/CMAENO2 names the same subject's AE recordwith that AESPID. Every such medication-event pair gives two RELREC rows,one for the CM record and one for the AE record, sharing a RELID thatnumbers the subject's pairs by AESEQ and then CMSEQ."""import csvimport osINPUT_DIR = "/app/input"OUTPUT_DIR = "/app/output"COLUMNS = ["STUDYID", "USUBJID", "RDOMAIN", "IDVAR", "IDVARVAL", "RELTYPE", "RELID"]def read_rows(name): with open(os.path.join(INPUT_DIR, name), newline="", encoding="utf-8") as handle: return list(csv.DictReader(handle))def clean(value): return "" if value is None else value.strip()def main(): ae = read_rows("ae.csv") cm = read_rows("cm.csv") aeseq_by_number = {} for record in ae: key = (record["STUDYID"], record["USUBJID"], clean(record["AESPID"])) aeseq_by_number[key] = int(record["AESEQ"]) pairs = {} for record in cm: subject = (record["STUDYID"], record["USUBJID"]) cmseq = int(record["CMSEQ"]) for var in ("CMAENO1", "CMAENO2"): number = clean(record.get(var)) if not number: continue aeseq = aeseq_by_number[subject + (number,)] pairs.setdefault(subject, []).append((aeseq, cmseq)) rows = [] for subject in sorted(pairs): studyid, usubjid = subject for relid, (aeseq, cmseq) in enumerate(sorted(pairs[subject]), 1): for rdomain, idvar, seq in (("AE", "AESEQ", aeseq), ("CM", "CMSEQ", cmseq)): rows.append({ "STUDYID": studyid, "USUBJID": usubjid, "RDOMAIN": rdomain, "IDVAR": idvar, "IDVARVAL": str(seq), "RELTYPE": "", "RELID": str(relid), }) os.makedirs(OUTPUT_DIR, exist_ok=True) out_path = os.path.join(OUTPUT_DIR, "relrec.csv") with open(out_path, "w", newline="", encoding="utf-8") as handle: writer = csv.DictWriter(handle, fieldnames=COLUMNS) writer.writeheader() writer.writerows(rows)if __name__ == "__main__": main()solutions/sdtm-relrec-links/result.sas
Edit53 lines/* opensas reference solution for the yamaa benchmark sdtm-relrec-links. Reads the benchmark inputs and derives the requested output. Execute with the existing opensas runtime: opensas /app/output/result.sas. */data ae; length STUDYID USUBJID AESPID AETERM $1024; infile "/app/input/ae.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. AESEQ AESPID : $char1024. AETERM : $char1024.;run;data cm; length STUDYID USUBJID CMTRT CMAENO1 CMAENO2 $1024; infile "/app/input/cm.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. CMSEQ CMTRT : $char1024. CMAENO1 : $char1024. CMAENO2 : $char1024.;run;data refs; set cm; length AENO $1024;if not missing(CMAENO1) then do; AENO=strip(CMAENO1); output; end;if not missing(CMAENO2) then do; AENO=strip(CMAENO2); output; end;keep STUDYID USUBJID CMSEQ AENO;run;proc sql; create table matched as select r.STUDYID, r.USUBJID, a.AESEQ, r.CMSEQ, a.AESPID as MATCHED_AENO from refs as r left join ae as a on r.STUDYID=a.STUDYID and r.USUBJID=a.USUBJID and r.AENO=strip(a.AESPID) order by r.STUDYID, r.USUBJID, a.AESEQ, r.CMSEQ;quit;/* A populated reference must name an event before any output is written. */data pairs; set matched;if missing(MATCHED_AENO) then abort return 1;drop MATCHED_AENO;run;data numbered; set pairs; by STUDYID USUBJID;if first.USUBJID then RELNUM=0;RELNUM+1;run;data result; set numbered;length RDOMAIN IDVAR IDVARVAL RELTYPE RELID $1024;RELTYPE=''; RELID=strip(put(RELNUM,best32.));RDOMAIN='AE'; IDVAR='AESEQ'; IDVARVAL=strip(put(AESEQ,best32.)); output;RDOMAIN='CM'; IDVAR='CMSEQ'; IDVARVAL=strip(put(CMSEQ,best32.)); output;run;proc sql; create table final as select STUDYID, USUBJID, RDOMAIN, IDVAR, IDVARVAL, RELTYPE, RELID from result order by STUDYID, USUBJID, RELNUM, RDOMAIN;quit;proc export data=final outfile="/app/output/relrec.csv" dbms=csv replace; run;