As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.
We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.
Summary
README.mdPublic entries: Python and R. Passing coverage requires an installed inventory.
Goal: derive one record per collected adverse event, keeping
the reported term (AETERM) and adding the coded preferred term
(AEDECOD) and body system (AEBODSYS).
Input: collected adverse-event rows carrying the reported term, plus a Medical Dictionary for Regulatory Activities (MedDRA) extract carrying the lowest-level, preferred, and body-system names.
Variables:
AETERMis the reported term in upper case, so letter case never decides whether a term codes.AEDECODis the preferred term of the dictionary entry whose lowest-level term matches the reported term.AEBODSYSis the body system of that same entry.
Note: a reported term codes when it equals a lowest-level term,
whatever the letter case of either; the preferred term comes from
the matched dictionary entry, so it can differ from the reported
term. A term with no match gives NOT CODED in both coded
variables. NOT CODED is not a MedDRA term: it marks an event whose
coding must be resolved before delivery. The coded terms follow the
recorded dictionary, MedDRA version 26.1, since the same reported
term can code differently between releases.
Inputs
Under review2 source filesinput/ae_raw.csv
Edit7 rows| STUDYID | USUBJID | AESEQ | AETERM |
|---|---|---|---|
| YAMAA-01 | 1 | Headache | |
| YAMAA-01 | 2 | Nausea | |
| YAMAA-01 | 1 | Dizziness | |
| YAMAA-01 | 1 | Cephalgia | |
| YAMAA-01 | 1 | Queasy | |
| YAMAA-01 | 1 | Felt a bit off | |
| YAMAA-01 | 1 | headache |
input/meddra_26_1.csv
Edit5 rows| LLTNAME | PTNAME | SOCNAME |
|---|---|---|
| Headache | Headache | Nervous system disorders |
| Cephalgia | Headache | Nervous system disorders |
| Nausea | Nausea | Gastrointestinal disorders |
| Queasy | Nausea | Gastrointestinal disorders |
| Dizziness | Dizziness | Nervous system disorders |
Expected output
Under review7 expected rowsexpected/ae.csv
Edit7 rows| DOMAIN | STUDYID | USUBJID | AESEQ | AETERM | AEDECOD | AEBODSYS |
|---|---|---|---|---|---|---|
| AE | YAMAA-01 | 1 | HEADACHE | Headache | Nervous system disorders | |
| AE | YAMAA-01 | 2 | NAUSEA | Nausea | Gastrointestinal disorders | |
| AE | YAMAA-01 | 1 | DIZZINESS | Dizziness | Nervous system disorders | |
| AE | YAMAA-01 | 1 | CEPHALGIA | Headache | Nervous system disorders | |
| AE | YAMAA-01 | 1 | QUEASY | Nausea | Gastrointestinal disorders | |
| AE | YAMAA-01 | 1 | FELT A BIT OFF | NOT CODED | NOT CODED | |
| AE | YAMAA-01 | 1 | HEADACHE | Headache | Nervous system disorders |
Human review and assessment
| Variable Name | Variable Label | Type | Length | Controlled Terms or Format | Origin | Core | Conversion Definition | Variable Type | Variable Order | Comments for Define |
|---|---|---|---|---|---|---|---|---|---|---|
| DOMAIN | Domain Abbreviation | Char | 2 | Assigned | Req | Set constant value "AE". | SDTM | 1 | ||
| STUDYID | Study Identifier | Char | 20 | Protocol | Req | Copy value from AE_RAW.STUDYID. | SDTM | 2 | ||
| USUBJID | Unique Subject Identifier | Char | 30 | Assigned | Req | Copy value from AE_RAW.USUBJID. | SDTM | 3 | ||
| AESEQ | Sequence Number | Num | 8 | Assigned | Req | Copy value from AE_RAW.AESEQ. | SDTM | 4 | ||
| AETERM | Reported Term for the Adverse Event | Char | 200 | Collected | Req | Convert AE_RAW.AETERM to upper case. | SDTM | 5 | ||
| AEDECOD | Dictionary-Derived Term | Char | 200 | Medical Dictionary for Regulatory Activities (MedDRA, version 26.1) | Assigned | Req | Take PTNAME from the MEDDRA record whose LLTNAME, in upper case, equals AETERM, and no match yields "NOT CODED". | SDTM | 6 | |
| AEBODSYS | Body System or Organ Class | Char | 200 | Medical Dictionary for Regulatory Activities (MedDRA, version 26.1) | Assigned | Exp | Take SOCNAME from the MEDDRA record whose LLTNAME, in upper case, equals AETERM, and no match yields "NOT CODED". | SDTM | 7 |
| Codelist ID | Name | Value | Decode | Codelist NCI alias | Value NCI alias | Extensible | Extended value | SAS format | Dictionary | Version |
|---|---|---|---|---|---|---|---|---|---|---|
| MEDDRA | Medical Dictionary for Regulatory Activities | No | MedDRA | 26.1 |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.7 | 2026-10-20 | yamaa docs build | Read a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default | ||
| 1.6 | 2026-10-17 | yamaa docs build | Describe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight | ||
| 1.5 | 2026-10-17 | yamaa docs build | Name collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part | ||
| 1.4 | 2026-10-15 | yamaa docs build | State datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks | ||
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/sdtm-ae-coding/full.md
Edit19 linesFollowing CDISC SDTM standards, use the provided AE_RAW and MEDDRA_26_1 datasets to create an AE dataset with one record per adverse event.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, AESEQ, AETERM, AEDECOD, AEBODSYS
DOMAIN is "AE". AETERM is the reported term in upper case. AEDECOD is the preferred term of the MedDRA dictionary entry whose lowest-level term equals AETERM when both are compared in upper case. AEBODSYS is the body system of that same entry. The preferred term comes from the matched dictionary entry, so AEDECOD can differ from AETERM. A reported term with no match gives "NOT CODED" in both AEDECOD and AEBODSYS. "NOT CODED" is not a MedDRA term: it marks an event whose coding must be resolved before delivery. The coded terms follow the recorded dictionary, MedDRA version 26.1, since the same reported term can code differently between releases.
Read the source datasets from /app/input and save the completed dataset as /app/output/ae.csv.
prompts/sdtm-ae-coding/conventions.md
Edit12 linesFollowing CDISC SDTM standards, use the provided AE_RAW and MEDDRA_26_1 datasets to create an AE dataset with one record per adverse event.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, AESEQ, AETERM, AEDECOD, AEBODSYS
DOMAIN is "AE". AETERM is in upper case. AEDECOD and AEBODSYS can be "NOT CODED". The coded terms follow the recorded dictionary, MedDRA version 26.1.
Read the source datasets from /app/input and save the completed dataset as /app/output/ae.csv.
prompts/sdtm-ae-coding/brief.md
Edit8 linesFollowing CDISC SDTM standards, use the provided AE_RAW and MEDDRA_26_1 datasets to create an AE dataset with one record per adverse event.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, AESEQ, AETERM, AEDECOD, AEBODSYS
Read the source datasets from /app/input and save the completed dataset as /app/output/ae.csv.
R
Edit3 lineslibrary(yamaa)yamaa_project("project.yaml")$spec("AE")$run()$outputPython
Edit3 linesimport yamaayamaa.project("project.yaml").spec("AE").run().outputsolutions/sdtm-ae-coding/result.R
Edit36 lines# Reference solution for the yamaa benchmark sdtm-ae-coding (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)ae_raw <- read_csv( "/app/input/ae_raw.csv", col_types = cols(AESEQ = col_integer(), .default = col_character()))meddra <- read_csv( "/app/input/meddra_26_1.csv", col_types = cols(.default = col_character()))# The reported term in upper case, coded to the preferred term and body# system of the entry whose lowest-level term, in upper case, equals it. A# term with no match is not coded.ae <- ae_raw |> mutate(AETERM = toupper(AETERM)) |> left_join( mutate(meddra, LLTNAME = toupper(LLTNAME)), by = c("AETERM" = "LLTNAME") ) |> mutate( DOMAIN = "AE", AEDECOD = if_else(is.na(PTNAME), "NOT CODED", PTNAME), AEBODSYS = if_else(is.na(SOCNAME), "NOT CODED", SOCNAME) ) |> select(DOMAIN, STUDYID, USUBJID, AESEQ, AETERM, AEDECOD, AEBODSYS)dir.create("/app/output", showWarnings = FALSE)write_csv(ae, "/app/output/ae.csv", na = "")solutions/sdtm-ae-coding/result.py
Edit63 lines# Reference solution for the yamaa benchmark sdtm-ae-coding (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1."""Create the SDTM AE dataset by coding reported terms against MedDRA 26.1.Reads AE_RAW and MEDDRA_26_1 from /app/input and writes ae.csv to/app/output with one record per adverse event. AETERM is stored in uppercase and matched to the dictionary's lowest-level term in upper case; thepreferred term and body system come from the matched entry. A term withno match is coded "NOT CODED" in both fields."""import csvfrom pathlib import PathINPUT = Path("/app/input")OUTPUT = Path("/app/output")NOT_CODED = "NOT CODED"def read_rows(path): with open(path, newline="", encoding="utf-8") as handle: return list(csv.DictReader(handle))def main(): dictionary = {} for entry in read_rows(INPUT / "meddra_26_1.csv"): dictionary[entry["LLTNAME"].upper()] = (entry["PTNAME"], entry["SOCNAME"]) records = [] for event in read_rows(INPUT / "ae_raw.csv"): term = event["AETERM"].upper() if term in dictionary: decoded, body_system = dictionary[term] else: decoded, body_system = NOT_CODED, NOT_CODED records.append( { "DOMAIN": "AE", "STUDYID": event["STUDYID"], "USUBJID": event["USUBJID"], "AESEQ": event["AESEQ"], "AETERM": term, "AEDECOD": decoded, "AEBODSYS": body_system, } ) OUTPUT.mkdir(parents=True, exist_ok=True) columns = ["DOMAIN", "STUDYID", "USUBJID", "AESEQ", "AETERM", "AEDECOD", "AEBODSYS"] with open(OUTPUT / "ae.csv", "w", newline="", encoding="utf-8") as handle: writer = csv.DictWriter(handle, fieldnames=columns) writer.writeheader() writer.writerows(records)if __name__ == "__main__": main()solutions/sdtm-ae-coding/result.sas
Edit25 lines/* opensas reference solution for the yamaa benchmark sdtm-ae-coding. Reads the benchmark inputs and derives the requested output. Execute with the existing opensas runtime: opensas /app/output/result.sas. */data ae_raw; length STUDYID USUBJID AETERM $1024; infile "/app/input/ae_raw.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. AESEQ AETERM : $char1024.;run;data meddra_26_1; length LLTNAME PTNAME SOCNAME $1024; infile "/app/input/meddra_26_1.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input LLTNAME : $char1024. PTNAME : $char1024. SOCNAME : $char1024.;run;proc sql; create table result as select 'AE' as DOMAIN,a.STUDYID,a.USUBJID,a.AESEQ,upcase(a.AETERM) as AETERM,coalescec(b.PTNAME,'NOT CODED') as AEDECOD,coalescec(b.SOCNAME,'NOT CODED') as AEBODSYS from ae_raw a left join meddra_26_1 b on upcase(a.AETERM)=upcase(b.LLTNAME);quit;proc sql; create table final as select DOMAIN, STUDYID, USUBJID, AESEQ, AETERM, AEDECOD, AEBODSYS from result;quit;proc export data=final outfile="/app/output/ae.csv" dbms=csv replace; run;
yamaa