As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.
We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.
Summary
README.mdPublic entries: Python and R. Passing coverage requires an installed inventory.
Goal: code reported medication names with the World Health Organization drug dictionary (WHODrug), carrying the coder's chosen drug record and Anatomical Therapeutic Chemical (ATC) class into CM.
Input: ODM item data for the medication log: each reported
medication is one repeat of the IG.CM item group, the reported name
is item IT.CM.CMTRT, and the repeat number becomes CMSEQ; the
coder's WHODrug coding output (chosen drug record and ATC code per
collected record); and a WHODrug dictionary extract.
Variables:
CMTRTis the medication name as reported, kept exactly as written.CMDECODis the preferred name of the drug record the coder chose for the reported name; blank when the verbatim is not yet coded.CMCLASis the class name of the ATC code the coder assigned for this use; blank when the verbatim is not yet coded.CMCLASCDis the code of that class; blank when the verbatim is not yet coded.
Note: one drug can carry several ATC codes, and the coder assigns
the class for the use at hand: the same drug record can code to a
different CMCLAS on different records.
Inputs
Under review3 source filesinput/coding_output.csv
Edit3 rows| STUDYID | USUBJID | CMSEQ | DRUG_RECORD_NO | DRUG_CODE | ATC_CODE |
|---|---|---|---|---|---|
| YAMAA-01 | 1 | 1001 | IBU-200 | M01AE01 | |
| YAMAA-01 | 2 | 1002 | ASA-100 | N02BA01 | |
| YAMAA-01 | 1 | 1002 | ASA-100 | B01AC06 |
input/odm.csv
Edit4 rows| StudyOID | MetaDataVersionOID | SubjectKey | StudyEventOID | StudyEventRepeatKey | ItemGroupOID | ItemGroupRepeatKey | ItemOID | Value |
|---|---|---|---|---|---|---|---|---|
| YAMAA-01 | MDV.1 | 001-101 | TREATMENT | 1 | IG.CM | 1 | IT.CM.CMTRT | ADVIL |
| YAMAA-01 | MDV.1 | 001-101 | TREATMENT | 1 | IG.CM | 2 | IT.CM.CMTRT | Asprin |
| YAMAA-01 | MDV.1 | 001-102 | TREATMENT | 1 | IG.CM | 1 | IT.CM.CMTRT | ASPIRIN |
| YAMAA-01 | MDV.1 | 001-102 | TREATMENT | 1 | IG.CM | 2 | IT.CM.CMTRT | TURMERIC |
input/whodrug_extract.csv
Edit3 rows| DRUG_RECORD_NO | PREFERRED_NAME | DRUG_CODE | ATC_CODE | ATC_CLASS_CODE | ATC_CLASS_NAME |
|---|---|---|---|---|---|
| 1001 | IBUPROFEN | IBU-200 | M01AE01 | M01AE | Propionic acid derivatives |
| 1002 | ACETYLSALICYLIC ACID | ASA-100 | N02BA01 | N02BA | Salicylic acid and derivatives |
| 1002 | ACETYLSALICYLIC ACID | ASA-100 | B01AC06 | B01AC | Platelet aggregation inhibitors excl. heparin |
Expected output
Under review4 expected rowsexpected/cm.csv
Edit4 rows| DOMAIN | STUDYID | USUBJID | CMSEQ | CMTRT | CMDECOD | CMCLAS | CMCLASCD |
|---|---|---|---|---|---|---|---|
| CM | YAMAA-01 | 1 | ADVIL | IBUPROFEN | Propionic acid derivatives | M01AE | |
| CM | YAMAA-01 | 2 | Asprin | ACETYLSALICYLIC ACID | Salicylic acid and derivatives | N02BA | |
| CM | YAMAA-01 | 1 | ASPIRIN | ACETYLSALICYLIC ACID | Platelet aggregation inhibitors excl. heparin | B01AC | |
| CM | YAMAA-01 | 2 | TURMERIC | — | — | — |
Human review and assessment
| Variable Name | Variable Label | Type | Length | Controlled Terms or Format | Origin | Core | Conversion Definition | Variable Type | Variable Order | Comments for Define |
|---|---|---|---|---|---|---|---|---|---|---|
| DOMAIN | Domain Abbreviation | Char | Assigned | Set constant value "CM". | SDTM | 1 | ||||
| STUDYID | Study Identifier | Char | Collected | Copy value from ODM.StudyOID. | SDTM | 2 | ||||
| USUBJID | Unique Subject Identifier | Char | Collected | Copy value from ODM.SubjectKey. | SDTM | 3 | ||||
| CMSEQ | Sequence Number | Num | Collected | Copy value from ODM.ItemGroupRepeatKey. | SDTM | 4 | ||||
| CMTRT | Reported Name of the Medication | Char | Derived | Set per constructed row; see the Row construction sheet. | SDTM | 5 | ||||
| CMDECOD | Standardized Medication Name | Char | Derived | Take PREFERRED_NAME from the WHODRUG record whose DRUG_RECORD_NO equals DRUG_RECORD_NO and ATC_CODE equals ATC_CODE, and no match yields blank. | SDTM | 6 | ||||
| CMCLAS | Medication Class | Char | Derived | Take ATC_CLASS_NAME from the WHODRUG record whose DRUG_RECORD_NO equals DRUG_RECORD_NO and ATC_CODE equals ATC_CODE, and no match yields blank. | SDTM | 7 | ||||
| CMCLASCD | Medication Class Code | Char | Derived | Take ATC_CLASS_CODE from the WHODRUG record whose DRUG_RECORD_NO equals DRUG_RECORD_NO and ATC_CODE equals ATC_CODE, and no match yields blank. | SDTM | 8 |
| Template | Include when | CMTRT | DRUG_RECORD_NO | ATC_CODE |
|---|---|---|---|---|
| medication | Aggregate ONLY(ODM.Value) over records matching on the shared output keys, where ODM.ItemOID = 'IT.CM.CMTRT' | Copy DRUG_RECORD_NO from the CODING record whose STUDYID equals ODM.StudyOID, USUBJID equals ODM.SubjectKey and CMSEQ equals ODM.ItemGroupRepeatKey, and no match yields blank | Copy ATC_CODE from the CODING record whose STUDYID equals ODM.StudyOID, USUBJID equals ODM.SubjectKey and CMSEQ equals ODM.ItemGroupRepeatKey, and no match yields blank |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.7 | 2026-10-20 | yamaa docs build | Read a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default | ||
| 1.6 | 2026-10-17 | yamaa docs build | Describe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight | ||
| 1.5 | 2026-10-17 | yamaa docs build | Name collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part | ||
| 1.4 | 2026-10-15 | yamaa docs build | State datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks | ||
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/sdtm-cm-whodrug-coding/full.md
Edit24 linesFollowing CDISC SDTM standards, use the provided CODING_OUTPUT, ODM, and WHODRUG_EXTRACT datasets to create a CM dataset with one record per reported medication.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, CMSEQ, CMTRT, CMDECOD, CMCLAS, CMCLASCD
STUDYID is the study OID in ODM as given, and USUBJID is the subject key in ODM as given. CMSEQ is the repeat number of the reported medication for the subject. CMTRT is the medication name as reported, the value of the IT.CM.CMTRT item, kept exactly as written. Each record has the drug record and ATC code the coder's output assigns to its STUDYID, USUBJID, and CMSEQ. CMDECOD is the preferred name in the WHODrug extract for the drug record the coder chose. CMCLAS is the ATC class name in the WHODrug extract for that drug record together with the ATC code the coder assigned to the record. CMCLASCD is the ATC class code in the WHODrug extract for that same drug record and ATC code. CMDECOD, CMCLAS, and CMCLASCD have no value when the reported name is not yet coded. The same drug record can code to a different CMCLAS on different records, because the class is the one for the ATC code the coder assigned for the use at hand.
Read the source datasets from /app/input and save the completed dataset as /app/output/cm.csv.
prompts/sdtm-cm-whodrug-coding/conventions.md
Edit9 linesFollowing CDISC SDTM standards, use the provided CODING_OUTPUT, ODM, and WHODRUG_EXTRACT datasets to create a CM dataset with one record per reported medication.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, CMSEQ, CMTRT, CMDECOD, CMCLAS, CMCLASCD
Read the source datasets from /app/input and save the completed dataset as /app/output/cm.csv.
prompts/sdtm-cm-whodrug-coding/brief.md
Edit9 linesFollowing CDISC SDTM standards, use the provided CODING_OUTPUT, ODM, and WHODRUG_EXTRACT datasets to create a CM dataset with one record per reported medication.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, CMSEQ, CMTRT, CMDECOD, CMCLAS, CMCLASCD
Read the source datasets from /app/input and save the completed dataset as /app/output/cm.csv.
R
Edit3 lineslibrary(yamaa)yamaa_spec("spec.yaml")$run()$outputPython
Edit3 linesimport yamaayamaa.spec("spec.yaml").run().outputsolutions/sdtm-cm-whodrug-coding/result.R
Edit61 lines# Reference solution for the yamaa benchmark sdtm-cm-whodrug-coding (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)library(tidyr)odm <- read_csv( "/app/input/odm.csv", col_types = cols(.default = col_character()))coding <- read_csv( "/app/input/coding_output.csv", col_types = cols(CMSEQ = col_integer(), .default = col_character()))whodrug <- read_csv( "/app/input/whodrug_extract.csv", col_types = cols(.default = col_character()))# One record per reported medication; the repeat number is the sequence# number. The preferred name comes from the drug record the coder chose,# and the class from the ATC code assigned for this use, so the same drug# record can give a different class on different records. No coding, or a# code not in the extract, leaves the coded variables empty.reported <- odm |> filter(ItemGroupOID == "IG.CM") |> pivot_wider( id_cols = c(StudyOID, SubjectKey, ItemGroupRepeatKey), names_from = ItemOID, values_from = Value, values_fn = first ) |> mutate( STUDYID = StudyOID, USUBJID = SubjectKey, CMSEQ = as.integer(ItemGroupRepeatKey), CMTRT = `IT.CM.CMTRT` ) |> select(STUDYID, USUBJID, CMSEQ, CMTRT)cm <- reported |> left_join(coding, by = c("STUDYID", "USUBJID", "CMSEQ")) |> left_join(whodrug, by = c("DRUG_RECORD_NO", "ATC_CODE")) |> mutate( DOMAIN = "CM", CMDECOD = if_else( is.na(DRUG_RECORD_NO) | DRUG_RECORD_NO == "" | is.na(PREFERRED_NAME), NA_character_, PREFERRED_NAME ), CMCLAS = if_else(is.na(CMDECOD), NA_character_, ATC_CLASS_NAME), CMCLASCD = if_else(is.na(CMDECOD), NA_character_, ATC_CLASS_CODE) ) |> arrange(USUBJID, CMSEQ) |> select(DOMAIN, STUDYID, USUBJID, CMSEQ, CMTRT, CMDECOD, CMCLAS, CMCLASCD)dir.create("/app/output", showWarnings = FALSE)write_csv(cm, "/app/output/cm.csv", na = "")solutions/sdtm-cm-whodrug-coding/result.py
Edit90 lines# Reference solution for the yamaa benchmark sdtm-cm-whodrug-coding (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1."""Build an SDTM CM dataset from ODM reported medications, the coder'sWHODrug coding output, and the WHODrug extract.One record per reported medication (the IT.CM.CMTRT item of each IG.CMitem group in ODM). The coded variables come from the WHODrug extract rowfor the drug record and ATC code the coder assigned to that record; theystay blank when the reported name is not yet coded."""import polars as plINPUT_DIR = "/app/input"OUTPUT_DIR = "/app/output"COLUMNS = [ "DOMAIN", "STUDYID", "USUBJID", "CMSEQ", "CMTRT", "CMDECOD", "CMCLAS", "CMCLASCD",]def read_csv(path): # Read every column as text so values are kept exactly as written. return pl.read_csv(path, infer_schema_length=0)def main(): odm = read_csv(f"{INPUT_DIR}/odm.csv") coding = read_csv(f"{INPUT_DIR}/coding_output.csv") whodrug = read_csv(f"{INPUT_DIR}/whodrug_extract.csv") # One record per reported medication: the CMTRT item of each CM item # group. CMSEQ is the item group's repeat key within the subject. meds = odm.filter(pl.col("ItemOID") == "IT.CM.CMTRT") cm = meds.select( pl.lit("CM").alias("DOMAIN"), pl.col("StudyOID").alias("STUDYID"), pl.col("SubjectKey").alias("USUBJID"), pl.col("ItemGroupRepeatKey").cast(pl.Int64).alias("CMSEQ"), pl.col("Value").alias("CMTRT"), ).sort(["USUBJID", "CMSEQ"], maintain_order=True) # The drug record and ATC code the coder assigned to each record. cm = cm.join( coding.select("STUDYID", "USUBJID", "CMSEQ", "DRUG_RECORD_NO", "ATC_CODE") .with_columns(pl.col("CMSEQ").cast(pl.Int64)), on=["STUDYID", "USUBJID", "CMSEQ"], how="left", ) # Preferred name and ATC class for that drug record together with the # assigned ATC code; the same drug record can carry a different class # for a different assigned ATC code. cm = cm.join( whodrug.select( "DRUG_RECORD_NO", "ATC_CODE", "PREFERRED_NAME", "ATC_CLASS_NAME", "ATC_CLASS_CODE", ), on=["DRUG_RECORD_NO", "ATC_CODE"], how="left", ) cm = cm.rename( { "PREFERRED_NAME": "CMDECOD", "ATC_CLASS_NAME": "CMCLAS", "ATC_CLASS_CODE": "CMCLASCD", } ) # Uncoded values stay null and are written as empty fields. cm.select(COLUMNS).write_csv(f"{OUTPUT_DIR}/cm.csv", null_value="")if __name__ == "__main__": main()solutions/sdtm-cm-whodrug-coding/result.sas
Edit43 lines/* opensas reference solution for the yamaa benchmark sdtm-cm-whodrug-coding. Reads the benchmark inputs and derives the requested output. Execute with the existing opensas runtime: opensas /app/output/result.sas. */data coding_output; length STUDYID USUBJID DRUG_RECORD_NO DRUG_CODE ATC_CODE $1024; infile "/app/input/coding_output.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. CMSEQ DRUG_RECORD_NO : $char1024. DRUG_CODE : $char1024. ATC_CODE : $char1024.;run;data odm; length StudyOID MetaDataVersionOID SubjectKey StudyEventOID StudyEventRepeatKey ItemGroupOID ItemGroupRepeatKey ItemOID Value $1024; infile "/app/input/odm.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input StudyOID : $char1024. MetaDataVersionOID : $char1024. SubjectKey : $char1024. StudyEventOID : $char1024. StudyEventRepeatKey : $char1024. ItemGroupOID : $char1024. ItemGroupRepeatKey : $char1024. ItemOID : $char1024. Value : $char1024.;run;data whodrug_extract; length DRUG_RECORD_NO PREFERRED_NAME DRUG_CODE ATC_CODE ATC_CLASS_CODE ATC_CLASS_NAME $1024; infile "/app/input/whodrug_extract.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input DRUG_RECORD_NO : $char1024. PREFERRED_NAME : $char1024. DRUG_CODE : $char1024. ATC_CODE : $char1024. ATC_CLASS_CODE : $char1024. ATC_CLASS_NAME : $char1024.;run;proc sort data=odm out=ordered; by StudyOID SubjectKey ItemGroupRepeatKey; run;data wide; length CMTRT $1024; retain CMTRT; set ordered; where ItemGroupOID='IG.CM'; by StudyOID SubjectKey ItemGroupRepeatKey; if first.ItemGroupRepeatKey then do; CMTRT=''; end; if ItemOID='IT.CM.CMTRT' then CMTRT=Value; if last.ItemGroupRepeatKey then output;run;data medications; set wide; length DOMAIN $2 STUDYID USUBJID $1024; DOMAIN='CM'; STUDYID=StudyOID; USUBJID=SubjectKey; CMSEQ=input(ItemGroupRepeatKey,best32.); run;proc sql; create table result as select a.*,c.PREFERRED_NAME as CMDECOD,c.ATC_CLASS_NAME as CMCLAS,c.ATC_CLASS_CODE as CMCLASCD from medications aleft join coding_output b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID and a.CMSEQ=b.CMSEQleft join whodrug_extract c on b.DRUG_RECORD_NO=c.DRUG_RECORD_NO and b.DRUG_CODE=c.DRUG_CODE and b.ATC_CODE=c.ATC_CODE; quit;proc sql; create table final as select DOMAIN, STUDYID, USUBJID, CMSEQ, CMTRT, CMDECOD, CMCLAS, CMCLASCD from result;quit;proc export data=final outfile="/app/output/cm.csv" dbms=csv replace; run;
yamaa