Review ask

As a clinical trial statistician or programmer, challenge anything in this benchmark: describe a problem in Comments, or propose a fix with the Edit button on any source file.

  1. Task: Would a real study need this derivation, specified this way? Does it follow CDISC standards: variable names, controlled terms, record structure? Flag any rule that is ambiguous or that a sponsor would handle differently.
  2. Inputs: Do they look like data collected in a real trial, and do they include the situations you would expect to meet in one?
  3. Expected output: Would independent QC programming reproduce every value from these inputs and rules?

The other tabs beside Comments are background for assessment, not under review.

Summary

Lifecycle: draftEditUnder review
What this benchmark means
README.md

Goal: code reported medication names with the World Health Organization drug dictionary (WHODrug), carrying the coder's chosen drug record and Anatomical Therapeutic Chemical (ATC) class into CM.

Input: ODM item data for the medication log: each reported medication is one repeat of the IG.CM item group, the reported name is item IT.CM.CMTRT, and the repeat number becomes CMSEQ; the coder's WHODrug coding output (chosen drug record and ATC code per collected record); and a WHODrug dictionary extract.

Variables:

  • CMTRT is the medication name as reported, kept exactly as written.
  • CMDECOD is the preferred name of the drug record the coder chose for the reported name; blank when the verbatim is not yet coded.
  • CMCLAS is the class name of the ATC code the coder assigned for this use; blank when the verbatim is not yet coded.
  • CMCLASCD is the code of that class; blank when the verbatim is not yet coded.

Note: one drug can carry several ATC codes, and the coder assigns the class for the use at hand: the same drug record can code to a different CMCLAS on different records.

Standard: SDTM | Domain: CM

Inputs

Under review
3 source files

input/coding_output.csv

Edit
3 rows
input/coding_output.csv
STUDYIDUSUBJIDCMSEQDRUG_RECORD_NODRUG_CODEATC_CODE
YAMAA-0111001IBU-200M01AE01
YAMAA-0121002ASA-100N02BA01
YAMAA-0111002ASA-100B01AC06

input/odm.csv

Edit
4 rows
input/odm.csv
StudyOIDMetaDataVersionOIDSubjectKeyStudyEventOIDStudyEventRepeatKeyItemGroupOIDItemGroupRepeatKeyItemOIDValue
YAMAA-01MDV.1001-101TREATMENT1IG.CM1IT.CM.CMTRTADVIL
YAMAA-01MDV.1001-101TREATMENT1IG.CM2IT.CM.CMTRTAsprin
YAMAA-01MDV.1001-102TREATMENT1IG.CM1IT.CM.CMTRTASPIRIN
YAMAA-01MDV.1001-102TREATMENT1IG.CM2IT.CM.CMTRTTURMERIC

input/whodrug_extract.csv

Edit
3 rows
input/whodrug_extract.csv
DRUG_RECORD_NOPREFERRED_NAMEDRUG_CODEATC_CODEATC_CLASS_CODEATC_CLASS_NAME
1001IBUPROFENIBU-200M01AE01M01AEPropionic acid derivatives
1002ACETYLSALICYLIC ACIDASA-100N02BA01N02BASalicylic acid and derivatives
1002ACETYLSALICYLIC ACIDASA-100B01AC06B01ACPlatelet aggregation inhibitors excl. heparin

Expected output

Under review
4 expected rows

expected/cm.csv

Edit
4 rows
expected/cm.csv
DOMAINSTUDYIDUSUBJIDCMSEQCMTRTCMDECODCMCLASCMCLASCD
CMYAMAA-011ADVILIBUPROFENPropionic acid derivativesM01AE
CMYAMAA-012AsprinACETYLSALICYLIC ACIDSalicylic acid and derivativesN02BA
CMYAMAA-011ASPIRINACETYLSALICYLIC ACIDPlatelet aggregation inhibitors excl. heparinB01AC
CMYAMAA-012TURMERIC———
Shaded columns are derived— = missing value

Comments and assessment materials

Comments

Signed in with GitHub; saved in repository Discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Variable NameVariable LabelTypeLengthControlled Terms or FormatOriginCoreConversion DefinitionVariable TypeVariable OrderComments for Define
DOMAINDomain AbbreviationCharAssignedSet constant value "CM".SDTM1
STUDYIDStudy IdentifierCharCollectedCopy value from ODM.StudyOID.SDTM2
USUBJIDUnique Subject IdentifierCharCollectedCopy value from ODM.SubjectKey.SDTM3
CMSEQSequence NumberNumCollectedCopy value from ODM.ItemGroupRepeatKey.SDTM4
CMTRTReported Name of the MedicationCharDerivedSet per constructed row; see the Row construction sheet.SDTM5
CMDECODStandardized Medication NameCharDerivedTake PREFERRED_NAME from the WHODRUG record whose DRUG_RECORD_NO equals DRUG_RECORD_NO and ATC_CODE equals ATC_CODE, and no match yields blank.SDTM6
CMCLASMedication ClassCharDerivedTake ATC_CLASS_NAME from the WHODRUG record whose DRUG_RECORD_NO equals DRUG_RECORD_NO and ATC_CODE equals ATC_CODE, and no match yields blank.SDTM7
CMCLASCDMedication Class CodeCharDerivedTake ATC_CLASS_CODE from the WHODRUG record whose DRUG_RECORD_NO equals DRUG_RECORD_NO and ATC_CODE equals ATC_CODE, and no match yields blank.SDTM8
TemplateInclude whenCMTRTDRUG_RECORD_NOATC_CODE
medicationAggregate ONLY(ODM.Value) over records matching on the shared output keys, where ODM.ItemOID = 'IT.CM.CMTRT'Copy DRUG_RECORD_NO from the CODING record whose STUDYID equals ODM.StudyOID, USUBJID equals ODM.SubjectKey and CMSEQ equals ODM.ItemGroupRepeatKey, and no match yields blankCopy ATC_CODE from the CODING record whose STUDYID equals ODM.StudyOID, USUBJID equals ODM.SubjectKey and CMSEQ equals ODM.ItemGroupRepeatKey, and no match yields blank
VersionDateAuthorDescriptionReviewerSign-off
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/sdtm-cm-whodrug-coding/full.md

Edit
22 lines

Following CDISC SDTM standards, use the provided ODM, CODING, and WHODRUG datasets to create a CM dataset with one record per reported medication.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, CMSEQ, CMTRT, CMDECOD, CMCLAS, CMCLASCD

STUDYID is the ODM StudyOID and USUBJID is the SubjectKey, each as given.

CMSEQ is the medication repeat number within the subject. CMTRT keeps the medication name exactly as written. CMDECOD is the preferred name of the drug record the coder chose for the reported name, and has no value when the reported name is not yet coded. CMCLAS is the class name of the ATC code the coder assigned for this use, and CMCLASCD is the code of that class; both have no value when the reported name is not yet coded. One drug can carry several ATC codes, and the coder assigns the class for the use at hand, so the same drug record can give a different class on different records. A record with no coding, or a code not in the extract, leaves the coded variables with no value.

Read the source datasets from /app/input and save the completed dataset as /app/output/cm.csv.

prompts/sdtm-cm-whodrug-coding/conventions.md

Edit
9 lines

Following CDISC SDTM standards, use the provided ODM, CODING, and WHODRUG datasets to create a CM dataset with one record per reported medication.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, CMSEQ, CMTRT, CMDECOD, CMCLAS, CMCLASCD

Read the source datasets from /app/input and save the completed dataset as /app/output/cm.csv.

prompts/sdtm-cm-whodrug-coding/brief.md

Edit
9 lines

Following CDISC SDTM standards, use the provided ODM, CODING, and WHODRUG datasets to create a CM dataset with one record per reported medication.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, CMSEQ, CMTRT, CMDECOD, CMCLAS, CMCLASCD

Read the source datasets from /app/input and save the completed dataset as /app/output/cm.csv.

yamaa code

Runs the yamaa specification; its result must match the expected output

run.R

Edit
4 lines
library(yamaa)cm <- yamaa_domain("spec.yaml")$outputcm

run.py

Edit
4 lines
import yamaacm = yamaa.yamaa_domain("spec.yaml").outputcm

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/sdtm-cm-whodrug-coding/result.R

Edit
61 lines
# Reference solution for the yamaa benchmark sdtm-cm-whodrug-coding (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)library(tidyr)odm <- read_csv(  "/app/input/odm.csv",  col_types = cols(.default = col_character()))coding <- read_csv(  "/app/input/coding_output.csv",  col_types = cols(CMSEQ = col_integer(), .default = col_character()))whodrug <- read_csv(  "/app/input/whodrug_extract.csv",  col_types = cols(.default = col_character()))# One record per reported medication; the repeat number is the sequence# number. The preferred name comes from the drug record the coder chose,# and the class from the ATC code assigned for this use, so the same drug# record can give a different class on different records. No coding, or a# code not in the extract, leaves the coded variables empty.reported <- odm |>  filter(ItemGroupOID == "IG.CM") |>  pivot_wider(    id_cols = c(StudyOID, SubjectKey, ItemGroupRepeatKey),    names_from = ItemOID,    values_from = Value,    values_fn = first  ) |>  mutate(    STUDYID = StudyOID,    USUBJID = SubjectKey,    CMSEQ = as.integer(ItemGroupRepeatKey),    CMTRT = `IT.CM.CMTRT`  ) |>  select(STUDYID, USUBJID, CMSEQ, CMTRT)cm <- reported |>  left_join(coding, by = c("STUDYID", "USUBJID", "CMSEQ")) |>  left_join(whodrug, by = c("DRUG_RECORD_NO", "ATC_CODE")) |>  mutate(    DOMAIN = "CM",    CMDECOD = if_else(      is.na(DRUG_RECORD_NO) | DRUG_RECORD_NO == "" | is.na(PREFERRED_NAME),      NA_character_, PREFERRED_NAME    ),    CMCLAS = if_else(is.na(CMDECOD), NA_character_, ATC_CLASS_NAME),    CMCLASCD = if_else(is.na(CMDECOD), NA_character_, ATC_CLASS_CODE)  ) |>  arrange(USUBJID, CMSEQ) |>  select(DOMAIN, STUDYID, USUBJID, CMSEQ, CMTRT, CMDECOD, CMCLAS, CMCLASCD)dir.create("/app/output", showWarnings = FALSE)write_csv(cm, "/app/output/cm.csv", na = "")

solutions/sdtm-cm-whodrug-coding/result.py

Edit
82 lines
# Reference solution for the yamaa benchmark sdtm-cm-whodrug-coding (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.from pathlib import Pathimport polars as plodm = pl.read_csv("/app/input/odm.csv", infer_schema=False)coding = pl.read_csv("/app/input/coding_output.csv", infer_schema=False).with_columns(    pl.col("CMSEQ").cast(pl.Int64))whodrug = pl.read_csv("/app/input/whodrug_extract.csv", infer_schema=False)# One record per reported medication; the repeat number is the sequence# number. The preferred name comes from the drug record the coder chose,# and the class from the ATC code assigned for this use, so the same drug# record can give a different class on different records. No coding, or a# code not in the extract, leaves the coded variables empty.reported = (    odm.filter(pl.col("ItemGroupOID") == "IG.CM")    .pivot(        index=["StudyOID", "SubjectKey", "ItemGroupRepeatKey"],        on="ItemOID",        values="Value",        aggregate_function="first",    )    .with_columns(        STUDYID=pl.col("StudyOID"),        USUBJID=pl.col("SubjectKey"),        CMSEQ=pl.col("ItemGroupRepeatKey").cast(pl.Int64),        CMTRT=pl.col("IT.CM.CMTRT"),    )    .select("STUDYID", "USUBJID", "CMSEQ", "CMTRT"))cm = (    reported.join(coding, on=["STUDYID", "USUBJID", "CMSEQ"], how="left")    .join(        whodrug,        left_on=["DRUG_RECORD_NO", "ATC_CODE"],        right_on=["DRUG_RECORD_NO", "ATC_CODE"],        how="left",    )    .with_columns(        DOMAIN=pl.lit("CM"),        CMDECOD=pl.when(pl.col("PREFERRED_NAME").is_null())        .then(None)        .otherwise(pl.col("PREFERRED_NAME")),        CMCLAS=pl.col("ATC_CLASS_NAME"),        CMCLASCD=pl.col("ATC_CLASS_CODE"),    )    .with_columns(        CMDECOD=pl.when(            pl.col("DRUG_RECORD_NO").fill_null("") == ""        )        .then(None)        .otherwise(pl.col("CMDECOD")),        CMCLAS=pl.when(pl.col("CMDECOD").is_null())        .then(None)        .otherwise(pl.col("CMCLAS")),        CMCLASCD=pl.when(pl.col("CMDECOD").is_null())        .then(None)        .otherwise(pl.col("CMCLASCD")),    )    .sort(["USUBJID", "CMSEQ"])    .select(        "DOMAIN",        "STUDYID",        "USUBJID",        "CMSEQ",        "CMTRT",        "CMDECOD",        "CMCLAS",        "CMCLASCD",    ))Path("/app/output").mkdir(exist_ok=True)cm.write_csv("/app/output/cm.csv")

solutions/sdtm-cm-whodrug-coding/result.sas

Edit
43 lines
/* opensas reference solution for the yamaa benchmark sdtm-cm-whodrug-coding.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data coding_output;  length STUDYID USUBJID DRUG_RECORD_NO DRUG_CODE ATC_CODE $1024;  infile "/app/input/coding_output.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. CMSEQ DRUG_RECORD_NO : $char1024. DRUG_CODE : $char1024. ATC_CODE : $char1024.;run;data odm;  length StudyOID MetaDataVersionOID SubjectKey StudyEventOID StudyEventRepeatKey ItemGroupOID ItemGroupRepeatKey ItemOID Value $1024;  infile "/app/input/odm.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input StudyOID : $char1024. MetaDataVersionOID : $char1024. SubjectKey : $char1024. StudyEventOID : $char1024. StudyEventRepeatKey : $char1024. ItemGroupOID : $char1024. ItemGroupRepeatKey : $char1024. ItemOID : $char1024. Value : $char1024.;run;data whodrug_extract;  length DRUG_RECORD_NO PREFERRED_NAME DRUG_CODE ATC_CODE ATC_CLASS_CODE ATC_CLASS_NAME $1024;  infile "/app/input/whodrug_extract.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input DRUG_RECORD_NO : $char1024. PREFERRED_NAME : $char1024. DRUG_CODE : $char1024. ATC_CODE : $char1024. ATC_CLASS_CODE : $char1024. ATC_CLASS_NAME : $char1024.;run;proc sort data=odm out=ordered; by StudyOID SubjectKey ItemGroupRepeatKey; run;data wide;  length CMTRT $1024;  retain CMTRT;  set ordered;  where ItemGroupOID='IG.CM';  by StudyOID SubjectKey ItemGroupRepeatKey;  if first.ItemGroupRepeatKey then do;    CMTRT='';  end;    if ItemOID='IT.CM.CMTRT' then CMTRT=Value;  if last.ItemGroupRepeatKey then output;run;data medications; set wide; length DOMAIN $2 STUDYID USUBJID $1024; DOMAIN='CM'; STUDYID=StudyOID; USUBJID=SubjectKey; CMSEQ=input(ItemGroupRepeatKey,best32.); run;proc sql; create table result as select a.*,c.PREFERRED_NAME as CMDECOD,c.ATC_CLASS_NAME as CMCLAS,c.ATC_CLASS_CODE as CMCLASCD from medications aleft join coding_output b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID and a.CMSEQ=b.CMSEQleft join whodrug_extract c on b.DRUG_RECORD_NO=c.DRUG_RECORD_NO and b.DRUG_CODE=c.DRUG_CODE and b.ATC_CODE=c.ATC_CODE; quit;proc sql;  create table final as select DOMAIN, STUDYID, USUBJID, CMSEQ, CMTRT, CMDECOD, CMCLAS, CMCLASCD from result;quit;proc export data=final outfile="/app/output/cm.csv" dbms=csv replace; run;