yamaa / Benchmark

ADaM.ADCM

ATC Classes for Coded Medications

input files
3
subjects
3
expected rows
6
Review ask

As a clinical trial statistician or programmer, challenge anything in this benchmark: describe a problem in Comments, or propose a fix with the Edit button on any source file.

  1. Task: Would a real study need this derivation, specified this way? Does it follow CDISC ADaM: variable names, controlled terms, record structure? Flag any rule that is ambiguous or that a sponsor would handle differently.
  2. Inputs: Do they look like data collected in a real trial, and do they include the situations you would expect to meet in one?
  3. Expected output: Would independent QC programming reproduce every value from these inputs and rules?

The other tabs beside Comments are background for assessment, not under review.

Summary

Lifecycle: draftEditUnder reviewWhat this benchmark means
README.md

Goal: carry each medication's coded name (CMDECOD) and its Anatomical Therapeutic Chemical (ATC) classes at levels 1 to 4 into ADCM, as ATC1-ATC4 (class names) and ATC1CD-ATC4CD (codes), keeping the medication record sequence (CMSEQ) so each medication's path stays with its own record.

Input: medication records carrying the reported name (CMTRT) and the coded name (CMDECOD); Findings About records (FACM) assigning one ATC level per row - each row names the level (FATESTCD of ATC1 to ATC4) and the code (FAORRES) for one medication record; and a small ATC dictionary mapping each code to its class name.

Variables:

  • ATC1, ATC2, ATC3, ATC4: the ATC class name at each level, looked up from the dictionary by the level's code.
  • ATC1CD, ATC2CD, ATC3CD, ATC4CD: the ATC code at each level, read from the medication's FACM rows.

Note: a coded medication carries the full four-level path. A medication with no coded name and no path records keeps all eight ATC columns empty. Path rows that name no level from ATC1 to ATC4, or that point at no medication record, change nothing.

Inputs

Under review3 source files

input/atc_dict.csv

Edit12 rows
ATCLEVELATCCODEATCNAME
1BBLOOD AND BLOOD FORMING ORGANS
2B01ANTITHROMBOTIC AGENTS
3B01AANTITHROMBOTIC AGENTS
4B01ACPLATELET AGGREGATION INHIBITORS EXCL. HEPARIN
1CCARDIOVASCULAR SYSTEM
2C10LIPID MODIFYING AGENTS
3C10ALIPID MODIFYING AGENTS, PLAIN
4C10AAHMG COA REDUCTASE INHIBITORS
1AALIMENTARY TRACT AND METABOLISM
2A10DRUGS USED IN DIABETES
3A10BBLOOD GLUCOSE LOWERING DRUGS, EXCL. INSULINS
4A10BABIGUANIDES

input/cm.csv

Edit6 rows
STUDYIDUSUBJIDCMSEQCMTRTCMDECOD
YAMAA-011ASPIRIN 100MGACETYLSALICYLIC ACID
YAMAA-012LIPITOR 20MGATORVASTATIN
YAMAA-013HERBAL TEA—
YAMAA-011ASPIRIN 100MGACETYLSALICYLIC ACID
YAMAA-011METFORMIN 500MGMETFORMIN
YAMAA-012GREEN TEA EXTRACT—

input/facm.csv

Edit18 rows
STUDYIDUSUBJIDFASEQFALNKGRPCMSEQFATESTCDFAORRES
YAMAA-011GRP11ATC1B
YAMAA-012GRP11ATC2B01
YAMAA-013GRP11ATC3B01A
YAMAA-014GRP11ATC4B01AC
YAMAA-015GRP22ATC1C
YAMAA-016GRP22ATC2C10
YAMAA-017GRP22ATC3C10A
YAMAA-018GRP22ATC4C10AA
YAMAA-019GRP31ATC1B
YAMAA-0110GRP31ATC2B01
YAMAA-0111GRP31ATC3B01A
YAMAA-0112GRP31ATC4B01AC
YAMAA-0113GRP41ATC1A
YAMAA-0114GRP41ATC2A10
YAMAA-0115GRP41ATC3A10B
YAMAA-0116GRP41ATC4A10BA
YAMAA-0117GRP11ATC9X
YAMAA-0118GRP999ATC1B

Expected output

Under review6 expected rows

expected/adcm.csv

Edit6 rows
STUDYIDUSUBJIDCMSEQCMTRTCMDECODATC1ATC2ATC3ATC4ATC1CDATC2CDATC3CDATC4CD
YAMAA-011ASPIRIN 100MGACETYLSALICYLIC ACIDBLOOD AND BLOOD FORMING ORGANSANTITHROMBOTIC AGENTSANTITHROMBOTIC AGENTSPLATELET AGGREGATION INHIBITORS EXCL. HEPARINBB01B01AB01AC
YAMAA-012LIPITOR 20MGATORVASTATINCARDIOVASCULAR SYSTEMLIPID MODIFYING AGENTSLIPID MODIFYING AGENTS, PLAINHMG COA REDUCTASE INHIBITORSCC10C10AC10AA
YAMAA-013HERBAL TEA—————————
YAMAA-011ASPIRIN 100MGACETYLSALICYLIC ACIDBLOOD AND BLOOD FORMING ORGANSANTITHROMBOTIC AGENTSANTITHROMBOTIC AGENTSPLATELET AGGREGATION INHIBITORS EXCL. HEPARINBB01B01AB01AC
YAMAA-011METFORMIN 500MGMETFORMINALIMENTARY TRACT AND METABOLISMDRUGS USED IN DIABETESBLOOD GLUCOSE LOWERING DRUGS, EXCL. INSULINSBIGUANIDESAA10A10BA10BA
YAMAA-012GREEN TEA EXTRACT—————————
Shaded columns are derived— = missing value

Comments and assessment materials

Comments

Signed in with GitHub; saved in repository Discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Dataset NameVariable NameVariable LabelTypeCodelist/Controlled TermsCoreComputational MethodOrigin
ADCMSTUDYIDStudy IdentifiercharacterCopy value from CM.STUDYID.Predecessor
ADCMUSUBJIDUnique Subject IdentifiercharacterCopy value from CM.USUBJID.Predecessor
ADCMCMSEQSequence NumbernumericCopy value from CM.CMSEQ.Predecessor
ADCMCMTRTReported Name of Drug, Medication, or TherapycharacterCopy value from CM.CMTRT.Predecessor
ADCMCMDECODStandardized Medication NamecharacterCopy value from CM.CMDECOD.Predecessor
ADCMATC1CDATC Level 1 CodecharacterTake FAORRES from the FACM record whose STUDYID equals CM.STUDYID, USUBJID equals CM.USUBJID and CMSEQ equals CM.CMSEQ, where FACM.FATESTCD = 'ATC1', and no match yields blank.Derived
ADCMATC2CDATC Level 2 CodecharacterTake FAORRES from the FACM record whose STUDYID equals CM.STUDYID, USUBJID equals CM.USUBJID and CMSEQ equals CM.CMSEQ, where FACM.FATESTCD = 'ATC2', and no match yields blank.Derived
ADCMATC3CDATC Level 3 CodecharacterTake FAORRES from the FACM record whose STUDYID equals CM.STUDYID, USUBJID equals CM.USUBJID and CMSEQ equals CM.CMSEQ, where FACM.FATESTCD = 'ATC3', and no match yields blank.Derived
ADCMATC4CDATC Level 4 CodecharacterTake FAORRES from the FACM record whose STUDYID equals CM.STUDYID, USUBJID equals CM.USUBJID and CMSEQ equals CM.CMSEQ, where FACM.FATESTCD = 'ATC4', and no match yields blank.Derived
ADCMATC1ATC Level 1 TextcharacterTake ATCNAME from the ATCDICT record whose ATCCODE equals ATC1CD, and no match yields blank.Derived
ADCMATC2ATC Level 2 TextcharacterTake ATCNAME from the ATCDICT record whose ATCCODE equals ATC2CD, and no match yields blank.Derived
ADCMATC3ATC Level 3 TextcharacterTake ATCNAME from the ATCDICT record whose ATCCODE equals ATC3CD, and no match yields blank.Derived
ADCMATC4ATC Level 4 TextcharacterTake ATCNAME from the ATCDICT record whose ATCCODE equals ATC4CD, and no match yields blank.Derived
VersionDateAuthorDescriptionReviewerSign-off
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/adam-adcm-atc-classes/full.md

Edit18 lines

Following CDISC ADaM standards, use the provided CM, FACM, and ATCDICT datasets to create an ADCM dataset with one record per medication.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, CMSEQ, CMTRT, CMDECOD, ATC1, ATC2, ATC3, ATC4, ATC1CD, ATC2CD, ATC3CD, ATC4CD

Each FACM row assigns one ATC level to one medication record, and the dictionary maps each code to its class name. ATC1CD through ATC4CD are the ATC codes at levels 1 to 4 for the medication's own record, and ATC1 through ATC4 are the class names the dictionary gives for those codes. A coded medication carries the full four-level path. A medication with no coded name keeps all eight ATC columns empty. Path rows that name no level from ATC1 to ATC4, or that point at no medication record, change nothing.

Read the source datasets from /app/input and save the completed dataset as /app/output/adcm.csv.

prompts/adam-adcm-atc-classes/conventions.md

Edit9 lines

Following CDISC ADaM standards, use the provided CM, FACM, and ATCDICT datasets to create an ADCM dataset with one record per medication.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, CMSEQ, CMTRT, CMDECOD, ATC1, ATC2, ATC3, ATC4, ATC1CD, ATC2CD, ATC3CD, ATC4CD

Read the source datasets from /app/input and save the completed dataset as /app/output/adcm.csv.

prompts/adam-adcm-atc-classes/brief.md

Edit9 lines

Following CDISC ADaM standards, use the provided CM, FACM, and ATCDICT datasets to create an ADCM dataset with one record per medication.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, CMSEQ, CMTRT, CMDECOD, ATC1, ATC2, ATC3, ATC4, ATC1CD, ATC2CD, ATC3CD, ATC4CD

Read the source datasets from /app/input and save the completed dataset as /app/output/adcm.csv.

yamaa code

Runs the yamaa specification; its result must match the expected output

run.R

Edit4 lines
library(yamaa)adcm <- yamaa_domain("spec.yaml")$outputadcm

run.py

Edit4 lines
import yamaaadcm = yamaa.yamaa_domain("spec.yaml").outputadcm

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/adam-adcm-atc-classes/result.R

Edit57 lines
# Reference solution for the yamaa benchmark adam-adcm-atc-classes (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)library(tidyr)cm <- read_csv(  "/app/input/cm.csv",  col_types = cols(CMSEQ = col_integer(), .default = col_character()))facm <- read_csv(  "/app/input/facm.csv",  col_types = cols(    FASEQ = col_integer(), CMSEQ = col_integer(), .default = col_character()  ))atcdict <- read_csv(  "/app/input/atc_dict.csv",  col_types = cols(.default = col_character()))# One ATC level per finding row; rows naming no level from ATC1 to ATC4, or# pointing at no medication record, change nothing.levels <- c("ATC1", "ATC2", "ATC3", "ATC4")codes <- facm |>  filter(FATESTCD %in% levels) |>  mutate(code_col = paste0(FATESTCD, "CD")) |>  select(USUBJID, CMSEQ, code_col, FAORRES) |>  pivot_wider(names_from = code_col, values_from = FAORRES) |>  mutate(CMSEQ = as.integer(CMSEQ))lookup <- atcdict |> select(ATCCODE, ATCNAME)adcm <- cm |>  left_join(codes, by = c("USUBJID", "CMSEQ")) |>  # A medication with no coded name keeps every ATC column empty.  mutate(    across(      c(ATC1CD, ATC2CD, ATC3CD, ATC4CD),      ~ if_else(is.na(CMDECOD) | CMDECOD == "", NA_character_, .x)    )  ) |>  left_join(lookup |> rename(ATC1CD = ATCCODE, ATC1 = ATCNAME), by = "ATC1CD") |>  left_join(lookup |> rename(ATC2CD = ATCCODE, ATC2 = ATCNAME), by = "ATC2CD") |>  left_join(lookup |> rename(ATC3CD = ATCCODE, ATC3 = ATCNAME), by = "ATC3CD") |>  left_join(lookup |> rename(ATC4CD = ATCCODE, ATC4 = ATCNAME), by = "ATC4CD") |>  select(    STUDYID, USUBJID, CMSEQ, CMTRT, CMDECOD, ATC1, ATC2, ATC3, ATC4,    ATC1CD, ATC2CD, ATC3CD, ATC4CD  )dir.create("/app/output", showWarnings = FALSE)write_csv(adcm, "/app/output/adcm.csv", na = "")

solutions/adam-adcm-atc-classes/result.py

Edit75 lines
# Reference solution for the yamaa benchmark adam-adcm-atc-classes (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.from pathlib import Pathimport polars as plcm = pl.read_csv("/app/input/cm.csv", infer_schema=False).with_columns(    pl.col("CMSEQ").cast(pl.Int64))facm = pl.read_csv("/app/input/facm.csv", infer_schema=False).with_columns(    pl.col("CMSEQ").cast(pl.Int64, strict=False))atcdict = pl.read_csv("/app/input/atc_dict.csv", infer_schema=False)levels = ["ATC1", "ATC2", "ATC3", "ATC4"]# One ATC level per finding row; rows naming no level from ATC1 to ATC4, or# pointing at no medication record, change nothing.codes = (    facm.filter(pl.col("FATESTCD").is_in(levels))    .pivot(        index=["USUBJID", "CMSEQ"],        on="FATESTCD",        values="FAORRES",        aggregate_function="first",    )    .rename({level: f"{level}CD" for level in levels}))lookup = atcdict.select("ATCCODE", "ATCNAME")adcm = cm.join(codes, on=["USUBJID", "CMSEQ"], how="left")# A medication with no coded name keeps every ATC column empty.uncoded = pl.col("CMDECOD").is_null() | (pl.col("CMDECOD") == "")for level in levels:    code = f"{level}CD"    if code not in adcm.columns:        adcm = adcm.with_columns(pl.lit(None).cast(pl.String).alias(code))    adcm = adcm.with_columns(        pl.when(uncoded).then(None).otherwise(pl.col(code)).alias(code)    )for level in levels:    code, name = f"{level}CD", level    adcm = (        adcm.join(            lookup.rename({"ATCCODE": code, "ATCNAME": name}),            on=code,            how="left",        )    )adcm = adcm.select(    "STUDYID",    "USUBJID",    "CMSEQ",    "CMTRT",    "CMDECOD",    "ATC1",    "ATC2",    "ATC3",    "ATC4",    "ATC1CD",    "ATC2CD",    "ATC3CD",    "ATC4CD",)Path("/app/output").mkdir(exist_ok=True)adcm.write_csv("/app/output/adcm.csv")

solutions/adam-adcm-atc-classes/result.sas

Edit39 lines
/* opensas reference solution for the yamaa benchmark adam-adcm-atc-classes.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data atc_dict;  length ATCLEVEL ATCCODE ATCNAME $1024;  infile "/app/input/atc_dict.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input ATCLEVEL : $char1024. ATCCODE : $char1024. ATCNAME : $char1024.;run;data cm;  length STUDYID USUBJID CMTRT CMDECOD $1024;  infile "/app/input/cm.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. CMSEQ CMTRT : $char1024. CMDECOD : $char1024.;run;data facm;  length STUDYID USUBJID FALNKGRP FATESTCD FAORRES $1024;  infile "/app/input/facm.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. FASEQ FALNKGRP : $char1024. CMSEQ FATESTCD : $char1024. FAORRES : $char1024.;run;proc sql;  create table result as  select a.*, f1.FAORRES as ATC1CD, d1.ATCNAME as ATC1, f2.FAORRES as ATC2CD, d2.ATCNAME as ATC2, f3.FAORRES as ATC3CD, d3.ATCNAME as ATC3, f4.FAORRES as ATC4CD, d4.ATCNAME as ATC4 from cm as aleft join facm as f1 on a.STUDYID=f1.STUDYID and a.USUBJID=f1.USUBJID and a.CMSEQ=f1.CMSEQ and f1.FATESTCD='ATC1' and a.CMDECOD ne ''left join atc_dict as d1 on d1.ATCLEVEL='1' and d1.ATCCODE=f1.FAORRESleft join facm as f2 on a.STUDYID=f2.STUDYID and a.USUBJID=f2.USUBJID and a.CMSEQ=f2.CMSEQ and f2.FATESTCD='ATC2' and a.CMDECOD ne ''left join atc_dict as d2 on d2.ATCLEVEL='2' and d2.ATCCODE=f2.FAORRESleft join facm as f3 on a.STUDYID=f3.STUDYID and a.USUBJID=f3.USUBJID and a.CMSEQ=f3.CMSEQ and f3.FATESTCD='ATC3' and a.CMDECOD ne ''left join atc_dict as d3 on d3.ATCLEVEL='3' and d3.ATCCODE=f3.FAORRESleft join facm as f4 on a.STUDYID=f4.STUDYID and a.USUBJID=f4.USUBJID and a.CMSEQ=f4.CMSEQ and f4.FATESTCD='ATC4' and a.CMDECOD ne ''left join atc_dict as d4 on d4.ATCLEVEL='4' and d4.ATCCODE=f4.FAORRES;quit;proc sql;  create table final as select STUDYID, USUBJID, CMSEQ, CMTRT, CMDECOD, ATC1, ATC2, ATC3, ATC4, ATC1CD, ATC2CD, ATC3CD, ATC4CD from result;quit;proc export data=final outfile="/app/output/adcm.csv" dbms=csv replace; run;