yamaa As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.
We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.
Summary
EditUnder reviewWhat this benchmark meansREADME.mdPublic entries: Python and R. Passing coverage requires an installed inventory.
Goal: carry each medication's coded name (CMDECOD) and its
Anatomical Therapeutic Chemical (ATC) classes at levels 1 to 4 into
ADCM, as ATC1-ATC4 (class names) and ATC1CD-ATC4CD (codes),
keeping the medication record sequence (CMSEQ) so each
medication's path stays with its own record.
Input: medication records carrying the reported name (CMTRT)
and the coded name (CMDECOD); Findings About records (FACM)
assigning one ATC level per row - each row names the level
(FATESTCD of ATC1 to ATC4) and the code (FAORRES) for one
medication record; and a small ATC dictionary mapping each code to
its class name.
Variables:
ATC1,ATC2,ATC3,ATC4: the ATC class name at each level, looked up from the dictionary by the level's code.ATC1CD,ATC2CD,ATC3CD,ATC4CD: the ATC code at each level, read from the medication's FACM rows.
Note: a coded medication carries the full four-level path. A
medication with no coded name and no path records keeps all eight
ATC columns empty. Path rows that name no level from ATC1 to
ATC4, or that point at no medication record, change nothing.
Inputs
Under review3 source filesinput/atc_dict.csv
Edit12 rows| ATCLEVEL | ATCCODE | ATCNAME |
|---|---|---|
| 1 | B | BLOOD AND BLOOD FORMING ORGANS |
| 2 | B01 | ANTITHROMBOTIC AGENTS |
| 3 | B01A | ANTITHROMBOTIC AGENTS |
| 4 | B01AC | PLATELET AGGREGATION INHIBITORS EXCL. HEPARIN |
| 1 | C | CARDIOVASCULAR SYSTEM |
| 2 | C10 | LIPID MODIFYING AGENTS |
| 3 | C10A | LIPID MODIFYING AGENTS, PLAIN |
| 4 | C10AA | HMG COA REDUCTASE INHIBITORS |
| 1 | A | ALIMENTARY TRACT AND METABOLISM |
| 2 | A10 | DRUGS USED IN DIABETES |
| 3 | A10B | BLOOD GLUCOSE LOWERING DRUGS, EXCL. INSULINS |
| 4 | A10BA | BIGUANIDES |
input/cm.csv
Edit6 rows| STUDYID | USUBJID | CMSEQ | CMTRT | CMDECOD |
|---|---|---|---|---|
| YAMAA-01 | 1 | ASPIRIN 100MG | ACETYLSALICYLIC ACID | |
| YAMAA-01 | 2 | LIPITOR 20MG | ATORVASTATIN | |
| YAMAA-01 | 3 | HERBAL TEA | — | |
| YAMAA-01 | 1 | ASPIRIN 100MG | ACETYLSALICYLIC ACID | |
| YAMAA-01 | 1 | METFORMIN 500MG | METFORMIN | |
| YAMAA-01 | 2 | GREEN TEA EXTRACT | — |
input/facm.csv
Edit18 rows| STUDYID | USUBJID | FASEQ | FALNKGRP | CMSEQ | FATESTCD | FAORRES |
|---|---|---|---|---|---|---|
| YAMAA-01 | 1 | GRP1 | 1 | ATC1 | B | |
| YAMAA-01 | 2 | GRP1 | 1 | ATC2 | B01 | |
| YAMAA-01 | 3 | GRP1 | 1 | ATC3 | B01A | |
| YAMAA-01 | 4 | GRP1 | 1 | ATC4 | B01AC | |
| YAMAA-01 | 5 | GRP2 | 2 | ATC1 | C | |
| YAMAA-01 | 6 | GRP2 | 2 | ATC2 | C10 | |
| YAMAA-01 | 7 | GRP2 | 2 | ATC3 | C10A | |
| YAMAA-01 | 8 | GRP2 | 2 | ATC4 | C10AA | |
| YAMAA-01 | 9 | GRP3 | 1 | ATC1 | B | |
| YAMAA-01 | 10 | GRP3 | 1 | ATC2 | B01 | |
| YAMAA-01 | 11 | GRP3 | 1 | ATC3 | B01A | |
| YAMAA-01 | 12 | GRP3 | 1 | ATC4 | B01AC | |
| YAMAA-01 | 13 | GRP4 | 1 | ATC1 | A | |
| YAMAA-01 | 14 | GRP4 | 1 | ATC2 | A10 | |
| YAMAA-01 | 15 | GRP4 | 1 | ATC3 | A10B | |
| YAMAA-01 | 16 | GRP4 | 1 | ATC4 | A10BA | |
| YAMAA-01 | 17 | GRP1 | 1 | ATC9 | X | |
| YAMAA-01 | 18 | GRP9 | 99 | ATC1 | B |
Expected output
Under review6 expected rowsexpected/adcm.csv
Edit6 rows| STUDYID | USUBJID | CMSEQ | CMTRT | CMDECOD | ATC1 | ATC2 | ATC3 | ATC4 | ATC1CD | ATC2CD | ATC3CD | ATC4CD |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| YAMAA-01 | 1 | ASPIRIN 100MG | ACETYLSALICYLIC ACID | BLOOD AND BLOOD FORMING ORGANS | ANTITHROMBOTIC AGENTS | ANTITHROMBOTIC AGENTS | PLATELET AGGREGATION INHIBITORS EXCL. HEPARIN | B | B01 | B01A | B01AC | |
| YAMAA-01 | 2 | LIPITOR 20MG | ATORVASTATIN | CARDIOVASCULAR SYSTEM | LIPID MODIFYING AGENTS | LIPID MODIFYING AGENTS, PLAIN | HMG COA REDUCTASE INHIBITORS | C | C10 | C10A | C10AA | |
| YAMAA-01 | 3 | HERBAL TEA | — | — | — | — | — | — | — | — | — | |
| YAMAA-01 | 1 | ASPIRIN 100MG | ACETYLSALICYLIC ACID | BLOOD AND BLOOD FORMING ORGANS | ANTITHROMBOTIC AGENTS | ANTITHROMBOTIC AGENTS | PLATELET AGGREGATION INHIBITORS EXCL. HEPARIN | B | B01 | B01A | B01AC | |
| YAMAA-01 | 1 | METFORMIN 500MG | METFORMIN | ALIMENTARY TRACT AND METABOLISM | DRUGS USED IN DIABETES | BLOOD GLUCOSE LOWERING DRUGS, EXCL. INSULINS | BIGUANIDES | A | A10 | A10B | A10BA | |
| YAMAA-01 | 2 | GREEN TEA EXTRACT | — | — | — | — | — | — | — | — | — |
Human review and assessment
| Dataset Name | Variable Name | Variable Label | Type | Codelist/Controlled Terms | Core | Computational Method | Origin |
|---|---|---|---|---|---|---|---|
| ADCM | STUDYID | Study Identifier | character | Copy value from CM.STUDYID. | Predecessor | ||
| ADCM | USUBJID | Unique Subject Identifier | character | Copy value from CM.USUBJID. | Predecessor | ||
| ADCM | CMSEQ | Sequence Number | numeric | Copy value from CM.CMSEQ. | Predecessor | ||
| ADCM | CMTRT | Reported Name of Drug, Medication, or Therapy | character | Copy value from CM.CMTRT. | Predecessor | ||
| ADCM | CMDECOD | Standardized Medication Name | character | Copy value from CM.CMDECOD. | Predecessor | ||
| ADCM | ATC1CD | ATC Level 1 Code | character | Take FAORRES from the FACM record whose STUDYID equals CM.STUDYID, USUBJID equals CM.USUBJID and CMSEQ equals CM.CMSEQ, where FACM.FATESTCD = 'ATC1', and no match yields blank. | Derived | ||
| ADCM | ATC2CD | ATC Level 2 Code | character | Take FAORRES from the FACM record whose STUDYID equals CM.STUDYID, USUBJID equals CM.USUBJID and CMSEQ equals CM.CMSEQ, where FACM.FATESTCD = 'ATC2', and no match yields blank. | Derived | ||
| ADCM | ATC3CD | ATC Level 3 Code | character | Take FAORRES from the FACM record whose STUDYID equals CM.STUDYID, USUBJID equals CM.USUBJID and CMSEQ equals CM.CMSEQ, where FACM.FATESTCD = 'ATC3', and no match yields blank. | Derived | ||
| ADCM | ATC4CD | ATC Level 4 Code | character | Take FAORRES from the FACM record whose STUDYID equals CM.STUDYID, USUBJID equals CM.USUBJID and CMSEQ equals CM.CMSEQ, where FACM.FATESTCD = 'ATC4', and no match yields blank. | Derived | ||
| ADCM | ATC1 | ATC Level 1 Text | character | Take ATCNAME from the ATCDICT record whose ATCCODE equals ATC1CD, and no match yields blank. | Derived | ||
| ADCM | ATC2 | ATC Level 2 Text | character | Take ATCNAME from the ATCDICT record whose ATCCODE equals ATC2CD, and no match yields blank. | Derived | ||
| ADCM | ATC3 | ATC Level 3 Text | character | Take ATCNAME from the ATCDICT record whose ATCCODE equals ATC3CD, and no match yields blank. | Derived | ||
| ADCM | ATC4 | ATC Level 4 Text | character | Take ATCNAME from the ATCDICT record whose ATCCODE equals ATC4CD, and no match yields blank. | Derived |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.7 | 2026-10-20 | yamaa docs build | Read a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default | ||
| 1.6 | 2026-10-17 | yamaa docs build | Describe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight | ||
| 1.5 | 2026-10-17 | yamaa docs build | Name collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part | ||
| 1.4 | 2026-10-15 | yamaa docs build | State datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks | ||
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/adam-adcm-atc-classes/full.md
Edit19 linesFollowing CDISC ADaM standards, use the provided ATC_DICT, CM, and FACM datasets to create an ADCM dataset with one record per medication record.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, CMSEQ, CMTRT, CMDECOD, ATC1, ATC2, ATC3, ATC4, ATC1CD, ATC2CD, ATC3CD, ATC4CD
ATC1CD, ATC2CD, ATC3CD, and ATC4CD hold the codes in FAORRES on the medication's FACM rows whose FATESTCD is ATC1, ATC2, ATC3, and ATC4, matched to the medication record by STUDYID, USUBJID, and CMSEQ. ATC1, ATC2, ATC3, and ATC4 hold the class name in ATCNAME in the ATC_DICT row whose ATCCODE is the medication's ATC1CD, ATC2CD, ATC3CD, and ATC4CD. A medication with no coded name and no FACM rows for it has no value in ATC1, ATC2, ATC3, ATC4, ATC1CD, ATC2CD, ATC3CD, and ATC4CD. FACM rows whose FATESTCD is none of ATC1 to ATC4, and FACM rows that match no medication record, change nothing.
Read the source datasets from /app/input and save the completed dataset as /app/output/adcm.csv.
prompts/adam-adcm-atc-classes/conventions.md
Edit9 linesFollowing CDISC ADaM standards, use the provided ATC_DICT, CM, and FACM datasets to create an ADCM dataset with one record per medication record.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, CMSEQ, CMTRT, CMDECOD, ATC1, ATC2, ATC3, ATC4, ATC1CD, ATC2CD, ATC3CD, ATC4CD
Read the source datasets from /app/input and save the completed dataset as /app/output/adcm.csv.
prompts/adam-adcm-atc-classes/brief.md
Edit9 linesFollowing CDISC ADaM standards, use the provided ATC_DICT, CM, and FACM datasets to create an ADCM dataset with one record per medication record.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, CMSEQ, CMTRT, CMDECOD, ATC1, ATC2, ATC3, ATC4, ATC1CD, ATC2CD, ATC3CD, ATC4CD
Read the source datasets from /app/input and save the completed dataset as /app/output/adcm.csv.
R
Edit3 lineslibrary(yamaa)yamaa_spec("spec.yaml")$run()$outputPython
Edit3 linesimport yamaayamaa.spec("spec.yaml").run().outputsolutions/adam-adcm-atc-classes/result.R
Edit57 lines# Reference solution for the yamaa benchmark adam-adcm-atc-classes (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)library(tidyr)cm <- read_csv( "/app/input/cm.csv", col_types = cols(CMSEQ = col_integer(), .default = col_character()))facm <- read_csv( "/app/input/facm.csv", col_types = cols( FASEQ = col_integer(), CMSEQ = col_integer(), .default = col_character() ))atcdict <- read_csv( "/app/input/atc_dict.csv", col_types = cols(.default = col_character()))# One ATC level per finding row; rows naming no level from ATC1 to ATC4, or# pointing at no medication record, change nothing.levels <- c("ATC1", "ATC2", "ATC3", "ATC4")codes <- facm |> filter(FATESTCD %in% levels) |> mutate(code_col = paste0(FATESTCD, "CD")) |> select(USUBJID, CMSEQ, code_col, FAORRES) |> pivot_wider(names_from = code_col, values_from = FAORRES) |> mutate(CMSEQ = as.integer(CMSEQ))lookup <- atcdict |> select(ATCCODE, ATCNAME)adcm <- cm |> left_join(codes, by = c("USUBJID", "CMSEQ")) |> # A medication with no coded name keeps every ATC column empty. mutate( across( c(ATC1CD, ATC2CD, ATC3CD, ATC4CD), ~ if_else(is.na(CMDECOD) | CMDECOD == "", NA_character_, .x) ) ) |> left_join(lookup |> rename(ATC1CD = ATCCODE, ATC1 = ATCNAME), by = "ATC1CD") |> left_join(lookup |> rename(ATC2CD = ATCCODE, ATC2 = ATCNAME), by = "ATC2CD") |> left_join(lookup |> rename(ATC3CD = ATCCODE, ATC3 = ATCNAME), by = "ATC3CD") |> left_join(lookup |> rename(ATC4CD = ATCCODE, ATC4 = ATCNAME), by = "ATC4CD") |> select( STUDYID, USUBJID, CMSEQ, CMTRT, CMDECOD, ATC1, ATC2, ATC3, ATC4, ATC1CD, ATC2CD, ATC3CD, ATC4CD )dir.create("/app/output", showWarnings = FALSE)write_csv(adcm, "/app/output/adcm.csv", na = "")solutions/adam-adcm-atc-classes/result.py
Edit90 lines# Reference solution for the yamaa benchmark adam-adcm-atc-classes (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1."""Create ADCM with ATC class levels 1-4 per medication record.Reads CM, FACM and ATC_DICT from /app/input and writes adcm.csv to/app/output with one record per CM medication record."""import polars as plINPUT_DIR = "/app/input"OUTPUT_DIR = "/app/output"LEVELS = ["ATC1", "ATC2", "ATC3", "ATC4"]OUT_COLUMNS = [ "STUDYID", "USUBJID", "CMSEQ", "CMTRT", "CMDECOD", "ATC1", "ATC2", "ATC3", "ATC4", "ATC1CD", "ATC2CD", "ATC3CD", "ATC4CD",]def read_strings(name: str) -> pl.DataFrame: """Read a CSV with every column as a string, preserving blanks.""" return pl.read_csv(f"{INPUT_DIR}/{name}", infer_schema_length=0)def main() -> None: cm = read_strings("cm.csv") facm = read_strings("facm.csv") atc_dict = read_strings("atc_dict.csv") # ATC code -> class name lookup. code_to_name = dict( zip( atc_dict["ATCCODE"].to_list(), atc_dict["ATCNAME"].to_list(), ) ) # (STUDYID, USUBJID, CMSEQ, FATESTCD) -> FAORRES code, for ATC1-ATC4 # FACM rows only. FACM rows for other tests are ignored; rows that # match no medication record simply never get looked up. facm_codes = {} for row in facm.iter_rows(named=True): if row["FATESTCD"] in LEVELS: key = (row["STUDYID"], row["USUBJID"], row["CMSEQ"], row["FATESTCD"]) facm_codes[key] = row["FAORRES"] records = [] for row in cm.iter_rows(named=True): codes = {} for level in LEVELS: key = (row["STUDYID"], row["USUBJID"], row["CMSEQ"], level) codes[level] = facm_codes.get(key) or None record = { "STUDYID": row["STUDYID"], "USUBJID": row["USUBJID"], "CMSEQ": row["CMSEQ"], "CMTRT": row["CMTRT"], "CMDECOD": row["CMDECOD"] or None, } for level in LEVELS: code = codes[level] record[level] = code_to_name.get(code) if code else None for level in LEVELS: record[f"{level}CD"] = codes[level] records.append(record) adcm = pl.DataFrame( records, schema={col: pl.Utf8 for col in OUT_COLUMNS} ) adcm.write_csv(f"{OUTPUT_DIR}/adcm.csv")if __name__ == "__main__": main()solutions/adam-adcm-atc-classes/result.sas
Edit39 lines/* opensas reference solution for the yamaa benchmark adam-adcm-atc-classes. Reads the benchmark inputs and derives the requested output. Execute with the existing opensas runtime: opensas /app/output/result.sas. */data atc_dict; length ATCLEVEL ATCCODE ATCNAME $1024; infile "/app/input/atc_dict.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input ATCLEVEL : $char1024. ATCCODE : $char1024. ATCNAME : $char1024.;run;data cm; length STUDYID USUBJID CMTRT CMDECOD $1024; infile "/app/input/cm.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. CMSEQ CMTRT : $char1024. CMDECOD : $char1024.;run;data facm; length STUDYID USUBJID FALNKGRP FATESTCD FAORRES $1024; infile "/app/input/facm.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. FASEQ FALNKGRP : $char1024. CMSEQ FATESTCD : $char1024. FAORRES : $char1024.;run;proc sql; create table result as select a.*, f1.FAORRES as ATC1CD, d1.ATCNAME as ATC1, f2.FAORRES as ATC2CD, d2.ATCNAME as ATC2, f3.FAORRES as ATC3CD, d3.ATCNAME as ATC3, f4.FAORRES as ATC4CD, d4.ATCNAME as ATC4 from cm as aleft join facm as f1 on a.STUDYID=f1.STUDYID and a.USUBJID=f1.USUBJID and a.CMSEQ=f1.CMSEQ and f1.FATESTCD='ATC1' and a.CMDECOD ne ''left join atc_dict as d1 on d1.ATCLEVEL='1' and d1.ATCCODE=f1.FAORRESleft join facm as f2 on a.STUDYID=f2.STUDYID and a.USUBJID=f2.USUBJID and a.CMSEQ=f2.CMSEQ and f2.FATESTCD='ATC2' and a.CMDECOD ne ''left join atc_dict as d2 on d2.ATCLEVEL='2' and d2.ATCCODE=f2.FAORRESleft join facm as f3 on a.STUDYID=f3.STUDYID and a.USUBJID=f3.USUBJID and a.CMSEQ=f3.CMSEQ and f3.FATESTCD='ATC3' and a.CMDECOD ne ''left join atc_dict as d3 on d3.ATCLEVEL='3' and d3.ATCCODE=f3.FAORRESleft join facm as f4 on a.STUDYID=f4.STUDYID and a.USUBJID=f4.USUBJID and a.CMSEQ=f4.CMSEQ and f4.FATESTCD='ATC4' and a.CMDECOD ne ''left join atc_dict as d4 on d4.ATCLEVEL='4' and d4.ATCCODE=f4.FAORRES;quit;proc sql; create table final as select STUDYID, USUBJID, CMSEQ, CMTRT, CMDECOD, ATC1, ATC2, ATC3, ATC4, ATC1CD, ATC2CD, ATC3CD, ATC4CD from result;quit;proc export data=final outfile="/app/output/adcm.csv" dbms=csv replace; run;