yamaa / Benchmark

ADaM.ADSL

Normalize the Collected Country and Assign Its Region

input files
1
subjects
9
expected rows
9
Review ask

As a clinical trial statistician or programmer, challenge anything in this benchmark: describe a problem in Comments, or propose a fix with the Edit button on any source file.

  1. Task: Would a real study need this derivation, specified this way? Does it follow CDISC ADaM: variable names, controlled terms, record structure? Flag any rule that is ambiguous or that a sponsor would handle differently.
  2. Inputs: Do they look like data collected in a real trial, and do they include the situations you would expect to meet in one?
  3. Expected output: Would independent QC programming reproduce every value from these inputs and rules?

The other tabs beside Comments are background for assessment, not under review.

Summary

Lifecycle: reviewedEditUnder reviewWhat this benchmark means
README.md

Goal: give each subject a normalized country in COUNTRY and its region in REGION1.

Input: demographics (DM) rows carrying the collected country (COUNTRY) as reported, in mixed case or empty, including a country outside the three listed below.

Variables:

  • COUNTRY is the collected country from demographics in upper case; a subject with no collected country is UNKNOWN. Every collected country passes through: there is no allowed-countries check, so the run never stops on a new country.
  • REGION1 is the region for that country: USA and CAN give North America, DEU gives Europe, and any other country (including UNKNOWN) gives Rest of World.

Note: case differences never split a country (usa and USA are one), and the fallback values mean every subject has both a country and a region.

Inputs

Under review1 source file

input/dm.csv

Edit9 rows
STUDYIDUSUBJIDCOUNTRY
YAMAA-01usa
YAMAA-01CAN
YAMAA-01deu
YAMAA-01—
YAMAA-01UsA
YAMAA-01cAn
YAMAA-01unknown
YAMAA-01—
YAMAA-01fra

Expected output

Under review9 expected rows

expected/adsl.csv

Edit9 rows
STUDYIDUSUBJIDCOUNTRYREGION1
YAMAA-01USANorth America
YAMAA-01CANNorth America
YAMAA-01DEUEurope
YAMAA-01UNKNOWNRest of World
YAMAA-01USANorth America
YAMAA-01CANNorth America
YAMAA-01UNKNOWNRest of World
YAMAA-01UNKNOWNRest of World
YAMAA-01FRARest of World
Shaded columns are derived— = missing value

Comments and assessment materials

Comments

Signed in with GitHub; saved in repository Discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Dataset NameVariable NameVariable LabelTypeCodelist/Controlled TermsCoreComputational MethodOrigin
ADSLSTUDYIDStudy IdentifiercharacterCopy value from DM.STUDYID.Predecessor
ADSLUSUBJIDUnique Subject IdentifiercharacterCopy value from DM.USUBJID.Predecessor
ADSLCOUNTRYCountrycharacterConvert DM.COUNTRY to upper case; a missing source → "UNKNOWN".Derived
ADSLREGION1Geographic Region 1characterNorth America, Europe, Rest of WorldRecode COUNTRY: "USA" → "North America", "CAN" → "North America", "DEU" → "Europe"; missing or unlisted values → "Rest of World".Derived
VariablePermitted values
REGION1North America, Europe, Rest of World
VersionDateAuthorDescriptionReviewerSign-off
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/adam-adsl-country-region/full.md

Edit17 lines

Following CDISC ADaM standards, use the provided DM dataset to create an ADSL dataset with one record per subject.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, COUNTRY, REGION1

COUNTRY is the collected country in upper case, whichever country it is; a subject with no collected country is UNKNOWN. Case differences never split a country.

REGION1 is the region for that country: USA and CAN give "North America", DEU gives "Europe", and any other country (including UNKNOWN) gives "Rest of World". Every subject has both a country and a region.

Read the source datasets from /app/input and save the completed dataset as /app/output/adsl.csv.

prompts/adam-adsl-country-region/conventions.md

Edit13 lines

Following CDISC ADaM standards, use the provided DM dataset to create an ADSL dataset with one record per subject.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, COUNTRY, REGION1

COUNTRY is in upper case, or UNKNOWN.

REGION1 is "North America" for USA and CAN, "Europe" for DEU, or "Rest of World".

Read the source datasets from /app/input and save the completed dataset as /app/output/adsl.csv.

prompts/adam-adsl-country-region/brief.md

Edit8 lines

Following CDISC ADaM standards, use the provided DM dataset to create an ADSL dataset with one record per subject.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, COUNTRY, REGION1

Read the source datasets from /app/input and save the completed dataset as /app/output/adsl.csv.

yamaa code

Runs the yamaa specification; its result must match the expected output

run.R

Edit4 lines
library(yamaa)adsl <- yamaa_domain("spec.yaml")$outputadsl

run.py

Edit4 lines
import yamaaadsl = yamaa.yamaa_domain("spec.yaml").outputadsl

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/adam-adsl-country-region/result.R

Edit29 lines
# Reference solution for the yamaa benchmark adam-adsl-country-region (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)library(stringr)dm <- read_csv(  "/app/input/dm.csv",  col_types = cols(.default = col_character()))adsl <- dm |>  mutate(    COUNTRY = str_trim(coalesce(COUNTRY, "")),    COUNTRY = if_else(COUNTRY == "", "UNKNOWN", str_to_upper(COUNTRY)),    REGION1 = case_when(      COUNTRY %in% c("USA", "CAN") ~ "North America",      COUNTRY == "DEU" ~ "Europe",      .default = "Rest of World"    )  ) |>  select(STUDYID, USUBJID, COUNTRY, REGION1)dir.create("/app/output", showWarnings = FALSE)write_csv(adsl, "/app/output/adsl.csv", na = "")

solutions/adam-adsl-country-region/result.py

Edit33 lines
# Reference solution for the yamaa benchmark adam-adsl-country-region (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.from pathlib import Pathimport polars as pldm = pl.read_csv("/app/input/dm.csv", infer_schema=False)adsl = (    dm.with_columns(        COUNTRY=pl.col("COUNTRY").fill_null("").str.strip_chars().str.to_uppercase()    )    .with_columns(        COUNTRY=pl.when(pl.col("COUNTRY") == "")        .then(pl.lit("UNKNOWN"))        .otherwise(pl.col("COUNTRY"))    )    .with_columns(        REGION1=pl.when(pl.col("COUNTRY").is_in(["USA", "CAN"]))        .then(pl.lit("North America"))        .when(pl.col("COUNTRY") == "DEU")        .then(pl.lit("Europe"))        .otherwise(pl.lit("Rest of World"))    )    .select("STUDYID", "USUBJID", "COUNTRY", "REGION1"))Path("/app/output").mkdir(parents=True, exist_ok=True)adsl.write_csv("/app/output/adsl.csv")

solutions/adam-adsl-country-region/result.sas

Edit24 lines
/* opensas reference solution for the yamaa benchmark adam-adsl-country-region.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data dm;  length STUDYID USUBJID COUNTRY $1024;  infile "/app/input/dm.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. COUNTRY : $char1024.;run;data result;  length REGION1 $20;  set dm;  COUNTRY=upcase(COUNTRY);  if missing(COUNTRY) then COUNTRY='UNKNOWN';  if COUNTRY in ('USA','CAN') then REGION1='North America';  else if COUNTRY='DEU' then REGION1='Europe';  else REGION1='Rest of World';run;proc sql;  create table final as select STUDYID, USUBJID, COUNTRY, REGION1 from result;quit;proc export data=final outfile="/app/output/adsl.csv" dbms=csv replace; run;