Record race and ethnicity, with one supplemental record per race

Review ask

As a clinical trial statistician or programmer, challenge anything in this benchmark: describe a problem in Comments, or propose a fix with the Edit button on any source file.

Task: Would a real study need this derivation, specified this way? Does it follow CDISC SDTM: variable names, controlled terms, record structure? Flag any rule that is ambiguous or that a sponsor would handle differently.
Inputs: Do they look like data collected in a real trial, and do they include the situations you would expect to meet in one?
Expected output: Would independent QC programming reproduce every value from these inputs and rules?

The other tabs beside Comments are background for assessment, not under review.

Summary

EditUnder reviewWhat this benchmark means

README.md

Goal: build one Demographics (DM) record per subject carrying race (RACE) and ethnicity (ETHNIC), and one supplemental Demographics (SUPPDM) record per reported race for each subject who marked several.

Input: electronic data capture (EDC) output in long form, one row per collected item; each race option a subject marks contributes one race row, and ethnicity contributes its own row. A race marked twice counts once.

Variables:

RACE (DM): reported race in controlled terms: WHITE, ASIAN, BLACK OR AFRICAN AMERICAN, AMERICAN INDIAN OR ALASKA NATIVE, NATIVE HAWAIIAN OR OTHER PACIFIC ISLANDER, MULTIPLE when several races were marked, OTHER for a listed free-text answer that fits none of the named races, UNKNOWN for a refused answer, and NOT REPORTED for a recorded non-answer; blank when no race item was marked or a single answer is not in the sponsor's list.
ETHNIC (DM): reported ethnicity in controlled terms: HISPANIC OR LATINO, NOT HISPANIC OR LATINO, NOT REPORTED, or UNKNOWN; blank when no ethnicity item was collected or its answer is not in the sponsor's list.
IDVARVAL (SUPPDM): the subject identifier, linking the record to its DM row through IDVAR = USUBJID.
QNAM / QLABEL (SUPPDM): RACE1 / Race 1, RACE2 / Race 2, and so on, one per distinct race the subject marked, numbered in alphabetical order of the collected answer because the input carries no report order.
QVAL (SUPPDM): that race in controlled terms, as in RACE.
QORIG (SUPPDM): the origin, always CRF; QEVAL is blank.

Note: SUPPDM carries race records exactly for the subjects whose DM record says MULTIPLE, however many races they marked.

YAML specification

2 spec files

spec_dm.yaml
Edit120 lines
1schema_version: "1.0"2domain: DM3keys: [STUDYID, USUBJID]45input:6  ODM: input/odm.csv78output:9  path: dm.csv10  columns: [DOMAIN, STUDYID, USUBJID, SUBJID, RACE, ETHNIC]1112columns:13  - name: DOMAIN14    type: str15    label: Domain Abbreviation16    derivation: {literal: DM}1718  - name: STUDYID19    type: str20    label: Study Identifier21    derivation: ODM.StudyOID2223  - name: USUBJID24    type: str25    label: Unique Subject Identifier26    derivation: ODM.SubjectKey2728  - name: SUBJID29    type: str30    label: Subject Identifier for the Study31    derivation: USUBJID3233  # The alphabetically first and last race a subject marked. They differ34  # exactly when several distinct races were marked, and both are blank when35  # no race was marked.36  - name: RACEMIN37    type: str38    label: Alphabetically First Reported Race39    derivation:40      aggregate:41        key: {StudyOID: STUDYID, SubjectKey: USUBJID}42        filter: "ODM.ItemOID = 'IT.DM.RACE'"43        expr: "MIN(ODM.Value)"4445  - name: RACESINGLE46    type: str47    label: Reported Race When One Race Was Reported48    derivation:49      aggregate:50        key: {StudyOID: STUDYID, SubjectKey: USUBJID}51        filter: "ODM.ItemOID = 'IT.DM.RACE'"52        expr: "MAX(ODM.Value)"5354  # One ethnicity record per subject; a second one stops the run.55  - name: ETHNICRAW56    type: str57    label: Collected Ethnicity58    derivation:59      aggregate:60        key: {StudyOID: STUDYID, SubjectKey: USUBJID}61        filter: "ODM.ItemOID = 'IT.DM.ETHNIC'"62        expr: "ONLY(ODM.Value)"6364  # Several distinct races give MULTIPLE. Otherwise the one race is mapped to65  # controlled terms, and no race or an answer not listed gives a blank.66  - name: RACE67    type: str68    label: Race69    derivation:70      case:71        - when: "RACEMIN < RACESINGLE"72          then: {literal: MULTIPLE}73        - otherwise:74            mapping:75              source: RACESINGLE76              dict:77                "American Indian or Alaska Native": AMERICAN INDIAN78                  OR ALASKA NATIVE79                "Asian": ASIAN80                "Black or African American": BLACK OR AFRICAN AMERICAN81                "Native Hawaiian or Other Pacific Islander": NATIVE HAWAIIAN82                  OR OTHER PACIFIC ISLANDER83                "White": WHITE84                "Other, specify: Fijian": OTHER85                "Not reported": NOT REPORTED86                "Subject refused": UNKNOWN87              missing: null88              unmapped: null89    verifications:90      allowed_values:91        values: [AMERICAN INDIAN OR ALASKA NATIVE, ASIAN,92          BLACK OR AFRICAN AMERICAN,93          NATIVE HAWAIIAN OR OTHER PACIFIC ISLANDER, WHITE, MULTIPLE,94          OTHER, NOT REPORTED, UNKNOWN]9596  # No ethnicity record, or an answer not listed, gives a blank.97  - name: ETHNIC98    type: str99    label: Ethnicity100    derivation:101      mapping:102        source: ETHNICRAW103        dict:104          "Hispanic or Latino": HISPANIC OR LATINO105          "Not Hispanic or Latino": NOT HISPANIC OR LATINO106          "Not reported": NOT REPORTED107          "Unknown": UNKNOWN108        missing: null109        unmapped: null110    verifications:111      allowed_values:112        values: [HISPANIC OR LATINO, NOT HISPANIC OR LATINO,113          NOT REPORTED, UNKNOWN]114115verifications:116  - unique:117      columns: [STUDYID, USUBJID]118  - row_count:119      min: 12120      max: 12
spec_suppdm.yaml
Edit155 lines
1schema_version: "1.0"2domain: SUPPDM3keys: [STUDYID, RDOMAIN, USUBJID, IDVAR, IDVARVAL, QNAM]45input:6  ODM: input/odm.csv7  # DM is built by spec_dm.yaml in the same run, so the subjects that get8  # supplemental race records are exactly the ones DM calls MULTIPLE.9  DM:10    path: dm.csv11    schema: spec_dm.yaml1213output:14  path: suppdm.csv15  columns: [STUDYID, RDOMAIN, USUBJID, IDVAR, IDVARVAL, QNAM, QLABEL, QVAL,16            QORIG, QEVAL]17  order_by: [STUDYID, RDOMAIN, USUBJID, IDVAR, IDVARVAL, QNAM]1819intermediates:20  - id: DMREC21    dataset: DM22    key: {STUDYID: ODM.StudyOID, USUBJID: ODM.SubjectKey}2324columns:25  - name: STUDYID26    type: str27    label: Study Identifier2829  - name: RDOMAIN30    type: str31    label: Related Domain Abbreviation32    derivation: {literal: DM}3334  - name: USUBJID35    type: str36    label: Unique Subject Identifier3738  - name: IDVAR39    type: str40    label: Identifying Variable41    derivation: {literal: USUBJID}4243  - name: IDVARVAL44    type: str45    label: Identifying Variable Value46    derivation: USUBJID47    verifications:48      not_missing: {}4950  - name: ITEMOID51    type: str52    label: Collected Item5354  - name: DMRACE55    type: str56    label: Race in Demographics5758  - name: QVALRAW59    type: str60    label: Collected Race6162  # Races are numbered in alphabetical order of the collected answer; the63  # input carries no report order.64  - name: RACENUM65    type: int66    label: Race Number Within Subject67    derivation:68      row_number:69        window:70          group_by: [STUDYID, USUBJID]71          order_by: [QVALRAW]7273  # str_template reads text only, so the number takes its text form first.74  - name: RACENUMC75    type: str76    label: Race Number Within Subject as Text77    derivation: RACENUM7879  - name: QNAM80    type: str81    label: Qualifier Variable Name82    derivation:83      str_template: "RACE{RACENUMC}"84    verifications:85      matches:86        pattern: "^RACE[1-9][0-9]*$"8788  - name: QLABEL89    type: str90    label: Qualifier Label91    derivation:92      str_template: "Race {RACENUMC}"9394  # The same race dictionary as RACE in spec_dm.yaml.95  - name: QVAL96    type: str97    label: Data Value98    derivation:99      mapping:100        source: QVALRAW101        dict:102          "American Indian or Alaska Native": AMERICAN INDIAN OR ALASKA NATIVE103          "Asian": ASIAN104          "Black or African American": BLACK OR AFRICAN AMERICAN105          "Native Hawaiian or Other Pacific Islander": NATIVE HAWAIIAN106            OR OTHER PACIFIC ISLANDER107          "White": WHITE108          "Other, specify: Fijian": OTHER109          "Not reported": NOT REPORTED110          "Subject refused": UNKNOWN111        missing: null112    verifications:113      - not_missing: {}114      - allowed_values:115          values: [AMERICAN INDIAN OR ALASKA NATIVE, ASIAN,116            BLACK OR AFRICAN AMERICAN,117            NATIVE HAWAIIAN OR OTHER PACIFIC ISLANDER, WHITE,118            OTHER, NOT REPORTED, UNKNOWN]119120  - name: QORIG121    type: str122    label: Origin123    derivation: {literal: CRF}124125  - name: QEVAL126    type: str127    label: Evaluator128    derivation: {literal: null}129130rows:131  # One candidate per distinct collected value of each subject's items, so a132  # race marked twice still yields one record. Only race values of subjects133  # DM calls MULTIPLE are kept, however many races they reported.134  - id: reported_race135    dataset: ODM136    group_by: [ODM.StudyOID, ODM.SubjectKey, ODM.ItemOID, ODM.Value]137    filter: "ITEMOID = 'IT.DM.RACE' AND DMRACE = 'MULTIPLE'"138    derivations:139      STUDYID: ODM.StudyOID140      USUBJID: ODM.SubjectKey141      ITEMOID: ODM.ItemOID142      QVALRAW: ODM.Value143      DMRACE: DMREC.RACE144145verifications:146  - unique:147      columns: [STUDYID, RDOMAIN, USUBJID, IDVAR, IDVARVAL, QNAM]148  - row_count:149      min: 11150      max: 11151  - assert:152      id: race-value-has-parent-identifier153      when: "QVAL IS NOT NULL"154      require: "RDOMAIN IS NOT NULL AND IDVAR IS NOT NULL155        AND IDVARVAL IS NOT NULL"

Inputs

Under review1 source file

input/odm.csv

Edit30 rows

input/odm.csv
StudyOID	MetaDataVersionOID	SubjectKey	StudyEventOID	StudyEventRepeatKey	ItemGroupOID	ItemGroupRepeatKey	ItemOID	Value
YAMAA-01	MDV.1	YAMAA-01-101	SCREENING	1	IG.DM	1	IT.DM.RACE	White
YAMAA-01	MDV.1	YAMAA-01-101	SCREENING	1	IG.DM	1	IT.DM.ETHNIC	Not Hispanic or Latino
YAMAA-01	MDV.1	YAMAA-01-102	SCREENING	1	IG.DM	1	IT.DM.RACE	Asian
YAMAA-01	MDV.1	YAMAA-01-102	SCREENING	1	IG.DM	1	IT.DM.RACE	Black or African American
YAMAA-01	MDV.1	YAMAA-01-102	SCREENING	1	IG.DM	1	IT.DM.ETHNIC	Hispanic or Latino
YAMAA-01	MDV.1	YAMAA-01-103	SCREENING	1	IG.DM	1	IT.DM.ETHNIC	Not Hispanic or Latino
YAMAA-01	MDV.1	YAMAA-01-104	SCREENING	1	IG.DM	1	IT.DM.RACE	American Indian or Alaska Native
YAMAA-01	MDV.1	YAMAA-01-104	SCREENING	1	IG.DM	1	IT.DM.RACE	White
YAMAA-01	MDV.1	YAMAA-01-104	SCREENING	1	IG.DM	1	IT.DM.ETHNIC	Not reported
YAMAA-01	MDV.1	YAMAA-01-105	SCREENING	1	IG.DM	1	IT.DM.RACE	Other, specify: Fijian
YAMAA-01	MDV.1	YAMAA-01-105	SCREENING	1	IG.DM	1	IT.DM.ETHNIC	Unknown
YAMAA-01	MDV.1	YAMAA-01-106	SCREENING	1	IG.DM	1	IT.DM.RACE	Subject refused
YAMAA-01	MDV.1	YAMAA-01-106	SCREENING	1	IG.DM	1	IT.DM.ETHNIC	Hispanic or Latino
YAMAA-01	MDV.1	YAMAA-01-107	SCREENING	1	IG.DM	1	IT.DM.RACE	Asian
YAMAA-01	MDV.1	YAMAA-01-107	SCREENING	1	IG.DM	1	IT.DM.RACE	Black or African American
YAMAA-01	MDV.1	YAMAA-01-107	SCREENING	1	IG.DM	1	IT.DM.RACE	White
YAMAA-01	MDV.1	YAMAA-01-107	SCREENING	1	IG.DM	1	IT.DM.ETHNIC	Not Hispanic or Latino
YAMAA-01	MDV.1	YAMAA-01-108	SCREENING	1	IG.DM	1	IT.DM.RACE	White
YAMAA-01	MDV.1	YAMAA-01-108	SCREENING	1	IG.DM	1	IT.DM.RACE	White
YAMAA-01	MDV.1	YAMAA-01-108	SCREENING	1	IG.DM	1	IT.DM.ETHNIC	Not Hispanic or Latino
YAMAA-01	MDV.1	YAMAA-01-109	SCREENING	1	IG.DM	1	IT.DM.RACE	White
YAMAA-01	MDV.1	YAMAA-01-109	SCREENING	1	IG.DM	1	IT.DM.RACE	Asian
YAMAA-01	MDV.1	YAMAA-01-109	SCREENING	1	IG.DM	1	IT.DM.RACE	White
YAMAA-01	MDV.1	YAMAA-01-109	SCREENING	1	IG.DM	1	IT.DM.ETHNIC	Not Hispanic or Latino
YAMAA-01	MDV.1	YAMAA-01-110	SCREENING	1	IG.DM	1	IT.DM.RACE	Not reported
YAMAA-01	MDV.1	YAMAA-01-111	SCREENING	1	IG.DM	1	IT.DM.RACE	Pacific Islander
YAMAA-01	MDV.1	YAMAA-01-111	SCREENING	1	IG.DM	1	IT.DM.ETHNIC	Prefer not to answer
YAMAA-01	MDV.1	YAMAA-01-112	SCREENING	1	IG.DM	1	IT.DM.RACE	Not reported
YAMAA-01	MDV.1	YAMAA-01-112	SCREENING	1	IG.DM	1	IT.DM.RACE	White
YAMAA-01	MDV.1	YAMAA-01-112	SCREENING	1	IG.DM	1	IT.DM.ETHNIC	Not Hispanic or Latino

Expected output

Under review23 expected rows

expected/dm.csv

Edit12 rows

expected/dm.csv
DOMAIN	STUDYID	SUBJID	RACE	ETHNIC
DM	YAMAA-01	YAMAA-01-101	WHITE	NOT HISPANIC OR LATINO
DM	YAMAA-01	YAMAA-01-102	MULTIPLE	HISPANIC OR LATINO
DM	YAMAA-01	YAMAA-01-103	—	NOT HISPANIC OR LATINO
DM	YAMAA-01	YAMAA-01-104	MULTIPLE	NOT REPORTED
DM	YAMAA-01	YAMAA-01-105	OTHER	UNKNOWN
DM	YAMAA-01	YAMAA-01-106	UNKNOWN	HISPANIC OR LATINO
DM	YAMAA-01	YAMAA-01-107	MULTIPLE	NOT HISPANIC OR LATINO
DM	YAMAA-01	YAMAA-01-108	WHITE	NOT HISPANIC OR LATINO
DM	YAMAA-01	YAMAA-01-109	MULTIPLE	NOT HISPANIC OR LATINO
DM	YAMAA-01	YAMAA-01-110	NOT REPORTED	—
DM	YAMAA-01	YAMAA-01-111	—	—
DM	YAMAA-01	YAMAA-01-112	MULTIPLE	NOT HISPANIC OR LATINO

expected/suppdm.csv

Edit11 rows

expected/suppdm.csv
STUDYID	RDOMAIN	IDVAR	IDVARVAL	QNAM	QLABEL	QVAL	QORIG	QEVAL
YAMAA-01	DM	USUBJID	YAMAA-01-102	RACE1	Race 1	ASIAN	CRF	—
YAMAA-01	DM	USUBJID	YAMAA-01-102	RACE2	Race 2	BLACK OR AFRICAN AMERICAN	CRF	—
YAMAA-01	DM	USUBJID	YAMAA-01-104	RACE1	Race 1	AMERICAN INDIAN OR ALASKA NATIVE	CRF	—
YAMAA-01	DM	USUBJID	YAMAA-01-104	RACE2	Race 2	WHITE	CRF	—
YAMAA-01	DM	USUBJID	YAMAA-01-107	RACE1	Race 1	ASIAN	CRF	—
YAMAA-01	DM	USUBJID	YAMAA-01-107	RACE2	Race 2	BLACK OR AFRICAN AMERICAN	CRF	—
YAMAA-01	DM	USUBJID	YAMAA-01-107	RACE3	Race 3	WHITE	CRF	—
YAMAA-01	DM	USUBJID	YAMAA-01-109	RACE1	Race 1	ASIAN	CRF	—
YAMAA-01	DM	USUBJID	YAMAA-01-109	RACE2	Race 2	WHITE	CRF	—
YAMAA-01	DM	USUBJID	YAMAA-01-112	RACE1	Race 1	NOT REPORTED	CRF	—
YAMAA-01	DM	USUBJID	YAMAA-01-112	RACE2	Race 2	WHITE	CRF	—

Shaded columns are derived— = missing value

Comments and assessment materials

Variable Name	Variable Label	Type	Controlled Terms or Format	Origin	Conversion Definition	Variable Type	Variable Order
STUDYID	Study Identifier	Char		Derived	Set per constructed row; see the Row construction sheet.	SUPP	1
RDOMAIN	Related Domain Abbreviation	Char		Assigned	Set constant value "DM".	SUPP	2
USUBJID	Unique Subject Identifier	Char		Derived	Set per constructed row; see the Row construction sheet.	SUPP	3
IDVAR	Identifying Variable	Char		Assigned	Set constant value "USUBJID".	SUPP	4
IDVARVAL	Identifying Variable Value	Char		Derived	Copy value from USUBJID.	SUPP	5
QNAM	Qualifier Variable Name	Char		Derived	Fill the template "RACE{RACENUMC}".	SUPP	6
QLABEL	Qualifier Label	Char		Derived	Fill the template "Race {RACENUMC}".	SUPP	7
QVAL	Data Value	Char	AMERICAN INDIAN OR ALASKA NATIVE, ASIAN, BLACK OR AFRICAN AMERICAN, NATIVE HAWAIIAN OR OTHER PACIFIC ISLANDER, WHITE, OTHER, NOT REPORTED, UNKNOWN	Derived	Recode QVALRAW: "American Indian or Alaska Native" → "AMERICAN INDIAN OR ALASKA NATIVE", "Asian" → "ASIAN", "Black or African American" → "BLACK OR AFRICAN AMERICAN", "Native Hawaiian or Other Pacific Islander" → "NATIVE HAWAIIAN OR OTHER PACIFIC ISLANDER", "White" → "WHITE", "Other, specify: Fijian" → "OTHER", "Not reported" → "NOT REPORTED", "Subject refused" → "UNKNOWN"; missing values stay missing; unlisted values are errors.	SUPP	8
QORIG	Origin	Char		Assigned	Set constant value "CRF".	SUPP	9
QEVAL	Evaluator	Char		Assigned	Set constant value "None".	SUPP	10

Template	Include when	STUDYID	USUBJID	ITEMOID	QVALRAW	DMRACE
reported_race	ITEMOID = 'IT.DM.RACE' AND DMRACE = 'MULTIPLE'	Copy ODM.StudyOID	Copy ODM.SubjectKey	Copy ODM.ItemOID	Copy ODM.Value	Copy RACE from the DM record whose STUDYID equals ODM.StudyOID and USUBJID equals ODM.SubjectKey, and a row with no match is an error

Variable	Permitted values
QVAL	AMERICAN INDIAN OR ALASKA NATIVE, ASIAN, BLACK OR AFRICAN AMERICAN, NATIVE HAWAIIAN OR OTHER PACIFIC ISLANDER, WHITE, OTHER, NOT REPORTED, UNKNOWN

Version	Date	Author	Description
1.3	2026-09-29	yamaa docs build	Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.2	2026-09-29	yamaa docs build	Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.1	2026-09-29	yamaa docs build	Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0		yamaa docs build	Initial generation from spec.yaml

prompts/sdtm-dm-race-ethnicity/full.md

Edit42 lines

Following CDISC SDTM standards, use the provided ODM dataset to create a DM dataset with one record per subject and a SUPPDM dataset with one record per reported race for each subject who marked several.

The DM dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, SUBJID, RACE, ETHNIC The SUPPDM dataset should contain the following columns in this order: STUDYID, RDOMAIN, USUBJID, IDVAR, IDVARVAL, QNAM, QLABEL, QVAL, QORIG, QEVAL

STUDYID is the ODM StudyOID and USUBJID is the SubjectKey, each as given.

Each collected race answer maps to a controlled term:

"White", "Asian", "Black or African American", "American Indian or Alaska Native", and "Native Hawaiian or Other Pacific Islander" give the same name in upper case;
"Other, specify: Fijian" gives OTHER;
"Subject refused" gives UNKNOWN;
"Not reported" gives NOT REPORTED;
any other answer is not in the list and gives no value.

Every distinct answer a subject gave counts as one race, a non-answer such as "Not reported" included, and a race marked twice counts once. RACE is MULTIPLE when a subject has two or more races, and otherwise the mapped term of the single answer; it has no value when no race was marked or the single answer is not in the list. ETHNIC maps the collected answer: "Hispanic or Latino" gives HISPANIC OR LATINO, "Not Hispanic or Latino" gives NOT HISPANIC OR LATINO, "Not reported" gives NOT REPORTED, and "Unknown" gives UNKNOWN. ETHNIC has no value when no ethnicity was collected or its answer is not in the list. SUBJID repeats USUBJID.

SUPPDM carries records exactly for the subjects whose DM record says MULTIPLE, one per race. RDOMAIN is DM, IDVAR is USUBJID, and IDVARVAL is the subject identifier. QNAM is RACE1, RACE2, and so on, with QLABEL Race 1, Race 2, and so on, numbered in alphabetical order of the collected answer. QVAL maps the race as RACE does. QORIG is always CRF, and QEVAL has no value.

Read the source datasets from /app/input and save the completed DM dataset as /app/output/dm.csv and the completed SUPPDM dataset as /app/output/suppdm.csv.

prompts/sdtm-dm-race-ethnicity/conventions.md

Edit20 lines

Each collected race answer gives the same name in upper case, except that "Other, specify: Fijian" gives OTHER and "Subject refused" gives UNKNOWN; ETHNIC maps the collected answer to the same name in upper case. In SUPPDM, IDVAR is USUBJID; QNAM and QLABEL are RACE1 and Race 1, RACE2 and Race 2, and so on, in alphabetical order of the collected answer; QVAL maps the race as RACE does; and QORIG is CRF.

Read the source datasets from /app/input and save the completed DM dataset as /app/output/dm.csv and the completed SUPPDM dataset as /app/output/suppdm.csv.

prompts/sdtm-dm-race-ethnicity/brief.md

Edit13 lines

Read the source datasets from /app/input and save the completed DM dataset as /app/output/dm.csv and the completed SUPPDM dataset as /app/output/suppdm.csv.

run.R

Edit5 lines

1library(yamaa)23dm <- yamaa_domain("spec_dm.yaml")$output4suppdm <- yamaa_domain("spec_suppdm.yaml")$output5list(dm = dm, suppdm = suppdm)

run.py

Edit5 lines

1import yamaa23dm = yamaa.yamaa_domain("spec_dm.yaml").output4suppdm = yamaa.yamaa_domain("spec_suppdm.yaml").output5dm, suppdm

solutions/sdtm-dm-race-ethnicity/result.R

Edit118 lines

1# Reference solution for the yamaa benchmark sdtm-dm-race-ethnicity (R track).2#3# Written from the benchmark's full prompt and its inputs alone, as an agent4# would write it. Harbor's oracle agent runs it to check that the prompt can5# be solved and that the grader scores a correct answer 1.67library(dplyr, warn.conflicts = FALSE)8library(readr)9library(tidyr)10library(stringr)11library(purrr)1213odm <- read_csv(14  "/app/input/odm.csv",15  col_types = cols(.default = col_character())16)1718race_map <- c(19  "White" = "WHITE",20  "Asian" = "ASIAN",21  "Black or African American" = "BLACK OR AFRICAN AMERICAN",22  "American Indian or Alaska Native" = "AMERICAN INDIAN OR ALASKA NATIVE",23  "Native Hawaiian or Other Pacific Islander" = "NATIVE HAWAIIAN OR OTHER PACIFIC ISLANDER",24  "Other, specify: Fijian" = "OTHER",25  "Subject refused" = "UNKNOWN",26  "Not reported" = "NOT REPORTED"27)2829ethnic_map <- c(30  "Hispanic or Latino" = "HISPANIC OR LATINO",31  "Not Hispanic or Latino" = "NOT HISPANIC OR LATINO",32  "Not reported" = "NOT REPORTED",33  "Unknown" = "UNKNOWN"34)3536map_term <- function(x, lut) {37  out <- unname(lut[x])38  out[is.na(out)] <- NA_character_39  out40}4142subjects <- odm |>43  distinct(StudyOID, SubjectKey) |>44  mutate(45    STUDYID = StudyOID,46    USUBJID = SubjectKey47  )4849race_distinct <- odm |>50  filter(ItemOID == "IT.DM.RACE", Value != "", !is.na(Value)) |>51  distinct(StudyOID, SubjectKey, Value)5253race_per_subject <- race_distinct |>54  group_by(StudyOID, SubjectKey) |>55  summarise(answers = list(sort(unique(Value))), .groups = "drop")5657ethnic_per_subject <- odm |>58  filter(ItemOID == "IT.DM.ETHNIC", Value != "", !is.na(Value)) |>59  group_by(StudyOID, SubjectKey) |>60  summarise(ethnic_answer = first(Value), .groups = "drop")6162dm <- subjects |>63  left_join(race_per_subject, by = c("StudyOID", "SubjectKey")) |>64  left_join(ethnic_per_subject, by = c("StudyOID", "SubjectKey")) |>65  rowwise() |>66  mutate(67    DOMAIN = "DM",68    SUBJID = USUBJID,69    RACE = if (is.null(answers) || length(answers) == 0 || all(is.na(answers))) {70      NA_character_71    } else if (length(answers) >= 2) {72      "MULTIPLE"73    } else {74      map_term(answers[[1]], race_map)75    },76    ETHNIC = map_term(ethnic_answer, ethnic_map)77  ) |>78  ungroup() |>79  select(STUDYID, USUBJID, DOMAIN, SUBJID, RACE, ETHNIC, StudyOID, SubjectKey) |>80  arrange(USUBJID)8182multi <- dm |> filter(RACE == "MULTIPLE")8384suppdm <- race_distinct |>85  inner_join(86    multi |> select(StudyOID, SubjectKey),87    by = c("StudyOID", "SubjectKey")88  ) |>89  left_join(90    subjects |> select(StudyOID, SubjectKey, STUDYID, USUBJID),91    by = c("StudyOID", "SubjectKey")92  ) |>93  arrange(USUBJID, Value) |>94  group_by(STUDYID, USUBJID) |>95  mutate(96    n = row_number(),97    QNAM = paste0("RACE", n),98    QLABEL = paste("Race", n)99  ) |>100  ungroup() |>101  mutate(102    RDOMAIN = "DM",103    IDVAR = "USUBJID",104    IDVARVAL = USUBJID,105    QVAL = map_term(Value, race_map),106    QORIG = "CRF",107    QEVAL = NA_character_108  ) |>109  select(STUDYID, RDOMAIN, USUBJID, IDVAR, IDVARVAL, QNAM, QLABEL, QVAL, QORIG, QEVAL) |>110  arrange(USUBJID, QNAM)111112dir.create("/app/output", showWarnings = FALSE)113write_csv(114  dm |> select(DOMAIN, STUDYID, USUBJID, SUBJID, RACE, ETHNIC),115  "/app/output/dm.csv",116  na = ""117)118write_csv(suppdm, "/app/output/suppdm.csv", na = "")

solutions/sdtm-dm-race-ethnicity/result.py

Edit157 lines

1# Reference solution for the yamaa benchmark sdtm-dm-race-ethnicity (Python track).2#3# Written from the benchmark's full prompt and its inputs alone, as an agent4# would write it. Harbor's oracle agent runs it to check that the prompt can5# be solved and that the grader scores a correct answer 1.67from pathlib import Path89import polars as pl1011odm = pl.read_csv("/app/input/odm.csv", infer_schema=False)1213RACE_MAP = {14    "White": "WHITE",15    "Asian": "ASIAN",16    "Black or African American": "BLACK OR AFRICAN AMERICAN",17    "American Indian or Alaska Native": "AMERICAN INDIAN OR ALASKA NATIVE",18    "Native Hawaiian or Other Pacific Islander": "NATIVE HAWAIIAN OR OTHER PACIFIC ISLANDER",19    "Other, specify: Fijian": "OTHER",20    "Subject refused": "UNKNOWN",21    "Not reported": "NOT REPORTED",22}2324ETHNIC_MAP = {25    "Hispanic or Latino": "HISPANIC OR LATINO",26    "Not Hispanic or Latino": "NOT HISPANIC OR LATINO",27    "Not reported": "NOT REPORTED",28    "Unknown": "UNKNOWN",29}303132# Distinct collected answers per subject.33race_rows = odm.filter(pl.col("ItemOID") == "IT.DM.RACE", pl.col("Value") != "")34ethnic_rows = odm.filter(pl.col("ItemOID") == "IT.DM.ETHNIC", pl.col("Value") != "")3536subjects = (37    odm.select("StudyOID", "SubjectKey")38    .unique()39    .with_columns(40        STUDYID=pl.col("StudyOID"),41        USUBJID=pl.col("SubjectKey"),42    )43)4445race_distinct = (46    race_rows.select("StudyOID", "SubjectKey", "Value")47    .unique()48    .group_by("StudyOID", "SubjectKey")49    .agg(pl.col("Value").alias("answers"))50)5152ethnic_one = (53    ethnic_rows.select("StudyOID", "SubjectKey", "Value")54    .group_by("StudyOID", "SubjectKey")55    .agg(pl.col("Value").first().alias("ethnic_answer"))56)5758dm = (59    subjects.join(race_distinct, on=["StudyOID", "SubjectKey"], how="left")60    .join(ethnic_one, on=["StudyOID", "SubjectKey"], how="left")61    .with_columns(62        DOMAIN=pl.lit("DM"),63        SUBJID=pl.col("USUBJID"),64        RACE=pl.col("answers").map_elements(65            lambda ans: (66                None67                if ans is None or len(ans) == 068                else (69                    "MULTIPLE"70                    if len(ans) >= 271                    else RACE_MAP.get(ans[0])72                )73            ),74            return_dtype=pl.String,75        ),76        ETHNIC=pl.col("ethnic_answer").map_elements(77            lambda v: ETHNIC_MAP.get(v) if v is not None else None,78            return_dtype=pl.String,79        ),80    )81    .select("DOMAIN", "STUDYID", "USUBJID", "SUBJID", "RACE", "ETHNIC")82    .sort("USUBJID")83)8485# SUPPDM: one record per race for MULTIPLE subjects, alphabetical by answer.86multi = (87    subjects.join(race_distinct, on=["StudyOID", "SubjectKey"], how="left")88    .with_columns(89        RACE_TMP=pl.col("answers").map_elements(90            lambda ans: (91                None92                if ans is None or len(ans) == 093                else ("MULTIPLE" if len(ans) >= 2 else RACE_MAP.get(ans[0]))94            ),95            return_dtype=pl.String,96        )97    )98    .filter(pl.col("RACE_TMP") == "MULTIPLE")99)100101supp_rows: list[dict] = []102for row in multi.to_dicts():103    studyid = row["StudyOID"]104    usubjid_val = row["USUBJID"]105    answers = sorted(row["answers"])106    for i, ans in enumerate(answers, start=1):107        supp_rows.append(108            {109                "STUDYID": studyid,110                "RDOMAIN": "DM",111                "USUBJID": usubjid_val,112                "IDVAR": "USUBJID",113                "IDVARVAL": usubjid_val,114                "QNAM": f"RACE{i}",115                "QLABEL": f"Race {i}",116                "QVAL": RACE_MAP.get(ans),117                "QORIG": "CRF",118                "QEVAL": None,119            }120        )121122if supp_rows:123    suppdm = pl.DataFrame(124        supp_rows,125        schema={126            "STUDYID": pl.String,127            "RDOMAIN": pl.String,128            "USUBJID": pl.String,129            "IDVAR": pl.String,130            "IDVARVAL": pl.String,131            "QNAM": pl.String,132            "QLABEL": pl.String,133            "QVAL": pl.String,134            "QORIG": pl.String,135            "QEVAL": pl.String,136        },137    ).sort(["USUBJID", "QNAM"])138else:139    suppdm = pl.DataFrame(140        [],141        schema={142            "STUDYID": pl.String,143            "RDOMAIN": pl.String,144            "USUBJID": pl.String,145            "IDVAR": pl.String,146            "IDVARVAL": pl.String,147            "QNAM": pl.String,148            "QLABEL": pl.String,149            "QVAL": pl.String,150            "QORIG": pl.String,151            "QEVAL": pl.String,152        },153    )154155Path("/app/output").mkdir(exist_ok=True)156dm.write_csv("/app/output/dm.csv")157suppdm.write_csv("/app/output/suppdm.csv")

solutions/sdtm-dm-race-ethnicity/result.sas

Edit47 lines

1/* opensas reference solution for the yamaa benchmark sdtm-dm-race-ethnicity.2   Reads the benchmark inputs and derives the requested output.3   Execute with the existing opensas runtime: opensas /app/output/result.sas. */45data odm;6  length StudyOID MetaDataVersionOID SubjectKey StudyEventOID StudyEventRepeatKey ItemGroupOID ItemGroupRepeatKey ItemOID Value $1024;7  infile "/app/input/odm.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;8  input StudyOID : $char1024. MetaDataVersionOID : $char1024. SubjectKey : $char1024. StudyEventOID : $char1024. StudyEventRepeatKey : $char1024. ItemGroupOID : $char1024. ItemGroupRepeatKey : $char1024. ItemOID : $char1024. Value : $char1024.;9run;1011proc sql;12create table subjects as select distinct StudyOID as STUDYID,SubjectKey as USUBJID from odm;13create table races as select distinct StudyOID as STUDYID,SubjectKey as USUBJID,Value from odm where ItemOID='IT.DM.RACE' and not missing(Value);14quit;15data mapped; set races; length QVAL $1024; if Value in ('White','Asian','Black or African American','American Indian or Alaska Native','Native Hawaiian or Other Pacific Islander') then QVAL=upcase(Value);16else if Value='Other, specify: Fijian' then QVAL='OTHER'; else if Value='Subject refused' then QVAL='UNKNOWN'; else if Value='Not reported' then QVAL='NOT REPORTED'; run;17proc sql; create table racecounts as select STUDYID,USUBJID,count(*) as NRACE from mapped group by STUDYID,USUBJID; quit;18proc sort data=odm out=ordered; by StudyOID SubjectKey; run;19data ethnicity;20  length ETHNIC $1024;21  retain ETHNIC;22  set ordered;2324  by StudyOID SubjectKey;25  if first.SubjectKey then do;26    ETHNIC='';27  end;28    if ItemOID='IT.DM.ETHNIC' then ETHNIC=Value;29  if last.SubjectKey then output;30run;31data ethnic; set ethnicity; length STUDYID USUBJID $1024; STUDYID=StudyOID; USUBJID=SubjectKey;32if ETHNIC in ('Hispanic or Latino','Not Hispanic or Latino','Not reported','Unknown') then ETHNIC=upcase(ETHNIC); else ETHNIC=''; run;33proc sql;34create table dmjoined as select a.*,b.NRACE,c.QVAL,d.ETHNIC from subjects a left join racecounts b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID left join mapped c on a.STUDYID=c.STUDYID and a.USUBJID=c.USUBJID and b.NRACE=1 left join ethnic d on a.STUDYID=d.STUDYID and a.USUBJID=d.USUBJID;35create table multirace as select a.* from mapped a inner join racecounts b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID where b.NRACE>1;36quit;37data dmout; set dmjoined; length DOMAIN $2 SUBJID RACE $1024; DOMAIN='DM'; SUBJID=USUBJID; RACE=QVAL; if NRACE>1 then RACE='MULTIPLE'; keep DOMAIN STUDYID USUBJID SUBJID RACE ETHNIC; run;38proc sort data=multirace; by STUDYID USUBJID Value; run;39data suppout; set multirace; by STUDYID USUBJID; length RDOMAIN $2 IDVAR IDVARVAL QNAM QLABEL QORIG QEVAL $1024; retain N;40if first.USUBJID then N=0; N+1; RDOMAIN='DM'; IDVAR='USUBJID'; IDVARVAL=USUBJID; QNAM=cats('RACE',put(N,best32.)); QLABEL=catx(' ','Race',strip(put(N,best32.))); QORIG='CRF'; QEVAL=''; run;41/* KEEP drops variables but does not set their export order. */42proc sql;43create table dmexport as select DOMAIN,STUDYID,USUBJID,SUBJID,RACE,ETHNIC from dmout;44create table suppdmout as select STUDYID,RDOMAIN,USUBJID,IDVAR,IDVARVAL,QNAM,QLABEL,QVAL,QORIG,QEVAL from suppout;45quit;46proc export data=dmexport outfile='/app/output/dm.csv' dbms=csv replace; run;47proc export data=suppdmout outfile='/app/output/suppdm.csv' dbms=csv replace; run;

input/odm.csv

expected/dm.csv

expected/suppdm.csv

Comments

Mapping spec

Agent prompt

prompts/sdtm-dm-race-ethnicity/full.md

prompts/sdtm-dm-race-ethnicity/conventions.md

prompts/sdtm-dm-race-ethnicity/brief.md

yamaa code

run.R

run.py

Solution without yamaa

solutions/sdtm-dm-race-ethnicity/result.R

solutions/sdtm-dm-race-ethnicity/result.py

solutions/sdtm-dm-race-ethnicity/result.sas