yamaa As a clinical trial statistician or programmer, challenge anything in this benchmark: describe a problem in Comments, or propose a fix with the Edit button on any source file.
- Task: Would a real study need this derivation, specified this way? Does it follow CDISC ADaM: variable names, controlled terms, record structure? Flag any rule that is ambiguous or that a sponsor would handle differently.
- Inputs: Do they look like data collected in a real trial, and do they include the situations you would expect to meet in one?
- Expected output: Would independent QC programming reproduce every value from these inputs and rules?
The other tabs beside Comments are background for assessment, not under review.
Summary
EditUnder reviewWhat this benchmark meansREADME.mdGoal: select each subject's actual treatment (TRT01A), the first and
last exposure dates and times (TRTSDT, TRTSDTM, TRTEDT, TRTEDTM),
the imputation flags (TRTSTMF, TRTETMF), the treatment duration in days
(TRTDURD), and the safety population flag (SAFFL).
Input: one demographics (DM) record per subject carrying the
actual arm (ACTARM), plus exposure (EX) records carrying the
treatment name (EXTRT), sequence number (EXSEQ), start date
(EXSTDTC), and end date (EXENDTC). Records are built from the
demographics records, so an exposure record with no matching
demographics record adds no record.
Variables:
TRT01A: treatment actually received, uppercased: the treatment of the earliest qualifying exposure (a qualifying record namesVITAMIN D3orPLACEBOand carries a start), with ties on the start broken by the lowerEXSEQ; then the actual arm (ACTARM) when no exposure qualifies; then the textNOT TREATEDwhen neither source names a treatment.TRTSDTandTRTSDTM: date and date/time of the earliest qualifying exposure start; a start collected without a time uses00:00:00. Both are blank when the subject has no qualifying exposure record.TRTSTMF:Hwhen the first exposure start was collected without a time, so its time was imputed; otherwise blank.TRTEDTandTRTEDTM: date and date/time of the latest exposure end amongVITAMIN D3andPLACEBOrecords (a record need not have a start to name an end); an end collected without a time uses23:59:59. Both are blank when no such record has an end.TRTETMF:Hwhen the last exposure end was collected without a time, so its time was imputed; otherwise blank.TRTDURD: number of days fromTRTSDTtoTRTEDT, counting both the first and the last day (inclusive), so a single treatment day gives one; blank when either date is blank.SAFFL:Ywhen the subject has a treatment start date,Notherwise (never blank).
Note: the exposure dates, date/times, and duration are all present or all blank together; the latest end can come from a record whose start was never collected.
Inputs
Under review2 source filesinput/dm.csv
Edit8 rows| STUDYID | USUBJID | ACTARM |
|---|---|---|
| YAMAA-01 | Vitamin D3 | |
| YAMAA-01 | Placebo | |
| YAMAA-01 | Vitamin D3 | |
| YAMAA-01 | Placebo | |
| YAMAA-01 | Vitamin D3 | |
| YAMAA-01 | — | |
| YAMAA-01 | Vitamin D3 | |
| YAMAA-01 | Placebo |
input/ex.csv
Edit13 rows| STUDYID | USUBJID | EXSEQ | EXTRT | EXDOSE | EXSTDTC | EXENDTC |
|---|---|---|---|---|---|---|
| YAMAA-01 | 1 | VITAMIN D3 | 4000 | 2025-01-08 | 2025-01-14T18:00:00 | |
| YAMAA-01 | 2 | VITAMIN D3 | 4000 | 2025-01-15T08:00:00 | 2025-01-28 | |
| YAMAA-01 | 1 | PLACEBO | 0 | 2025-02-08T09:00:00 | 2025-02-14T17:00:00 | |
| YAMAA-01 | 2 | PLACEBO | 0 | 2025-02-15T09:00:00 | 2025-02-28T17:00:00 | |
| YAMAA-01 | 1 | VITAMIN D3 | 4000 | 2025-03-05 | 2025-03-05 | |
| YAMAA-01 | 1 | VITAMIN D3 | 4000 | 2025-04-03T10:00:00 | 2025-04-10 | |
| YAMAA-01 | 2 | PLACEBO | 0 | 2025-04-03T10:00:00 | 2025-04-16T12:00:00 | |
| YAMAA-01 | 1 | RESCUE MED | 200 | 2025-05-01 | 2025-05-07 | |
| YAMAA-01 | 1 | VITAMIN D3 | 4000 | 2025-06-08T09:00:00 | 2025-06-14T16:00:00 | |
| YAMAA-01 | 2 | VITAMIN D3 | 4000 | — | 2025-06-20 | |
| YAMAA-01 | 1 | PLACEBO | 0 | 2025-07-01T10:00:00 | 2025-07-07T10:00:00 | |
| YAMAA-01 | 2 | PLACEBO | 0 | 2025-07-01T10:00:00 | 2025-07-07T10:00:00 | |
| YAMAA-01 | 1 | VITAMIN D3 | 4000 | 2025-03-08T10:00:00 | 2025-03-21T16:00:00 |
Expected output
Under review8 expected rowsexpected/adsl.csv
Edit8 rows| STUDYID | USUBJID | TRT01A | TRTSDT | TRTSDTM | TRTSTMF | TRTEDT | TRTEDTM | TRTETMF | TRTDURD | SAFFL |
|---|---|---|---|---|---|---|---|---|---|---|
| YAMAA-01 | VITAMIN D3 | 2025-01-08 | 2025-01-08T00:00:00 | H | 2025-01-28 | 2025-01-28T23:59:59 | H | 21 | Y | |
| YAMAA-01 | PLACEBO | 2025-02-08 | 2025-02-08T09:00:00 | — | 2025-02-28 | 2025-02-28T17:00:00 | — | 21 | Y | |
| YAMAA-01 | VITAMIN D3 | 2025-03-05 | 2025-03-05T00:00:00 | H | 2025-03-05 | 2025-03-05T23:59:59 | H | 1 | Y | |
| YAMAA-01 | VITAMIN D3 | 2025-04-03 | 2025-04-03T10:00:00 | — | 2025-04-16 | 2025-04-16T12:00:00 | — | 14 | Y | |
| YAMAA-01 | VITAMIN D3 | — | — | — | — | — | — | — | N | |
| YAMAA-01 | NOT TREATED | — | — | — | — | — | — | — | N | |
| YAMAA-01 | VITAMIN D3 | 2025-06-08 | 2025-06-08T09:00:00 | — | 2025-06-20 | 2025-06-20T23:59:59 | H | 13 | Y | |
| YAMAA-01 | PLACEBO | 2025-07-01 | 2025-07-01T10:00:00 | — | 2025-07-07 | 2025-07-07T10:00:00 | — | 7 | Y |
Comments and assessment materials
| Dataset Name | Variable Name | Variable Label | Type | Codelist/Controlled Terms | Core | Computational Method | Origin |
|---|---|---|---|---|---|---|---|
| ADSL | STUDYID | Study Identifier | character | Copy value from DM.STUDYID. | Predecessor | ||
| ADSL | USUBJID | Unique Subject Identifier | character | Copy value from DM.USUBJID. | Predecessor | ||
| ADSL | TRT01A | Actual Treatment for Period 01 | character | VITAMIN D3, PLACEBO, NOT TREATED | Convert TRT01SRC to upper case. | Derived | |
| ADSL | TRTSDTM | Datetime of First Exposure to Treatment | numeric | ISO 8601 | Impute incomplete datetime from TRTSDTC, defaulting the time to the first instant of the day; a missing source → blank; invalid text → blank. | Derived | |
| ADSL | TRTSTMF | Time of First Exposure Imputation Flag | character | H | If TRTSTPR = 'D' then "H"; otherwise blank. | Derived | |
| ADSL | TRTEDTM | Datetime of Last Exposure to Treatment | numeric | ISO 8601 | Impute incomplete datetime from TRTEDTC, defaulting the time to the last instant of the day; a missing source → blank; invalid text → blank. | Derived | |
| ADSL | TRTETMF | Time of Last Exposure Imputation Flag | character | H | If TRTETPR = 'D' then "H"; otherwise blank. | Derived | |
| ADSL | TRTSDT | Date of First Exposure to Treatment | numeric | ISO 8601 | Take the date part of TRTSDTM. | Derived | |
| ADSL | TRTEDT | Date of Last Exposure to Treatment | numeric | ISO 8601 | Take the date part of TRTEDTM. | Derived | |
| ADSL | TRTDURD | Total Treatment Duration (Days) | numeric | Whole days from TRTSDT to TRTEDT, counting both endpoints. | Derived | ||
| ADSL | SAFFL | Safety Population Flag | character | Y, N | "Y" when TRTSDT IS NOT NULL; "N" otherwise; "N" when the condition is unknown. | Derived |
| Variable | Permitted values |
|---|---|
| TRT01A | VITAMIN D3, PLACEBO, NOT TREATED |
| TRTSTMF | H |
| TRTETMF | H |
| SAFFL | Y, N |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/adam-adsl-treatment/full.md
Edit33 linesFollowing CDISC ADaM standards, use the provided DM and EX datasets to create an ADSL dataset with one record per subject.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, TRT01A, TRTSDT, TRTSDTM, TRTSTMF, TRTEDT, TRTEDTM, TRTETMF, TRTDURD, SAFFL
TRT01A is the treatment actually received, in upper case: the treatment of the earliest qualifying exposure (a record naming VITAMIN D3 or PLACEBO that carries a start), with ties on the start broken by the lower sequence number; then the actual arm when no exposure qualifies; then the text NOT TREATED when neither source names a treatment. An exposure record with no matching demographics record adds no record.
TRTSDT and TRTSDTM are the date and date-time of the earliest qualifying exposure start; both have no value when the subject has no qualifying exposure record. A start collected as a date alone reads midnight, and TRTSTMF holds H then, and has no value otherwise.
TRTEDT and TRTEDTM are the date and date-time of the latest exposure end among VITAMIN D3 and PLACEBO records; a record need not have a start to name an end. An end collected as a date alone reads the last moment of the day, and TRTETMF holds H then, and has no value otherwise. Both have no value when no such record has an end.
TRTDURD is the number of days from TRTSDT to TRTEDT, counting both the first and the last day, so a single treatment day gives one; it has no value when either date has none. SAFFL is Y when the subject has a treatment start date and N otherwise.
Read the source datasets from /app/input and save the completed dataset as /app/output/adsl.csv.
prompts/adam-adsl-treatment/conventions.md
Edit15 linesFollowing CDISC ADaM standards, use the provided DM and EX datasets to create an ADSL dataset with one record per subject.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, TRT01A, TRTSDT, TRTSDTM, TRTSTMF, TRTEDT, TRTEDTM, TRTETMF, TRTDURD, SAFFL
TRT01A is the treatment in upper case, or the text NOT TREATED.
TRTSTMF and TRTETMF each hold H or have no value.
TRTDURD counts both the first and the last day. SAFFL is Y or N.
Read the source datasets from /app/input and save the completed dataset as /app/output/adsl.csv.
prompts/adam-adsl-treatment/brief.md
Edit9 linesFollowing CDISC ADaM standards, use the provided DM and EX datasets to create an ADSL dataset with one record per subject.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, TRT01A, TRTSDT, TRTSDTM, TRTSTMF, TRTEDT, TRTEDTM, TRTETMF, TRTDURD, SAFFL
Read the source datasets from /app/input and save the completed dataset as /app/output/adsl.csv.
run.R
Edit4 lineslibrary(yamaa)adsl <- yamaa_domain("spec.yaml")$outputadslrun.py
Edit4 linesimport yamaaadsl = yamaa.yamaa_domain("spec.yaml").outputadslsolutions/adam-adsl-treatment/result.R
Edit80 lines# Reference solution for the yamaa benchmark adam-adsl-treatment (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)library(stringr)library(lubridate)dm <- read_csv( "/app/input/dm.csv", col_types = cols(.default = col_character()))ex <- read_csv( "/app/input/ex.csv", col_types = cols(EXSEQ = col_integer(), .default = col_character()))qual <- c("VITAMIN D3", "PLACEBO")parse_start <- function(x) { x[!is.na(x) & x == ""] <- NA_character_ dt <- suppressWarnings(ymd_hms(x, quiet = TRUE, tz = "UTC")) d <- suppressWarnings(ymd(x, quiet = TRUE, tz = "UTC")) coalesce(dt, as.POSIXct(d, tz = "UTC"))}parse_end <- function(x) { base <- parse_start(x) is_date <- !is.na(x) & x != "" & !str_detect(x, "T") base[is_date] <- base[is_date] + 23 * 3600 + 59 * 60 + 59 base}ex_dt <- ex |> mutate( `__start_dt` = parse_start(EXSTDTC), `__end_dt` = parse_end(EXENDTC) )first_start <- ex_dt |> filter(EXTRT %in% qual, !is.na(`__start_dt`)) |> arrange(`__start_dt`, EXSEQ) |> distinct(STUDYID, USUBJID, .keep_all = TRUE) |> select(STUDYID, USUBJID, TRT01RAW = EXTRT, `__stdtm` = `__start_dt`, `__stdtc` = EXSTDTC)last_end <- ex_dt |> filter(EXTRT %in% qual, !is.na(`__end_dt`)) |> arrange(desc(`__end_dt`)) |> distinct(STUDYID, USUBJID, .keep_all = TRUE) |> select(STUDYID, USUBJID, `__endtm` = `__end_dt`, `__endtc` = EXENDTC)adsl <- dm |> left_join(first_start, by = c("STUDYID", "USUBJID")) |> left_join(last_end, by = c("STUDYID", "USUBJID")) |> mutate( `__src` = coalesce(TRT01RAW, ACTARM), TRT01A = case_when( is.na(`__src`) | `__src` == "" ~ "NOT TREATED", .default = str_to_upper(`__src`) ), TRTSDTM = format(`__stdtm`, "%Y-%m-%dT%H:%M:%S", tz = "UTC"), TRTSDT = as.Date(`__stdtm`, tz = "UTC"), TRTSTMF = if_else(!is.na(`__stdtc`) & !str_detect(`__stdtc`, "T"), "H", NA_character_), TRTEDTM = format(`__endtm`, "%Y-%m-%dT%H:%M:%S", tz = "UTC"), TRTEDT = as.Date(`__endtm`, tz = "UTC"), TRTETMF = if_else(!is.na(`__endtc`) & !str_detect(`__endtc`, "T"), "H", NA_character_), TRTDURD = as.integer(TRTEDT - TRTSDT) + 1L, SAFFL = if_else(is.na(TRTSDT), "N", "Y") ) |> select( STUDYID, USUBJID, TRT01A, TRTSDT, TRTSDTM, TRTSTMF, TRTEDT, TRTEDTM, TRTETMF, TRTDURD, SAFFL ) |> arrange(USUBJID)dir.create("/app/output", showWarnings = FALSE)write_csv(adsl, "/app/output/adsl.csv", na = "")solutions/adam-adsl-treatment/result.py
Edit93 lines# Reference solution for the yamaa benchmark adam-adsl-treatment (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.from pathlib import Pathimport polars as pldm = pl.read_csv("/app/input/dm.csv", infer_schema=False, null_values="")ex = pl.read_csv("/app/input/ex.csv", infer_schema=False, null_values="").with_columns( pl.col("EXSEQ").cast(pl.Int64, strict=False),)qual = ["VITAMIN D3", "PLACEBO"]# A start collected as a date alone reads midnight; an end collected as a# date alone reads the last moment of the day.ex_dt = ex.with_columns( __start_dt=pl.col("EXSTDTC").str.to_datetime(strict=False), __end_base=pl.col("EXENDTC").str.to_datetime(strict=False),).with_columns( __end_dt=pl.when(pl.col("EXENDTC").str.contains("T")) .then(pl.col("__end_base")) .when(pl.col("EXENDTC").str.len_chars() == 10) .then(pl.col("__end_base") + pl.duration(hours=23, minutes=59, seconds=59)) .otherwise(pl.col("__end_base")),)start_qual = ex_dt.filter( pl.col("EXTRT").is_in(qual) & pl.col("__start_dt").is_not_null()).sort(["__start_dt", "EXSEQ"])first_start = start_qual.unique(["STUDYID", "USUBJID"], keep="first", maintain_order=True).select( "STUDYID", "USUBJID", TRT01RAW="EXTRT", TRTSDTM_DT="__start_dt", __first_std="EXSTDTC")end_qual = ex_dt.filter( pl.col("EXTRT").is_in(qual) & pl.col("__end_dt").is_not_null()).sort(["__end_dt"], descending=True)last_end = end_qual.unique(["STUDYID", "USUBJID"], keep="first", maintain_order=True).select( "STUDYID", "USUBJID", TRTEDTM_DT="__end_dt", __last_endtc="EXENDTC")adsl = ( dm.join(first_start, on=["STUDYID", "USUBJID"], how="left", maintain_order="left") .join(last_end, on=["STUDYID", "USUBJID"], how="left") .with_columns( __src=pl.coalesce("TRT01RAW", "ACTARM"), ) .with_columns( TRT01A=pl.when(pl.col("__src").is_null()) .then(pl.lit("NOT TREATED")) .otherwise(pl.col("__src").str.to_uppercase()), TRTSDTM=pl.col("TRTSDTM_DT").dt.strftime("%Y-%m-%dT%H:%M:%S"), TRTSDT=pl.col("TRTSDTM_DT").cast(pl.Date), TRTSTMF=pl.when( pl.col("__first_std").is_not_null() & ~pl.col("__first_std").str.contains("T") ) .then(pl.lit("H")) .otherwise(None), TRTEDTM=pl.col("TRTEDTM_DT").dt.strftime("%Y-%m-%dT%H:%M:%S"), TRTEDT=pl.col("TRTEDTM_DT").cast(pl.Date), TRTETMF=pl.when( pl.col("__last_endtc").is_not_null() & ~pl.col("__last_endtc").str.contains("T") ) .then(pl.lit("H")) .otherwise(None), ) .with_columns( TRTDURD=(pl.col("TRTEDT") - pl.col("TRTSDT")).dt.total_days() + 1, SAFFL=pl.when(pl.col("TRTSDT").is_not_null()) .then(pl.lit("Y")) .otherwise(pl.lit("N")), ) .select( "STUDYID", "USUBJID", "TRT01A", "TRTSDT", "TRTSDTM", "TRTSTMF", "TRTEDT", "TRTEDTM", "TRTETMF", "TRTDURD", "SAFFL", ) .sort("USUBJID"))Path("/app/output").mkdir(parents=True, exist_ok=True)adsl.write_csv("/app/output/adsl.csv", null_value="")solutions/adam-adsl-treatment/result.sas
Edit50 lines/* opensas reference solution for the yamaa benchmark adam-adsl-treatment. Reads the benchmark inputs and derives the requested output. Execute with the existing opensas runtime: opensas /app/output/result.sas. */data dm; length STUDYID USUBJID ACTARM $1024; infile "/app/input/dm.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. ACTARM : $char1024.;run;data ex; length STUDYID USUBJID EXTRT EXSTDTC EXENDTC $1024; infile "/app/input/ex.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. EXSEQ EXTRT : $char1024. EXDOSE EXSTDTC : $char1024. EXENDTC : $char1024.;run;data exposure; length startmoment endmoment $19; set ex; if upcase(EXTRT) in ('VITAMIN D3','PLACEBO'); startmoment=EXSTDTC; endmoment=EXENDTC; if lengthn(EXSTDTC)=10 then startmoment=cats(EXSTDTC,'T00:00:00'); else if lengthn(EXSTDTC)=16 then startmoment=cats(EXSTDTC,':00'); if lengthn(EXENDTC)=10 then endmoment=cats(EXENDTC,'T23:59:59'); else if lengthn(EXENDTC)=16 then endmoment=cats(EXENDTC,':00');run;proc sort data=exposure(where=(EXSTDTC ne '')) out=starts; by STUDYID USUBJID startmoment EXSEQ; run;data firstex; set starts; by STUDYID USUBJID; if first.USUBJID; run;proc sort data=exposure(where=(EXENDTC ne '')) out=ends; by STUDYID USUBJID descending endmoment; run;data lastex; set ends; by STUDYID USUBJID; if first.USUBJID; run;proc sql; create table joined as select a.*,b.EXTRT,b.EXSTDTC,b.startmoment,c.EXENDTC,c.endmoment from dm as a left join firstex as b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID left join lastex as c on a.STUDYID=c.STUDYID and a.USUBJID=c.USUBJID; quit;data result; length TRT01A $1024 TRTSDT TRTEDT $10 TRTSDTM TRTEDTM $19 TRTSTMF TRTETMF SAFFL $1; set joined; TRT01A=upcase(coalescec(EXTRT,ACTARM,'NOT TREATED')); TRTSDT=''; TRTEDT=''; TRTSTMF=''; TRTETMF=''; TRTDURD=.; if lengthn(EXSTDTC)>=10 then TRTSDT=substr(EXSTDTC,1,10); if lengthn(EXENDTC)>=10 then TRTEDT=substr(EXENDTC,1,10); TRTSDTM=tranwrd(startmoment,'T',' '); TRTEDTM=tranwrd(endmoment,'T',' '); if lengthn(EXSTDTC)=10 then TRTSTMF='H'; if lengthn(EXENDTC)=10 then TRTETMF='H'; if not missing(TRTSDT) and not missing(TRTEDT) then TRTDURD=input(TRTEDT,yymmdd10.)-input(TRTSDT,yymmdd10.)+1; SAFFL='N'; if not missing(TRTSDT) then SAFFL='Y';run;proc sql; create table final as select STUDYID, USUBJID, TRT01A, TRTSDT, TRTSDTM, TRTSTMF, TRTEDT, TRTEDTM, TRTETMF, TRTDURD, SAFFL from result;quit;proc export data=final outfile="/app/output/adsl.csv" dbms=csv replace; run;
Comments
Signed in with GitHub; saved in repository Discussions