yamaa As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.
We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.
Summary
EditUnder reviewWhat this benchmark meansREADME.mdPublic entries: Python and R. Passing coverage requires an installed inventory.
Goal: derive the combined population flag POPFL from the
safety flag SAFFL and the intent-to-treat flag ITTFL, plus
the dates and age they read: first exposure date TRTSDT,
randomization date RANDDT, collected age AGE, age band
AGEGR1, and age rank AGERNK.
Input: one demographic record per subject carrying age and the randomization date, and exposure records carrying the exposure start date.
Variables:
POPFLholdsYonly when bothSAFFLandITTFLholdY, andNotherwise, marking subjects in both populations.SAFFLholdsYwhen the subject has a first exposure date, andNotherwise, marking subjects who started treatment.ITTFLholdsYwhen the subject was randomized, andNotherwise, marking subjects in the intent-to-treat population.TRTSDTholds the earliest dated exposure start for the subject, and stays empty when no exposure record carries a date.RANDDTholds the collected randomization date, and stays empty when the subject was not randomized.AGEGR1holds<65whenAGEis below 65 and>=65when it is 65 or above; a subject with no age stops the run.AGERNKholds the subject rank byAGEwithin the study, youngest first, withUSUBJIDbreaking ties.
Note: the dates and age are settled before the flags that read them: the safety and intent-to-treat flags come from their dates, and the combined flag comes from both flags.
Inputs
Under review2 source filesinput/dm.csv
Edit7 rows| STUDYID | USUBJID | AGE | RANDDT |
|---|---|---|---|
| YAMAA-01 | 34 | 2025-01-05 | |
| YAMAA-01 | 65 | 2025-02-05 | |
| YAMAA-01 | 52 | — | |
| YAMAA-01 | 65 | 2025-02-20 | |
| YAMAA-01 | 41 | — | |
| YAMAA-01 | 88 | 2025-03-01 | |
| YAMAA-01 | 29 | 2025-04-01 |
input/ex.csv
Edit7 rows| STUDYID | USUBJID | EXSEQ | EXSTDTC |
|---|---|---|---|
| YAMAA-01 | 1 | 2025-01-08 | |
| YAMAA-01 | 2 | 2025-01-15 | |
| YAMAA-01 | 3 | — | |
| YAMAA-01 | 1 | 2025-02-08 | |
| YAMAA-01 | 1 | 2025-02-28 | |
| YAMAA-01 | 1 | 2025-03-05 | |
| YAMAA-01 | 1 | — |
Expected output
Under review7 expected rowsexpected/adsl.csv
Edit7 rows| POPFL | SAFFL | ITTFL | TRTSDT | RANDDT | AGEGR1 | AGERNK | STUDYID | USUBJID | AGE |
|---|---|---|---|---|---|---|---|---|---|
| Y | Y | Y | 2025-01-08 | 2025-01-05 | <65 | 2 | YAMAA-01 | 34 | |
| Y | Y | Y | 2025-02-08 | 2025-02-05 | >=65 | 5 | YAMAA-01 | 65 | |
| N | N | N | — | — | <65 | 4 | YAMAA-01 | 52 | |
| N | N | Y | — | 2025-02-20 | >=65 | 6 | YAMAA-01 | 65 | |
| N | Y | N | 2025-02-28 | — | <65 | 3 | YAMAA-01 | 41 | |
| Y | Y | Y | 2025-03-05 | 2025-03-01 | >=65 | 7 | YAMAA-01 | 88 | |
| N | N | Y | — | 2025-04-01 | <65 | 1 | YAMAA-01 | 29 |
Human review and assessment
| Dataset Name | Variable Name | Variable Label | Type | Codelist/Controlled Terms | Core | Computational Method | Origin |
|---|---|---|---|---|---|---|---|
| ADSL | TRTSDT | Date of First Exposure to Treatment | numeric | ISO 8601 | Aggregate MIN(EX.EXSTDTC) over records matching on the shared output keys. | Derived | |
| ADSL | SAFFL | Safety Population Flag | character | "Y" when TRTSDT IS NOT NULL; "N" otherwise; "N" when the condition is unknown. | Derived | ||
| ADSL | RANDDT | Randomization Date | numeric | ISO 8601 | Copy value from DM.RANDDT. | Predecessor | |
| ADSL | ITTFL | Intent-to-Treat Population Flag | character | "Y" when RANDFL = 'Y'; "N" otherwise; "N" when the condition is unknown. | Derived | ||
| ADSL | POPFL | Analysis Population Flag | character | "Y" when SAFFL = 'Y' AND ITTFL = 'Y'; "N" otherwise; "N" when the condition is unknown. | Derived | ||
| ADSL | STUDYID | Study Identifier | character | Copy value from DM.STUDYID. | Predecessor | ||
| ADSL | USUBJID | Unique Subject Identifier | character | Copy value from DM.USUBJID. | Predecessor | ||
| ADSL | AGE | Age | numeric | Copy value from DM.AGE. | Predecessor | ||
| ADSL | AGEGR1 | Pooled Age Group 1 | character | Band AGE at 65 into "<65" and ">=65", intervals closed on the left. | Derived | ||
| ADSL | AGERNK | Age Rank | numeric | Row number within each group of STUDYID, ordered by AGE then USUBJID. | Derived |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.7 | 2026-10-20 | yamaa docs build | Read a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default | ||
| 1.6 | 2026-10-17 | yamaa docs build | Describe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight | ||
| 1.5 | 2026-10-17 | yamaa docs build | Name collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part | ||
| 1.4 | 2026-10-15 | yamaa docs build | State datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks | ||
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/adam-adsl-flag-chain/full.md
Edit18 linesFollowing CDISC ADaM standards, use the provided DM and EX datasets to create an ADSL dataset with one record per subject.
The output dataset should contain the following columns in this order: POPFL, SAFFL, ITTFL, TRTSDT, RANDDT, AGEGR1, AGERNK, STUDYID, USUBJID, AGE
TRTSDT is the subject's earliest dated exposure start, and has no value when no exposure record for the subject carries a date. SAFFL is "Y" when the subject has a first exposure date and "N" otherwise. RANDDT is the randomization date collected for the subject, and has no value when the subject was not randomized. ITTFL is "Y" when the subject was randomized and "N" otherwise. POPFL is "Y" only when SAFFL and ITTFL are both "Y", and "N" otherwise. AGEGR1 is "<65" when AGE is below 65 and ">=65" when AGE is 65 or above. AGERNK is the subject's rank by AGE within the study, from 1 for the youngest, with ties in AGE broken by USUBJID.
Read the source datasets from /app/input and save the completed dataset as /app/output/adsl.csv.
prompts/adam-adsl-flag-chain/conventions.md
Edit13 linesFollowing CDISC ADaM standards, use the provided DM and EX datasets to create an ADSL dataset with one record per subject.
The output dataset should contain the following columns in this order: POPFL, SAFFL, ITTFL, TRTSDT, RANDDT, AGEGR1, AGERNK, STUDYID, USUBJID, AGE
SAFFL is "Y" or "N". ITTFL is "Y" or "N". POPFL is "Y" or "N". AGEGR1 is "<65" when AGE is below 65 and ">=65" when AGE is 65 or above. AGERNK is the subject's rank by AGE within the study, from 1 for the youngest, with ties in AGE broken by USUBJID.
Read the source datasets from /app/input and save the completed dataset as /app/output/adsl.csv.
prompts/adam-adsl-flag-chain/brief.md
Edit8 linesFollowing CDISC ADaM standards, use the provided DM and EX datasets to create an ADSL dataset with one record per subject.
The output dataset should contain the following columns in this order: POPFL, SAFFL, ITTFL, TRTSDT, RANDDT, AGEGR1, AGERNK, STUDYID, USUBJID, AGE
Read the source datasets from /app/input and save the completed dataset as /app/output/adsl.csv.
R
Edit3 lineslibrary(yamaa)yamaa_spec("spec.yaml")$run()$outputPython
Edit3 linesimport yamaayamaa.spec("spec.yaml").run().outputsolutions/adam-adsl-flag-chain/result.R
Edit47 lines# Reference solution for the yamaa benchmark adam-adsl-flag-chain (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)dm <- read_csv( "/app/input/dm.csv", col_types = cols(AGE = col_integer(), RANDDT = col_date(), .default = col_character()))ex <- read_csv( "/app/input/ex.csv", col_types = cols(EXSEQ = col_integer(), EXSTDTC = col_date(), .default = col_character()))first_exposure <- ex |> group_by(STUDYID, USUBJID) |> summarise( TRTSDT = if (all(is.na(EXSTDTC))) as.Date(NA) else min(EXSTDTC, na.rm = TRUE), .groups = "drop" )adsl <- dm |> left_join(first_exposure, by = c("STUDYID", "USUBJID")) |> mutate( SAFFL = if_else(!is.na(TRTSDT), "Y", "N"), RANDFL = if_else(!is.na(RANDDT), "Y", "N"), ITTFL = if_else(RANDFL == "Y", "Y", "N"), POPFL = if_else(SAFFL == "Y" & ITTFL == "Y", "Y", "N"), AGEGR1 = if_else(AGE < 65, "<65", ">=65") ) |> mutate(.idx = row_number()) |> arrange(STUDYID, AGE, USUBJID) |> group_by(STUDYID) |> mutate(AGERNK = row_number()) |> ungroup() |> arrange(.idx) |> select( POPFL, SAFFL, ITTFL, TRTSDT, RANDDT, AGEGR1, AGERNK, STUDYID, USUBJID, AGE )dir.create("/app/output", showWarnings = FALSE)write_csv(adsl, "/app/output/adsl.csv", na = "")solutions/adam-adsl-flag-chain/result.py
Edit72 lines# Reference solution for the yamaa benchmark adam-adsl-flag-chain (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.from pathlib import Pathimport polars as pldm = pl.read_csv("/app/input/dm.csv", infer_schema=False).with_columns( pl.col("RANDDT").str.to_date(strict=False), pl.col("AGE").cast(pl.Int64, strict=False),)ex = pl.read_csv("/app/input/ex.csv", infer_schema=False).with_columns( pl.col("EXSTDTC").str.to_date(strict=False),)first_exposure = ex.group_by(["STUDYID", "USUBJID"]).agg( TRTSDT=pl.col("EXSTDTC").min())adsl = ( dm.join(first_exposure, on=["STUDYID", "USUBJID"], how="left", maintain_order="left") .with_columns( SAFFL=pl.when(pl.col("TRTSDT").is_not_null()) .then(pl.lit("Y")) .otherwise(pl.lit("N")), RANDFL=pl.when(pl.col("RANDDT").is_not_null()) .then(pl.lit("Y")) .otherwise(pl.lit("N")), ) .with_columns( ITTFL=pl.when(pl.col("RANDFL") == "Y") .then(pl.lit("Y")) .otherwise(pl.lit("N")), ) .with_columns( POPFL=pl.when((pl.col("SAFFL") == "Y") & (pl.col("ITTFL") == "Y")) .then(pl.lit("Y")) .otherwise(pl.lit("N")), AGEGR1=pl.when(pl.col("AGE") < 65) .then(pl.lit("<65")) .when(pl.col("AGE") >= 65) .then(pl.lit(">=65")) .otherwise(None), ))# Rank by age within the study, youngest first, USUBJID breaking ties.ranked = ( adsl.with_columns(__idx=pl.int_range(pl.len())) .sort(["STUDYID", "AGE", "USUBJID"], nulls_last=True) .with_columns(AGERNK=pl.int_range(1, pl.len() + 1).over("STUDYID")) .sort("__idx") .drop("__idx") .select( "POPFL", "SAFFL", "ITTFL", "TRTSDT", "RANDDT", "AGEGR1", "AGERNK", "STUDYID", "USUBJID", "AGE", ))Path("/app/output").mkdir(exist_ok=True)ranked.write_csv("/app/output/adsl.csv")solutions/adam-adsl-flag-chain/result.sas
Edit39 lines/* opensas reference solution for the yamaa benchmark adam-adsl-flag-chain. Reads the benchmark inputs and derives the requested output. Execute with the existing opensas runtime: opensas /app/output/result.sas. */data dm; length STUDYID USUBJID RANDDT $1024; infile "/app/input/dm.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. AGE RANDDT : $char1024.;run;data ex; length STUDYID USUBJID EXSTDTC $1024; infile "/app/input/ex.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. EXSEQ EXSTDTC : $char1024.;run;proc sql; create table firstex as select STUDYID,USUBJID,min(EXSTDTC) as TRTSDT from ex where not missing(EXSTDTC) group by STUDYID,USUBJID; create table joined as select a.*,b.TRTSDT from dm as a left join firstex as b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID;quit;data ranked; set joined; NOAGE=missing(AGE); run;proc sort data=ranked; by STUDYID NOAGE AGE USUBJID; run;data result; length POPFL SAFFL ITTFL $1 AGEGR1 $4; set ranked; by STUDYID; if first.STUDYID then AGERNK=0; AGERNK+1; SAFFL='N'; ITTFL='N'; POPFL='N'; if not missing(TRTSDT) then SAFFL='Y'; if not missing(RANDDT) then ITTFL='Y'; if SAFFL='Y' and ITTFL='Y' then POPFL='Y'; if not missing(AGE) then do; if AGE<65 then AGEGR1='<65'; else AGEGR1='>=65'; end;run;proc sql; create table final as select POPFL, SAFFL, ITTFL, TRTSDT, RANDDT, AGEGR1, AGERNK, STUDYID, USUBJID, AGE from result;quit;proc export data=final outfile="/app/output/adsl.csv" dbms=csv replace; run;