yamaa As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.
We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.
Summary
EditUnder reviewWhat this benchmark meansREADME.mdPublic entries: Python and R. Passing coverage requires an installed inventory.
Goal: flag, for each planned test-versus-reference comparison of a
three-period crossover, the pharmacokinetic (PK) parameter records the
comparison uses (ANL01FL, ANL02FL), and carry the planned sequence
(TRTSEQP).
Input: one record per PK parameter, subject and period with the
actual treatment (TRTA) and the value (AVAL); the subject-level
dataset with the planned sequence; and a table of the planned
comparisons, each with its number (COMPNO), test treatment
(TESTTRT) and reference treatment (REFTRT).
Variables:
APERIODandTRTAare the record's period and the treatment actually given in it, unchanged.TRTSEQPis the subject's planned sequence of treatments.AVALis the parameter value, unchanged.ANL01FLisYon the records comparison 1 uses and empty on every other record;ANL02FLdoes the same for comparison 2. A record is used when its treatment is one of the comparison's two treatments, its own value is evaluable, and the same subject has an evaluable value of the same parameter under the comparison's other treatment. Evaluable means present and greater than zero.
Note: pairs are judged per parameter, so a subject can be in a comparison for one parameter and out of it for another, and a value that is missing or zero also takes its partner record out of that comparison. A treatment that is the test of one comparison and the reference of the other can carry both flags. Every record stays in the dataset.
Inputs
Under review3 source filesinput/adpp_raw.csv
Edit20 rows| STUDYID | USUBJID | PARAMCD | PARAM | APERIOD | TRTA | AVAL |
|---|---|---|---|---|---|---|
| YAMAA-01 | CMAX | Max Conc (ng/mL) | 1 | TEST FASTED | 120 | |
| YAMAA-01 | AUCIFO | AUC Infinity Obs (h*ng/mL) | 1 | TEST FASTED | 960 | |
| YAMAA-01 | CMAX | Max Conc (ng/mL) | 2 | TEST FED | 90 | |
| YAMAA-01 | AUCIFO | AUC Infinity Obs (h*ng/mL) | 2 | TEST FED | — | |
| YAMAA-01 | CMAX | Max Conc (ng/mL) | 3 | REFERENCE FASTED | 110 | |
| YAMAA-01 | AUCIFO | AUC Infinity Obs (h*ng/mL) | 3 | REFERENCE FASTED | 900 | |
| YAMAA-01 | CMAX | Max Conc (ng/mL) | 1 | TEST FED | 80 | |
| YAMAA-01 | AUCIFO | AUC Infinity Obs (h*ng/mL) | 1 | TEST FED | 1000 | |
| YAMAA-01 | CMAX | Max Conc (ng/mL) | 2 | REFERENCE FASTED | 0 | |
| YAMAA-01 | AUCIFO | AUC Infinity Obs (h*ng/mL) | 2 | REFERENCE FASTED | — | |
| YAMAA-01 | CMAX | Max Conc (ng/mL) | 3 | TEST FASTED | 100 | |
| YAMAA-01 | AUCIFO | AUC Infinity Obs (h*ng/mL) | 3 | TEST FASTED | 800 | |
| YAMAA-01 | CMAX | Max Conc (ng/mL) | 1 | TEST FASTED | 140 | |
| YAMAA-01 | AUCIFO | AUC Infinity Obs (h*ng/mL) | 1 | TEST FASTED | 1100 | |
| YAMAA-01 | CMAX | Max Conc (ng/mL) | 2 | TEST FED | 105 | |
| YAMAA-01 | AUCIFO | AUC Infinity Obs (h*ng/mL) | 2 | TEST FED | 1200 | |
| YAMAA-01 | CMAX | Max Conc (ng/mL) | 1 | REFERENCE FASTED | 95 | |
| YAMAA-01 | AUCIFO | AUC Infinity Obs (h*ng/mL) | 1 | REFERENCE FASTED | 850 | |
| YAMAA-01 | CMAX | Max Conc (ng/mL) | 2 | TEST FED | 70 | |
| YAMAA-01 | AUCIFO | AUC Infinity Obs (h*ng/mL) | 2 | TEST FED | 880 |
input/adsl.csv
Edit5 rows| STUDYID | USUBJID | TRTSEQP |
|---|---|---|
| YAMAA-01 | TEST FASTED/TEST FED/REFERENCE FASTED | |
| YAMAA-01 | TEST FED/REFERENCE FASTED/TEST FASTED | |
| YAMAA-01 | TEST FASTED/TEST FED/REFERENCE FASTED | |
| YAMAA-01 | REFERENCE FASTED/TEST FED/TEST FASTED | |
| YAMAA-01 | TEST FED/TEST FASTED/REFERENCE FASTED |
input/compare.csv
Edit2 rows| COMPNO | COMPARISON | TESTTRT | REFTRT |
|---|---|---|---|
| 1 | FOOD EFFECT | TEST FED | TEST FASTED |
| 2 | RELATIVE BIOAVAILABILITY | TEST FASTED | REFERENCE FASTED |
Expected output
Under review20 expected rowsexpected/adpp.csv
Edit20 rows| STUDYID | USUBJID | PARAMCD | PARAM | APERIOD | TRTA | TRTSEQP | AVAL | ANL01FL | ANL02FL |
|---|---|---|---|---|---|---|---|---|---|
| YAMAA-01 | CMAX | Max Conc (ng/mL) | 1 | TEST FASTED | TEST FASTED/TEST FED/REFERENCE FASTED | 120 | Y | Y | |
| YAMAA-01 | AUCIFO | AUC Infinity Obs (h*ng/mL) | 1 | TEST FASTED | TEST FASTED/TEST FED/REFERENCE FASTED | 960 | — | Y | |
| YAMAA-01 | CMAX | Max Conc (ng/mL) | 2 | TEST FED | TEST FASTED/TEST FED/REFERENCE FASTED | 90 | Y | — | |
| YAMAA-01 | AUCIFO | AUC Infinity Obs (h*ng/mL) | 2 | TEST FED | TEST FASTED/TEST FED/REFERENCE FASTED | — | — | — | |
| YAMAA-01 | CMAX | Max Conc (ng/mL) | 3 | REFERENCE FASTED | TEST FASTED/TEST FED/REFERENCE FASTED | 110 | — | Y | |
| YAMAA-01 | AUCIFO | AUC Infinity Obs (h*ng/mL) | 3 | REFERENCE FASTED | TEST FASTED/TEST FED/REFERENCE FASTED | 900 | — | Y | |
| YAMAA-01 | CMAX | Max Conc (ng/mL) | 1 | TEST FED | TEST FED/REFERENCE FASTED/TEST FASTED | 80 | Y | — | |
| YAMAA-01 | AUCIFO | AUC Infinity Obs (h*ng/mL) | 1 | TEST FED | TEST FED/REFERENCE FASTED/TEST FASTED | 1000 | Y | — | |
| YAMAA-01 | CMAX | Max Conc (ng/mL) | 2 | REFERENCE FASTED | TEST FED/REFERENCE FASTED/TEST FASTED | 0 | — | — | |
| YAMAA-01 | AUCIFO | AUC Infinity Obs (h*ng/mL) | 2 | REFERENCE FASTED | TEST FED/REFERENCE FASTED/TEST FASTED | — | — | — | |
| YAMAA-01 | CMAX | Max Conc (ng/mL) | 3 | TEST FASTED | TEST FED/REFERENCE FASTED/TEST FASTED | 100 | Y | — | |
| YAMAA-01 | AUCIFO | AUC Infinity Obs (h*ng/mL) | 3 | TEST FASTED | TEST FED/REFERENCE FASTED/TEST FASTED | 800 | Y | — | |
| YAMAA-01 | CMAX | Max Conc (ng/mL) | 1 | TEST FASTED | TEST FASTED/TEST FED/REFERENCE FASTED | 140 | Y | — | |
| YAMAA-01 | AUCIFO | AUC Infinity Obs (h*ng/mL) | 1 | TEST FASTED | TEST FASTED/TEST FED/REFERENCE FASTED | 1100 | Y | — | |
| YAMAA-01 | CMAX | Max Conc (ng/mL) | 2 | TEST FED | TEST FASTED/TEST FED/REFERENCE FASTED | 105 | Y | — | |
| YAMAA-01 | AUCIFO | AUC Infinity Obs (h*ng/mL) | 2 | TEST FED | TEST FASTED/TEST FED/REFERENCE FASTED | 1200 | Y | — | |
| YAMAA-01 | CMAX | Max Conc (ng/mL) | 1 | REFERENCE FASTED | REFERENCE FASTED/TEST FED/TEST FASTED | 95 | — | — | |
| YAMAA-01 | AUCIFO | AUC Infinity Obs (h*ng/mL) | 1 | REFERENCE FASTED | REFERENCE FASTED/TEST FED/TEST FASTED | 850 | — | — | |
| YAMAA-01 | CMAX | Max Conc (ng/mL) | 2 | TEST FED | REFERENCE FASTED/TEST FED/TEST FASTED | 70 | — | — | |
| YAMAA-01 | AUCIFO | AUC Infinity Obs (h*ng/mL) | 2 | TEST FED | REFERENCE FASTED/TEST FED/TEST FASTED | 880 | — | — |
Human review and assessment
| Dataset Name | Variable Name | Variable Label | Type | Codelist/Controlled Terms | Core | Computational Method | Origin |
|---|---|---|---|---|---|---|---|
| ADPP | STUDYID | Study Identifier | character | Copy value from ADPP_RAW.STUDYID. | Predecessor | ||
| ADPP | USUBJID | Unique Subject Identifier | character | Copy value from ADPP_RAW.USUBJID. | Predecessor | ||
| ADPP | PARAMCD | Parameter Code | character | Copy value from ADPP_RAW.PARAMCD. | Predecessor | ||
| ADPP | PARAM | Parameter | character | Copy value from ADPP_RAW.PARAM. | Predecessor | ||
| ADPP | APERIOD | Period | numeric | Copy value from ADPP_RAW.APERIOD. | Predecessor | ||
| ADPP | TRTA | Actual Treatment | character | Copy value from ADPP_RAW.TRTA. | Predecessor | ||
| ADPP | TRTSEQP | Planned Sequence of Treatments | character | Copy value from ADSL.TRTSEQP. | Predecessor | ||
| ADPP | AVAL | Analysis Value | numeric | Copy value from ADPP_RAW.AVAL. | Predecessor | ||
| ADPP | ANL01FL | Analysis Flag 01 | character | Y | "Y" when AVAL > 0 AND PAVAL1 > 0; blank otherwise. | Derived | |
| ADPP | ANL02FL | Analysis Flag 02 | character | Y | "Y" when AVAL > 0 AND PAVAL2 > 0; blank otherwise. | Derived |
| Variable | Permitted values |
|---|---|
| ANL01FL | Y |
| ANL02FL | Y |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.7 | 2026-10-20 | yamaa docs build | Read a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default | ||
| 1.6 | 2026-10-17 | yamaa docs build | Describe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight | ||
| 1.5 | 2026-10-17 | yamaa docs build | Name collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part | ||
| 1.4 | 2026-10-15 | yamaa docs build | State datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks | ||
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/adam-adpp-comparison-sets/full.md
Edit25 linesFollowing CDISC ADaM standards, use the provided ADPP_RAW, ADSL, and COMPARE datasets to create an ADPP dataset with one record per subject, parameter, and period.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, APERIOD, TRTA, TRTSEQP, AVAL, ANL01FL, ANL02FL
TRTSEQP is the subject's planned sequence of treatments. ANL01FL is "Y" on the records comparison 1 uses and has no value on every other record, and ANL02FL is "Y" on the records comparison 2 uses and has no value on every other record. A record is used by a comparison when its treatment is one of the comparison's two treatments, its own AVAL is evaluable, and the same subject has an evaluable AVAL for the same parameter under the comparison's other treatment. The two treatments of a comparison are the test treatment and the reference treatment COMPARE lists for it. An evaluable AVAL is present and greater than zero. A value that is missing or zero also takes its partner record out of that comparison. Pairs are judged per parameter, so a subject can be in a comparison for one parameter and out of it for another. A treatment that is the test treatment of one comparison and the reference treatment of the other can carry both flags. Every record stays in the dataset.
Read the source datasets from /app/input and save the completed dataset as /app/output/adpp.csv.
prompts/adam-adpp-comparison-sets/conventions.md
Edit12 linesFollowing CDISC ADaM standards, use the provided ADPP_RAW, ADSL, and COMPARE datasets to create an ADPP dataset with one record per subject, parameter, and period.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, APERIOD, TRTA, TRTSEQP, AVAL, ANL01FL, ANL02FL
ANL01FL is "Y" or has no value, and ANL02FL is "Y" or has no value.
Read the source datasets from /app/input and save the completed dataset as /app/output/adpp.csv.
prompts/adam-adpp-comparison-sets/brief.md
Edit10 linesFollowing CDISC ADaM standards, use the provided ADPP_RAW, ADSL, and COMPARE datasets to create an ADPP dataset with one record per subject, parameter, and period.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, APERIOD, TRTA, TRTSEQP, AVAL, ANL01FL, ANL02FL
Read the source datasets from /app/input and save the completed dataset as /app/output/adpp.csv.
R
Edit3 lineslibrary(yamaa)yamaa_spec("spec.yaml")$run()$outputPython
Edit3 linesimport yamaayamaa.spec("spec.yaml").run().outputsolutions/adam-adpp-comparison-sets/result.R
Edit56 lines# Reference solution for the yamaa benchmark adam-adpp-comparison-sets (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)raw <- read_csv( "/app/input/adpp_raw.csv", col_types = cols( APERIOD = col_integer(), AVAL = col_double(), .default = col_character() ))adsl <- read_csv("/app/input/adsl.csv", col_types = cols(.default = col_character()))compare <- read_csv( "/app/input/compare.csv", col_types = cols(COMPNO = col_integer(), .default = col_character()))keys <- c("STUDYID", "USUBJID", "PARAMCD")stopifnot(!anyDuplicated(raw[c(keys, "TRTA")]))adpp <- raw |> left_join(select(adsl, STUDYID, USUBJID, TRTSEQP), by = c("STUDYID", "USUBJID"))# Flag nn belongs to comparison nn. A record is used when its treatment is# one of the two, its own value is evaluable (present and > 0), and the same# subject has an evaluable value of the same parameter under the other one.partner <- raw |> transmute(STUDYID, USUBJID, PARAMCD, OTHER = TRTA, PAVAL = AVAL)for (number in 1:2) { row <- filter(compare, COMPNO == number) stopifnot(nrow(row) == 1) test <- row$TESTTRT ref <- row$REFTRT flag <- sprintf("ANL%02dFL", number) adpp <- adpp |> mutate(OTHER = case_when(TRTA == test ~ ref, TRTA == ref ~ test)) |> left_join(partner, by = c(keys, "OTHER")) |> mutate( "{flag}" := if_else( !is.na(OTHER) & !is.na(AVAL) & AVAL > 0 & !is.na(PAVAL) & PAVAL > 0, "Y", NA_character_ ) ) |> select(-OTHER, -PAVAL)}adpp <- adpp |> select(STUDYID, USUBJID, PARAMCD, PARAM, APERIOD, TRTA, TRTSEQP, AVAL, ANL01FL, ANL02FL)dir.create("/app/output", showWarnings = FALSE)write_csv(adpp, "/app/output/adpp.csv", na = "")solutions/adam-adpp-comparison-sets/result.py
Edit82 lines# Reference solution for the yamaa benchmark adam-adpp-comparison-sets (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1."""Build ADPP from ADPP_RAW, ADSL, and COMPARE.One record per subject, parameter, and period. ANL01FL / ANL02FL markthe records used by comparison 1 / comparison 2: a record is used whenits treatment is one of the comparison's two treatments, its own AVALis evaluable (present and greater than zero), and the same subject hasan evaluable AVAL for the same parameter under the comparison's othertreatment."""import csvINPUT_DIR = "/app/input"OUTPUT_DIR = "/app/output"def read_csv(name): with open(f"{INPUT_DIR}/{name}", newline="") as f: return list(csv.DictReader(f))def evaluable(value): if value is None or value == "": return False return float(value) > 0def main(): raw = read_csv("adpp_raw.csv") adsl = read_csv("adsl.csv") compare = read_csv("compare.csv") trtseqp = {row["USUBJID"]: row["TRTSEQP"] for row in adsl} comparisons = { int(row["COMPNO"]): (row["TESTTRT"], row["REFTRT"]) for row in compare } # Evaluable (subject, parameter, treatment) combinations present. evaluable_keys = set() for row in raw: if evaluable(row["AVAL"]): evaluable_keys.add((row["USUBJID"], row["PARAMCD"], row["TRTA"])) def used(row, compno): test, ref = comparisons[compno] if row["TRTA"] not in (test, ref): return False if not evaluable(row["AVAL"]): return False other = ref if row["TRTA"] == test else test return (row["USUBJID"], row["PARAMCD"], other) in evaluable_keys columns = [ "STUDYID", "USUBJID", "PARAMCD", "PARAM", "APERIOD", "TRTA", "TRTSEQP", "AVAL", "ANL01FL", "ANL02FL", ] with open(f"{OUTPUT_DIR}/adpp.csv", "w", newline="") as f: writer = csv.DictWriter(f, fieldnames=columns) writer.writeheader() for row in raw: writer.writerow({ "STUDYID": row["STUDYID"], "USUBJID": row["USUBJID"], "PARAMCD": row["PARAMCD"], "PARAM": row["PARAM"], "APERIOD": row["APERIOD"], "TRTA": row["TRTA"], "TRTSEQP": trtseqp.get(row["USUBJID"], ""), "AVAL": row["AVAL"], "ANL01FL": "Y" if used(row, 1) else "", "ANL02FL": "Y" if used(row, 2) else "", })if __name__ == "__main__": main()solutions/adam-adpp-comparison-sets/result.sas
Edit95 lines/* opensas reference solution for the yamaa benchmark adam-adpp-comparison-sets. Reads the benchmark inputs and derives the requested output. Execute with the existing opensas runtime: opensas /app/output/result.sas. */data raw; length STUDYID USUBJID PARAMCD PARAM TRTA $200; infile "/app/input/adpp_raw.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char200. USUBJID : $char200. PARAMCD : $char200. PARAM : $char200. APERIOD TRTA : $char200. AVAL; ORD = _n_;run;data adsl; length STUDYID USUBJID TRTSEQP $200; infile "/app/input/adsl.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char200. USUBJID : $char200. TRTSEQP : $char200.;run;data compare; length COMPARISON TESTTRT REFTRT $200; infile "/app/input/compare.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input COMPNO COMPARISON : $char200. TESTTRT : $char200. REFTRT : $char200.;run;/* Each comparison's two treatments, one column pair per comparison. */data pairs; length TEST1 REF1 TEST2 REF2 $200; retain TEST1 REF1 TEST2 REF2 N1 N2 0; set compare end=last; if COMPNO = 1 then do; TEST1 = TESTTRT; REF1 = REFTRT; N1 = N1 + 1; end; if COMPNO = 2 then do; TEST2 = TESTTRT; REF2 = REFTRT; N2 = N2 + 1; end; if last then do; if N1 ne 1 or N2 ne 1 then abort; output; end; keep TEST1 REF1 TEST2 REF2;run;/* The other treatment of each comparison for every record. */data others; length OTHER1 OTHER2 $200; if _n_ = 1 then set pairs; set raw; if TRTA = TEST1 then OTHER1 = REF1; else if TRTA = REF1 then OTHER1 = TEST1; if TRTA = TEST2 then OTHER2 = REF2; else if TRTA = REF2 then OTHER2 = TEST2;run;data partner; length PSTUDY PSUBJ PPARAM PTRT $200; set raw; PSTUDY = STUDYID; PSUBJ = USUBJID; PPARAM = PARAMCD; PTRT = TRTA; PAVAL = AVAL; keep PSTUDY PSUBJ PPARAM PTRT PAVAL;run;proc sql; create table with1 as select o.*, p.PAVAL as PAVAL1 from others as o left join partner as p on o.STUDYID = p.PSTUDY and o.USUBJID = p.PSUBJ and o.PARAMCD = p.PPARAM and o.OTHER1 = p.PTRT; create table with2 as select o.*, p.PAVAL as PAVAL2 from with1 as o left join partner as p on o.STUDYID = p.PSTUDY and o.USUBJID = p.PSUBJ and o.PARAMCD = p.PPARAM and o.OTHER2 = p.PTRT; create table withseq as select o.*, a.TRTSEQP from with2 as o left join adsl as a on o.STUDYID = a.STUDYID and o.USUBJID = a.USUBJID;quit;/* Y when the record and its partner under the comparison's other treatment are both present and greater than 0. */data final; length ANL01FL ANL02FL $1; set withseq; if not missing(OTHER1) and not missing(AVAL) and AVAL > 0 and not missing(PAVAL1) and PAVAL1 > 0 then ANL01FL = 'Y'; else ANL01FL = ' '; if not missing(OTHER2) and not missing(AVAL) and AVAL > 0 and not missing(PAVAL2) and PAVAL2 > 0 then ANL02FL = 'Y'; else ANL02FL = ' ';run;proc sort data=final; by ORD; run;proc sql; create table adpp as select STUDYID, USUBJID, PARAMCD, PARAM, APERIOD, TRTA, TRTSEQP, AVAL, ANL01FL, ANL02FL from final;quit;proc export data=adpp outfile="/app/output/adpp.csv" dbms=csv replace; run;