As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.
We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.
Summary
README.mdPublic entries: Python and R. Passing coverage requires an installed inventory.
Goal: derive, for each subject of a three-period study, the geometric
mean of the metabolite-to-parent ratio of maximum concentration across
periods (PARAMCD MRCMAXGM), and the standard deviation of the replicate
assays of the subject's 2-hour concentration sample (C2HREPSD).
Input: the SDTM pharmacokinetic (PK) parameters (PP), one metabolite
ratio of maximum concentration (MRCMAX) per subject and period, with no
result when the ratio was not calculated; and the SDTM PK concentrations
(PC), one record per assay of a sample, the replicate assays of one sample
sharing its planned time point (PCTPT) and numbered by PCREPNUM.
Variables:
PARAMCD,PARAM:MRCMAXGM"Geometric Mean Metabolite Ratio Cmax" andC2HREPSD"Replicate SD of 2 H Concentration".AVAL: forMRCMAXGM, the geometric mean of the subject's ratios above zero: the exponential of the mean of their natural logarithms. A ratio of zero, from a metabolite concentration below the limit of quantification, has no logarithm and is left out, as is a ratio that was not calculated. ForC2HREPSD, the sample standard deviation, with divisor one less than the number of assays, of the subject's 2-hour concentrations; it has no value for a sample assayed once. Neither is rounded.AVALU:RATIOforMRCMAXGMandng/mLforC2HREPSD.
Inputs
Under review2 source filesinput/pc.csv
Edit13 rows| STUDYID | DOMAIN | USUBJID | PCSEQ | PCTESTCD | PCTEST | PCSTRESN | PCSTRESU | PCREPNUM | VISITNUM | VISIT | PCTPT | PCTPTNUM |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| YAMAA-01 | PC | 1 | DRUGX | Drug X | 1.2 | ng/mL | 1 | 1 | PERIOD 1 | PREDOSE | 0 | |
| YAMAA-01 | PC | 2 | DRUGX | Drug X | 102.4 | ng/mL | 1 | 1 | PERIOD 1 | 2H | 2 | |
| YAMAA-01 | PC | 3 | DRUGX | Drug X | 98.6 | ng/mL | 2 | 1 | PERIOD 1 | 2H | 2 | |
| YAMAA-01 | PC | 4 | DRUGX | Drug X | 100.1 | ng/mL | 3 | 1 | PERIOD 1 | 2H | 2 | |
| YAMAA-01 | PC | 1 | DRUGX | Drug X | 0.8 | ng/mL | 1 | 1 | PERIOD 1 | PREDOSE | 0 | |
| YAMAA-01 | PC | 2 | DRUGX | Drug X | 85 | ng/mL | 1 | 1 | PERIOD 1 | 2H | 2 | |
| YAMAA-01 | PC | 3 | DRUGX | Drug X | 88 | ng/mL | 2 | 1 | PERIOD 1 | 2H | 2 | |
| YAMAA-01 | PC | 1 | DRUGX | Drug X | 1.5 | ng/mL | 1 | 1 | PERIOD 1 | PREDOSE | 0 | |
| YAMAA-01 | PC | 2 | DRUGX | Drug X | 120.5 | ng/mL | 1 | 1 | PERIOD 1 | 2H | 2 | |
| YAMAA-01 | PC | 1 | DRUGX | Drug X | 0.9 | ng/mL | 1 | 1 | PERIOD 1 | PREDOSE | 0 | |
| YAMAA-01 | PC | 2 | DRUGX | Drug X | 64.2 | ng/mL | 1 | 1 | PERIOD 1 | 2H | 2 | |
| YAMAA-01 | PC | 3 | DRUGX | Drug X | 66 | ng/mL | 2 | 1 | PERIOD 1 | 2H | 2 | |
| YAMAA-01 | PC | 4 | DRUGX | Drug X | 63.9 | ng/mL | 3 | 1 | PERIOD 1 | 2H | 2 |
input/pp.csv
Edit12 rows| STUDYID | DOMAIN | USUBJID | PPSEQ | PPTESTCD | PPTEST | PPSTRESN | PPSTRESU | PPSTAT | VISITNUM | VISIT |
|---|---|---|---|---|---|---|---|---|---|---|
| YAMAA-01 | PP | 1 | MRCMAX | Metabolite Ratio Cmax | 0.4 | RATIO | — | 1 | PERIOD 1 | |
| YAMAA-01 | PP | 2 | MRCMAX | Metabolite Ratio Cmax | 0.5 | RATIO | — | 2 | PERIOD 2 | |
| YAMAA-01 | PP | 3 | MRCMAX | Metabolite Ratio Cmax | 0.8 | RATIO | — | 3 | PERIOD 3 | |
| YAMAA-01 | PP | 1 | MRCMAX | Metabolite Ratio Cmax | 1.1 | RATIO | — | 1 | PERIOD 1 | |
| YAMAA-01 | PP | 2 | MRCMAX | Metabolite Ratio Cmax | 0.95 | RATIO | — | 2 | PERIOD 2 | |
| YAMAA-01 | PP | 3 | MRCMAX | Metabolite Ratio Cmax | 1.3 | RATIO | — | 3 | PERIOD 3 | |
| YAMAA-01 | PP | 1 | MRCMAX | Metabolite Ratio Cmax | 0.6 | RATIO | — | 1 | PERIOD 1 | |
| YAMAA-01 | PP | 2 | MRCMAX | Metabolite Ratio Cmax | 0 | RATIO | — | 2 | PERIOD 2 | |
| YAMAA-01 | PP | 3 | MRCMAX | Metabolite Ratio Cmax | 0.9 | RATIO | — | 3 | PERIOD 3 | |
| YAMAA-01 | PP | 1 | MRCMAX | Metabolite Ratio Cmax | 0.3 | RATIO | — | 1 | PERIOD 1 | |
| YAMAA-01 | PP | 2 | MRCMAX | Metabolite Ratio Cmax | — | — | NOT DONE | 2 | PERIOD 2 | |
| YAMAA-01 | PP | 3 | MRCMAX | Metabolite Ratio Cmax | 0.45 | RATIO | — | 3 | PERIOD 3 |
Expected output
Under review8 expected rowsexpected/adpp.csv
Edit8 rows| STUDYID | USUBJID | PARAMCD | PARAM | AVAL | AVALU |
|---|---|---|---|---|---|
| YAMAA-01 | C2HREPSD | Replicate SD of 2 H Concentration | 1.9139836293274184 | ng/mL | |
| YAMAA-01 | MRCMAXGM | Geometric Mean Metabolite Ratio Cmax | 0.5428835233189813 | RATIO | |
| YAMAA-01 | C2HREPSD | Replicate SD of 2 H Concentration | 2.1213203435596424 | ng/mL | |
| YAMAA-01 | MRCMAXGM | Geometric Mean Metabolite Ratio Cmax | 1.107524173693196 | RATIO | |
| YAMAA-01 | C2HREPSD | Replicate SD of 2 H Concentration | — | ng/mL | |
| YAMAA-01 | MRCMAXGM | Geometric Mean Metabolite Ratio Cmax | 0.7348469228349535 | RATIO | |
| YAMAA-01 | C2HREPSD | Replicate SD of 2 H Concentration | 1.1357816691600546 | ng/mL | |
| YAMAA-01 | MRCMAXGM | Geometric Mean Metabolite Ratio Cmax | 0.3674234614174767 | RATIO |
Human review and assessment
| Dataset Name | Variable Name | Variable Label | Type | Codelist/Controlled Terms | Core | Computational Method | Origin |
|---|---|---|---|---|---|---|---|
| ADPP | STUDYID | Study Identifier | character | Set per constructed row; see the Row construction sheet. | Derived | ||
| ADPP | USUBJID | Unique Subject Identifier | character | Set per constructed row; see the Row construction sheet. | Derived | ||
| ADPP | PARAMCD | Parameter Code | character | Set per constructed row; see the Row construction sheet. | Derived | ||
| ADPP | PARAM | Parameter | character | Set per constructed row; see the Row construction sheet. | Derived | ||
| ADPP | AVAL | Analysis Value | numeric | Set per constructed row; see the Row construction sheet. | Derived | ||
| ADPP | AVALU | Analysis Value Unit | character | Set per constructed row; see the Row construction sheet. | Derived |
| Template | Include when | STUDYID | USUBJID | PARAMCD | PARAM | AVAL | AVALU |
|---|---|---|---|---|---|---|---|
| ratio_geomean | Copy PP.STUDYID | Copy PP.USUBJID | Constant "MRCMAXGM" | Constant "Geometric Mean Metabolite Ratio Cmax" | Aggregate GEOMEAN(PP.PPSTRESN) over records matching on the shared output keys, where PP.PPTESTCD = 'MRCMAX' AND PP.PPSTRESN > 0 | Constant "RATIO" | |
| replicate_sd | Copy PC.STUDYID | Copy PC.USUBJID | Constant "C2HREPSD" | Constant "Replicate SD of 2 H Concentration" | Aggregate STDDEV(PC.PCSTRESN) over records matching on the shared output keys, where PC.PCTPT = '2H' | Constant "ng/mL" |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.7 | 2026-10-20 | yamaa docs build | Read a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default | ||
| 1.6 | 2026-10-17 | yamaa docs build | Describe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight | ||
| 1.5 | 2026-10-17 | yamaa docs build | Name collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part | ||
| 1.4 | 2026-10-15 | yamaa docs build | State datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks | ||
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/adam-adpp-geometric-mean-ratio/full.md
Edit26 linesFollowing CDISC ADaM standards, use the provided PC and PP datasets to create an ADPP dataset with one record per subject per parameter.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, AVAL, AVALU
Each subject has two records, one for each parameter:
- PARAMCD "MRCMAXGM", PARAM "Geometric Mean Metabolite Ratio Cmax", AVALU "RATIO".
- PARAMCD "C2HREPSD", PARAM "Replicate SD of 2 H Concentration", AVALU "ng/mL".
For MRCMAXGM, AVAL is the geometric mean, across the periods, of the subject's MRCMAX metabolite ratios (PPSTRESN) above zero: the exponential of the mean of their natural logarithms. A ratio of zero, from a metabolite concentration below the limit of quantification, has no logarithm and is left out, as is a ratio that was not calculated.
For C2HREPSD, AVAL is the sample standard deviation, with divisor one less than the number of assays, of the subject's replicate concentrations (PCSTRESN) at planned time point "2H". AVAL has no value when the subject's 2-hour sample was assayed once; that subject still has its C2HREPSD record. AVAL is not rounded for either parameter.
Read the source datasets from /app/input and save the completed dataset as /app/output/adpp.csv.
prompts/adam-adpp-geometric-mean-ratio/conventions.md
Edit12 linesFollowing CDISC ADaM standards, use the provided PC and PP datasets to create an ADPP dataset with one record per subject per parameter.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, AVAL, AVALU
PARAMCD is "MRCMAXGM" with PARAM "Geometric Mean Metabolite Ratio Cmax" and AVALU "RATIO", or "C2HREPSD" with PARAM "Replicate SD of 2 H Concentration" and AVALU "ng/mL". AVAL is not rounded.
Read the source datasets from /app/input and save the completed dataset as /app/output/adpp.csv.
prompts/adam-adpp-geometric-mean-ratio/brief.md
Edit8 linesFollowing CDISC ADaM standards, use the provided PC and PP datasets to create an ADPP dataset with one record per subject per parameter.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, AVAL, AVALU
Read the source datasets from /app/input and save the completed dataset as /app/output/adpp.csv.
R
Edit3 lineslibrary(yamaa)yamaa_spec("spec.yaml")$run()$outputPython
Edit3 linesimport yamaayamaa.spec("spec.yaml").run().outputsolutions/adam-adpp-geometric-mean-ratio/result.R
Edit54 lines# Reference solution for the yamaa benchmark adam-adpp-geometric-mean-ratio (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)pp <- read_csv( "/app/input/pp.csv", col_types = cols(PPSTRESN = col_double(), .default = col_character()))pc <- read_csv( "/app/input/pc.csv", col_types = cols(PCSTRESN = col_double(), .default = col_character()))geomean <- pp |> group_by(STUDYID, USUBJID) |> summarise( AVAL = { x <- PPSTRESN[PPTESTCD == "MRCMAX" & !is.na(PPSTRESN) & PPSTRESN > 0] if (length(x) > 0) exp(mean(log(x))) else NA_real_ }, .groups = "drop" ) |> mutate( PARAMCD = "MRCMAXGM", PARAM = "Geometric Mean Metabolite Ratio Cmax", AVALU = "RATIO" )replicate_sd <- pc |> group_by(STUDYID, USUBJID) |> summarise( AVAL = { x <- PCSTRESN[PCTPT == "2H" & !is.na(PCSTRESN)] if (length(x) >= 2) sd(x) else NA_real_ }, .groups = "drop" ) |> mutate( PARAMCD = "C2HREPSD", PARAM = "Replicate SD of 2 H Concentration", AVALU = "ng/mL" )adpp <- bind_rows(geomean, replicate_sd) |> arrange(STUDYID, USUBJID, PARAMCD) |> select(STUDYID, USUBJID, PARAMCD, PARAM, AVAL, AVALU)dir.create("/app/output", showWarnings = FALSE)write_csv(adpp, "/app/output/adpp.csv", na = "")solutions/adam-adpp-geometric-mean-ratio/result.py
Edit87 lines# Reference solution for the yamaa benchmark adam-adpp-geometric-mean-ratio (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.import csvimport mathimport statisticsPP_CSV = "/app/input/pp.csv"PC_CSV = "/app/input/pc.csv"OUTPUT_CSV = "/app/output/adpp.csv"def read(path): with open(path, newline="") as f: return list(csv.DictReader(f))def number(text): text = text.strip() return float(text) if text else Nonedef cell(value): return "" if value is None else repr(value)def main(): pp = read(PP_CSV) pc = read(PC_CSV) subjects = [] for row in pp + pc: key = (row["STUDYID"], row["USUBJID"]) if key not in subjects: subjects.append(key) out_rows = [] for study, subject in subjects: ratios = [ number(r["PPSTRESN"]) for r in pp if (r["STUDYID"], r["USUBJID"]) == (study, subject) and r["PPTESTCD"] == "MRCMAX" ] ratios = [x for x in ratios if x is not None and x > 0] geomean = ( math.exp(sum(math.log(x) for x in ratios) / len(ratios)) if ratios else None ) out_rows.append( { "STUDYID": study, "USUBJID": subject, "PARAMCD": "MRCMAXGM", "PARAM": "Geometric Mean Metabolite Ratio Cmax", "AVAL": cell(geomean), "AVALU": "RATIO", } ) replicates = [ number(r["PCSTRESN"]) for r in pc if (r["STUDYID"], r["USUBJID"]) == (study, subject) and r["PCTPT"] == "2H" ] replicates = [x for x in replicates if x is not None] sd = statistics.stdev(replicates) if len(replicates) >= 2 else None out_rows.append( { "STUDYID": study, "USUBJID": subject, "PARAMCD": "C2HREPSD", "PARAM": "Replicate SD of 2 H Concentration", "AVAL": cell(sd), "AVALU": "ng/mL", } ) fieldnames = ["STUDYID", "USUBJID", "PARAMCD", "PARAM", "AVAL", "AVALU"] with open(OUTPUT_CSV, "w", newline="") as f: writer = csv.DictWriter(f, fieldnames=fieldnames) writer.writeheader() writer.writerows(out_rows)if __name__ == "__main__": main()solutions/adam-adpp-geometric-mean-ratio/result.sas
Edit69 lines/* opensas reference solution for the yamaa benchmark adam-adpp-geometric-mean-ratio. Reads the benchmark inputs and derives the requested output. Execute with the existing opensas runtime: opensas /app/output/result.sas. */data pp; length STUDYID DOMAIN USUBJID PPTESTCD PPTEST PPSTRESU PPSTAT VISIT $40; infile "/app/input/pp.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char40. DOMAIN : $char40. USUBJID : $char40. PPSEQ PPTESTCD : $char40. PPTEST : $char40. PPSTRESN PPSTRESU : $char40. PPSTAT : $char40. VISITNUM VISIT : $char40.;run;data pc; length STUDYID DOMAIN USUBJID PCTESTCD PCTEST PCSTRESU VISIT PCTPT $40; infile "/app/input/pc.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char40. DOMAIN : $char40. USUBJID : $char40. PCSEQ PCTESTCD : $char40. PCTEST : $char40. PCSTRESN PCSTRESU : $char40. PCREPNUM VISITNUM VISIT : $char40. PCTPT : $char40. PCTPTNUM;run;/* The natural logarithm of each metabolite ratio above zero. */data logs; set pp; if PPTESTCD = 'MRCMAX' and not missing(PPSTRESN) and PPSTRESN > 0; LOGRATIO = log(PPSTRESN);run;/* The replicate 2-hour concentrations and their mean per subject. */data reps; set pc; if PCTPT = '2H' and not missing(PCSTRESN);run;proc sql; create table means as select STUDYID, USUBJID, mean(PCSTRESN) as M, count(*) as N from reps group by STUDYID, USUBJID; create table squares as select r.STUDYID, r.USUBJID, m.N, (r.PCSTRESN - m.M) * (r.PCSTRESN - m.M) as SQ from reps as r inner join means as m on m.STUDYID = r.STUDYID and m.USUBJID = r.USUBJID; create table subjects as select distinct STUDYID, USUBJID from pp union select distinct STUDYID, USUBJID from pc; create table geomean as select STUDYID, USUBJID, exp(mean(LOGRATIO)) as AVAL from logs group by STUDYID, USUBJID; create table sd as select STUDYID, USUBJID, case when N >= 2 then sqrt(sum(SQ) / (N - 1)) else . end as AVAL from squares group by STUDYID, USUBJID, N; create table final as select s.STUDYID, s.USUBJID, 'MRCMAXGM' as PARAMCD length=8, 'Geometric Mean Metabolite Ratio Cmax' as PARAM length=40, g.AVAL, 'RATIO' as AVALU length=8 from subjects as s left join geomean as g on g.STUDYID = s.STUDYID and g.USUBJID = s.USUBJID union all select s.STUDYID, s.USUBJID, 'C2HREPSD' as PARAMCD length=8, 'Replicate SD of 2 H Concentration' as PARAM length=40, d.AVAL, 'ng/mL' as AVALU length=8 from subjects as s left join sd as d on d.STUDYID = s.STUDYID and d.USUBJID = s.USUBJID order by 1, 2, 3;quit;proc export data=final outfile="/app/output/adpp.csv" dbms=csv replace; run;
yamaa