Review ask

As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.

We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.

Summary

Lifecycle: draftEditUnder review
What this benchmark means
README.md

Public entries: Python and R. Passing coverage requires an installed inventory.

Goal: derive, for each subject of a three-period study, the geometric mean of the metabolite-to-parent ratio of maximum concentration across periods (PARAMCD MRCMAXGM), and the standard deviation of the replicate assays of the subject's 2-hour concentration sample (C2HREPSD).

Input: the SDTM pharmacokinetic (PK) parameters (PP), one metabolite ratio of maximum concentration (MRCMAX) per subject and period, with no result when the ratio was not calculated; and the SDTM PK concentrations (PC), one record per assay of a sample, the replicate assays of one sample sharing its planned time point (PCTPT) and numbered by PCREPNUM.

Variables:

  • PARAMCD, PARAM: MRCMAXGM "Geometric Mean Metabolite Ratio Cmax" and C2HREPSD "Replicate SD of 2 H Concentration".
  • AVAL: for MRCMAXGM, the geometric mean of the subject's ratios above zero: the exponential of the mean of their natural logarithms. A ratio of zero, from a metabolite concentration below the limit of quantification, has no logarithm and is left out, as is a ratio that was not calculated. For C2HREPSD, the sample standard deviation, with divisor one less than the number of assays, of the subject's 2-hour concentrations; it has no value for a sample assayed once. Neither is rounded.
  • AVALU: RATIO for MRCMAXGM and ng/mL for C2HREPSD.

Inputs

Under review
2 source files

input/pc.csv

Edit
13 rows
input/pc.csv
STUDYIDDOMAINUSUBJIDPCSEQPCTESTCDPCTESTPCSTRESNPCSTRESUPCREPNUMVISITNUMVISITPCTPTPCTPTNUM
YAMAA-01PC1DRUGXDrug X1.2ng/mL11PERIOD 1PREDOSE0
YAMAA-01PC2DRUGXDrug X102.4ng/mL11PERIOD 12H2
YAMAA-01PC3DRUGXDrug X98.6ng/mL21PERIOD 12H2
YAMAA-01PC4DRUGXDrug X100.1ng/mL31PERIOD 12H2
YAMAA-01PC1DRUGXDrug X0.8ng/mL11PERIOD 1PREDOSE0
YAMAA-01PC2DRUGXDrug X85ng/mL11PERIOD 12H2
YAMAA-01PC3DRUGXDrug X88ng/mL21PERIOD 12H2
YAMAA-01PC1DRUGXDrug X1.5ng/mL11PERIOD 1PREDOSE0
YAMAA-01PC2DRUGXDrug X120.5ng/mL11PERIOD 12H2
YAMAA-01PC1DRUGXDrug X0.9ng/mL11PERIOD 1PREDOSE0
YAMAA-01PC2DRUGXDrug X64.2ng/mL11PERIOD 12H2
YAMAA-01PC3DRUGXDrug X66ng/mL21PERIOD 12H2
YAMAA-01PC4DRUGXDrug X63.9ng/mL31PERIOD 12H2

input/pp.csv

Edit
12 rows
input/pp.csv
STUDYIDDOMAINUSUBJIDPPSEQPPTESTCDPPTESTPPSTRESNPPSTRESUPPSTATVISITNUMVISIT
YAMAA-01PP1MRCMAXMetabolite Ratio Cmax0.4RATIO—1PERIOD 1
YAMAA-01PP2MRCMAXMetabolite Ratio Cmax0.5RATIO—2PERIOD 2
YAMAA-01PP3MRCMAXMetabolite Ratio Cmax0.8RATIO—3PERIOD 3
YAMAA-01PP1MRCMAXMetabolite Ratio Cmax1.1RATIO—1PERIOD 1
YAMAA-01PP2MRCMAXMetabolite Ratio Cmax0.95RATIO—2PERIOD 2
YAMAA-01PP3MRCMAXMetabolite Ratio Cmax1.3RATIO—3PERIOD 3
YAMAA-01PP1MRCMAXMetabolite Ratio Cmax0.6RATIO—1PERIOD 1
YAMAA-01PP2MRCMAXMetabolite Ratio Cmax0RATIO—2PERIOD 2
YAMAA-01PP3MRCMAXMetabolite Ratio Cmax0.9RATIO—3PERIOD 3
YAMAA-01PP1MRCMAXMetabolite Ratio Cmax0.3RATIO—1PERIOD 1
YAMAA-01PP2MRCMAXMetabolite Ratio Cmax——NOT DONE2PERIOD 2
YAMAA-01PP3MRCMAXMetabolite Ratio Cmax0.45RATIO—3PERIOD 3

Expected output

Under review
8 expected rows

expected/adpp.csv

Edit
8 rows
expected/adpp.csv
STUDYIDUSUBJIDPARAMCDPARAMAVALAVALU
YAMAA-01C2HREPSDReplicate SD of 2 H Concentration1.9139836293274184ng/mL
YAMAA-01MRCMAXGMGeometric Mean Metabolite Ratio Cmax0.5428835233189813RATIO
YAMAA-01C2HREPSDReplicate SD of 2 H Concentration2.1213203435596424ng/mL
YAMAA-01MRCMAXGMGeometric Mean Metabolite Ratio Cmax1.107524173693196RATIO
YAMAA-01C2HREPSDReplicate SD of 2 H Concentration—ng/mL
YAMAA-01MRCMAXGMGeometric Mean Metabolite Ratio Cmax0.7348469228349535RATIO
YAMAA-01C2HREPSDReplicate SD of 2 H Concentration1.1357816691600546ng/mL
YAMAA-01MRCMAXGMGeometric Mean Metabolite Ratio Cmax0.3674234614174767RATIO
Shaded columns are derived— = missing value

Human review and assessment

Your review

3 questions and your comments · View benchmark discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Dataset NameVariable NameVariable LabelTypeCodelist/Controlled TermsCoreComputational MethodOrigin
ADPPSTUDYIDStudy IdentifiercharacterSet per constructed row; see the Row construction sheet.Derived
ADPPUSUBJIDUnique Subject IdentifiercharacterSet per constructed row; see the Row construction sheet.Derived
ADPPPARAMCDParameter CodecharacterSet per constructed row; see the Row construction sheet.Derived
ADPPPARAMParametercharacterSet per constructed row; see the Row construction sheet.Derived
ADPPAVALAnalysis ValuenumericSet per constructed row; see the Row construction sheet.Derived
ADPPAVALUAnalysis Value UnitcharacterSet per constructed row; see the Row construction sheet.Derived
TemplateInclude whenSTUDYIDUSUBJIDPARAMCDPARAMAVALAVALU
ratio_geomeanCopy PP.STUDYIDCopy PP.USUBJIDConstant "MRCMAXGM"Constant "Geometric Mean Metabolite Ratio Cmax"Aggregate GEOMEAN(PP.PPSTRESN) over records matching on the shared output keys, where PP.PPTESTCD = 'MRCMAX' AND PP.PPSTRESN > 0Constant "RATIO"
replicate_sdCopy PC.STUDYIDCopy PC.USUBJIDConstant "C2HREPSD"Constant "Replicate SD of 2 H Concentration"Aggregate STDDEV(PC.PCSTRESN) over records matching on the shared output keys, where PC.PCTPT = '2H'Constant "ng/mL"
VersionDateAuthorDescriptionReviewerSign-off
1.72026-10-20yamaa docs buildRead a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default
1.62026-10-17yamaa docs buildDescribe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight
1.52026-10-17yamaa docs buildName collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part
1.42026-10-15yamaa docs buildState datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/adam-adpp-geometric-mean-ratio/full.md

Edit
26 lines

Following CDISC ADaM standards, use the provided PC and PP datasets to create an ADPP dataset with one record per subject per parameter.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, AVAL, AVALU

Each subject has two records, one for each parameter:

  • PARAMCD "MRCMAXGM", PARAM "Geometric Mean Metabolite Ratio Cmax", AVALU "RATIO".
  • PARAMCD "C2HREPSD", PARAM "Replicate SD of 2 H Concentration", AVALU "ng/mL".

For MRCMAXGM, AVAL is the geometric mean, across the periods, of the subject's MRCMAX metabolite ratios (PPSTRESN) above zero: the exponential of the mean of their natural logarithms. A ratio of zero, from a metabolite concentration below the limit of quantification, has no logarithm and is left out, as is a ratio that was not calculated.

For C2HREPSD, AVAL is the sample standard deviation, with divisor one less than the number of assays, of the subject's replicate concentrations (PCSTRESN) at planned time point "2H". AVAL has no value when the subject's 2-hour sample was assayed once; that subject still has its C2HREPSD record. AVAL is not rounded for either parameter.

Read the source datasets from /app/input and save the completed dataset as /app/output/adpp.csv.

prompts/adam-adpp-geometric-mean-ratio/conventions.md

Edit
12 lines

Following CDISC ADaM standards, use the provided PC and PP datasets to create an ADPP dataset with one record per subject per parameter.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, AVAL, AVALU

PARAMCD is "MRCMAXGM" with PARAM "Geometric Mean Metabolite Ratio Cmax" and AVALU "RATIO", or "C2HREPSD" with PARAM "Replicate SD of 2 H Concentration" and AVALU "ng/mL". AVAL is not rounded.

Read the source datasets from /app/input and save the completed dataset as /app/output/adpp.csv.

prompts/adam-adpp-geometric-mean-ratio/brief.md

Edit
8 lines

Following CDISC ADaM standards, use the provided PC and PP datasets to create an ADPP dataset with one record per subject per parameter.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, AVAL, AVALU

Read the source datasets from /app/input and save the completed dataset as /app/output/adpp.csv.

yamaa code

Public calls generated from benchmark.yaml; results must match the expected output

R

Edit
3 lines
library(yamaa)yamaa_spec("spec.yaml")$run()$output

Python

Edit
3 lines
import yamaayamaa.spec("spec.yaml").run().output

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/adam-adpp-geometric-mean-ratio/result.R

Edit
54 lines
# Reference solution for the yamaa benchmark adam-adpp-geometric-mean-ratio (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)pp <- read_csv(  "/app/input/pp.csv",  col_types = cols(PPSTRESN = col_double(), .default = col_character()))pc <- read_csv(  "/app/input/pc.csv",  col_types = cols(PCSTRESN = col_double(), .default = col_character()))geomean <- pp |>  group_by(STUDYID, USUBJID) |>  summarise(    AVAL = {      x <- PPSTRESN[PPTESTCD == "MRCMAX" & !is.na(PPSTRESN) & PPSTRESN > 0]      if (length(x) > 0) exp(mean(log(x))) else NA_real_    },    .groups = "drop"  ) |>  mutate(    PARAMCD = "MRCMAXGM",    PARAM = "Geometric Mean Metabolite Ratio Cmax",    AVALU = "RATIO"  )replicate_sd <- pc |>  group_by(STUDYID, USUBJID) |>  summarise(    AVAL = {      x <- PCSTRESN[PCTPT == "2H" & !is.na(PCSTRESN)]      if (length(x) >= 2) sd(x) else NA_real_    },    .groups = "drop"  ) |>  mutate(    PARAMCD = "C2HREPSD",    PARAM = "Replicate SD of 2 H Concentration",    AVALU = "ng/mL"  )adpp <- bind_rows(geomean, replicate_sd) |>  arrange(STUDYID, USUBJID, PARAMCD) |>  select(STUDYID, USUBJID, PARAMCD, PARAM, AVAL, AVALU)dir.create("/app/output", showWarnings = FALSE)write_csv(adpp, "/app/output/adpp.csv", na = "")

solutions/adam-adpp-geometric-mean-ratio/result.py

Edit
87 lines
# Reference solution for the yamaa benchmark adam-adpp-geometric-mean-ratio (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.import csvimport mathimport statisticsPP_CSV = "/app/input/pp.csv"PC_CSV = "/app/input/pc.csv"OUTPUT_CSV = "/app/output/adpp.csv"def read(path):    with open(path, newline="") as f:        return list(csv.DictReader(f))def number(text):    text = text.strip()    return float(text) if text else Nonedef cell(value):    return "" if value is None else repr(value)def main():    pp = read(PP_CSV)    pc = read(PC_CSV)    subjects = []    for row in pp + pc:        key = (row["STUDYID"], row["USUBJID"])        if key not in subjects:            subjects.append(key)    out_rows = []    for study, subject in subjects:        ratios = [            number(r["PPSTRESN"])            for r in pp            if (r["STUDYID"], r["USUBJID"]) == (study, subject)            and r["PPTESTCD"] == "MRCMAX"        ]        ratios = [x for x in ratios if x is not None and x > 0]        geomean = (            math.exp(sum(math.log(x) for x in ratios) / len(ratios)) if ratios else None        )        out_rows.append(            {                "STUDYID": study,                "USUBJID": subject,                "PARAMCD": "MRCMAXGM",                "PARAM": "Geometric Mean Metabolite Ratio Cmax",                "AVAL": cell(geomean),                "AVALU": "RATIO",            }        )        replicates = [            number(r["PCSTRESN"])            for r in pc            if (r["STUDYID"], r["USUBJID"]) == (study, subject) and r["PCTPT"] == "2H"        ]        replicates = [x for x in replicates if x is not None]        sd = statistics.stdev(replicates) if len(replicates) >= 2 else None        out_rows.append(            {                "STUDYID": study,                "USUBJID": subject,                "PARAMCD": "C2HREPSD",                "PARAM": "Replicate SD of 2 H Concentration",                "AVAL": cell(sd),                "AVALU": "ng/mL",            }        )    fieldnames = ["STUDYID", "USUBJID", "PARAMCD", "PARAM", "AVAL", "AVALU"]    with open(OUTPUT_CSV, "w", newline="") as f:        writer = csv.DictWriter(f, fieldnames=fieldnames)        writer.writeheader()        writer.writerows(out_rows)if __name__ == "__main__":    main()

solutions/adam-adpp-geometric-mean-ratio/result.sas

Edit
69 lines
/* opensas reference solution for the yamaa benchmark adam-adpp-geometric-mean-ratio.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data pp;  length STUDYID DOMAIN USUBJID PPTESTCD PPTEST PPSTRESU PPSTAT VISIT $40;  infile "/app/input/pp.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char40. DOMAIN : $char40. USUBJID : $char40. PPSEQ        PPTESTCD : $char40. PPTEST : $char40. PPSTRESN PPSTRESU : $char40.        PPSTAT : $char40. VISITNUM VISIT : $char40.;run;data pc;  length STUDYID DOMAIN USUBJID PCTESTCD PCTEST PCSTRESU VISIT PCTPT $40;  infile "/app/input/pc.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char40. DOMAIN : $char40. USUBJID : $char40. PCSEQ        PCTESTCD : $char40. PCTEST : $char40. PCSTRESN PCSTRESU : $char40.        PCREPNUM VISITNUM VISIT : $char40. PCTPT : $char40. PCTPTNUM;run;/* The natural logarithm of each metabolite ratio above zero. */data logs;  set pp;  if PPTESTCD = 'MRCMAX' and not missing(PPSTRESN) and PPSTRESN > 0;  LOGRATIO = log(PPSTRESN);run;/* The replicate 2-hour concentrations and their mean per subject. */data reps;  set pc;  if PCTPT = '2H' and not missing(PCSTRESN);run;proc sql;  create table means as    select STUDYID, USUBJID, mean(PCSTRESN) as M, count(*) as N    from reps group by STUDYID, USUBJID;  create table squares as    select r.STUDYID, r.USUBJID, m.N,           (r.PCSTRESN - m.M) * (r.PCSTRESN - m.M) as SQ    from reps as r inner join means as m      on m.STUDYID = r.STUDYID and m.USUBJID = r.USUBJID;  create table subjects as    select distinct STUDYID, USUBJID from pp    union    select distinct STUDYID, USUBJID from pc;  create table geomean as    select STUDYID, USUBJID, exp(mean(LOGRATIO)) as AVAL    from logs group by STUDYID, USUBJID;  create table sd as    select STUDYID, USUBJID,           case when N >= 2 then sqrt(sum(SQ) / (N - 1)) else . end as AVAL    from squares group by STUDYID, USUBJID, N;  create table final as    select s.STUDYID, s.USUBJID, 'MRCMAXGM' as PARAMCD length=8,           'Geometric Mean Metabolite Ratio Cmax' as PARAM length=40,           g.AVAL, 'RATIO' as AVALU length=8    from subjects as s left join geomean as g      on g.STUDYID = s.STUDYID and g.USUBJID = s.USUBJID    union all    select s.STUDYID, s.USUBJID, 'C2HREPSD' as PARAMCD length=8,           'Replicate SD of 2 H Concentration' as PARAM length=40,           d.AVAL, 'ng/mL' as AVALU length=8    from subjects as s left join sd as d      on d.STUDYID = s.STUDYID and d.USUBJID = s.USUBJID    order by 1, 2, 3;quit;proc export data=final outfile="/app/output/adpp.csv" dbms=csv replace; run;