Review ask

As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.

We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.

Summary

Lifecycle: draftEditUnder review
What this benchmark means
README.md

Public entries: Python and R. Passing coverage requires an installed inventory.

Goal: give each pharmacokinetic (PK) concentration its analysis value (AVAL), setting a result below the lower limit of quantification by where it falls in its concentration-time profile.

Input: Study Data Tabulation Model (SDTM) PK concentrations (PC), one record per sample and analyte. A result below the limit is reported as text starting with < (PCSTRESC) with no number (PCSTRESN); a sample not done has PCSTAT "NOT DONE" and no result. Each sample has its planned time point (PCTPT, numbered in hours in PCTPTNUM) and the dose it follows (PCTPTREF).

Variables:

  • PCSEQ, PARAMCD: the sample's sequence number and its analyte (PCTESTCD), as in the input.
  • ATPTREF, ATPT, ATPTN: the dose and the planned time point, as in the input.
  • AVAL: the concentration in ng/mL. A quantified result is carried as recorded. A result below the limit is 0 when it comes before the first quantified result of its profile or after the last one, and has no value between them. In a profile with no quantified result, every result below the limit is 0. A sample not done has no value.
  • AVALU: the unit, ng/mL.

Note: a profile is the samples of one subject and analyte after one dose, in planned time order; a sample not done neither starts nor ends the quantified span. The positions and their values are an authored study convention, not a recovery of the unmeasured concentration.

Inputs

Under review
1 source file

input/pc.csv

Edit
20 rows
input/pc.csv
STUDYIDDOMAINUSUBJIDPCSEQPCTESTCDPCTESTPCORRESPCORRESUPCSTRESCPCSTRESNPCSTRESUPCSTATPCLLOQPCDTCPCTPTPCTPTNUMPCTPTREFPCRFTDTC
YAMAA-01PC1DRUGXDrug X<0.5ng/mL<0.5—ng/mL—0.52025-05-05T07:50PREDOSE0DAY 1 DOSE2025-05-05T08:00
YAMAA-01PC2DRUGXDrug X4.2ng/mL4.24.2ng/mL—0.52025-05-05T09:001H1DAY 1 DOSE2025-05-05T08:00
YAMAA-01PC3DRUGXDrug X9.6ng/mL9.69.6ng/mL—0.52025-05-05T10:002H2DAY 1 DOSE2025-05-05T08:00
YAMAA-01PC4DRUGXDrug X<0.5ng/mL<0.5—ng/mL—0.52025-05-05T12:004H4DAY 1 DOSE2025-05-05T08:00
YAMAA-01PC5DRUGXDrug X3.1ng/mL3.13.1ng/mL—0.52025-05-05T16:008H8DAY 1 DOSE2025-05-05T08:00
YAMAA-01PC6DRUGXDrug X<0.5ng/mL<0.5—ng/mL—0.52025-05-05T20:0012H12DAY 1 DOSE2025-05-05T08:00
YAMAA-01PC7DRUGXDrug X<0.5ng/mL<0.5—ng/mL—0.52025-05-06T08:0024H24DAY 1 DOSE2025-05-05T08:00
YAMAA-01PC8DRUGXDrug X<0.5ng/mL<0.5—ng/mL—0.52025-05-12T07:50PREDOSE0DAY 8 DOSE2025-05-12T08:00
YAMAA-01PC9DRUGXDrug X6.8ng/mL6.86.8ng/mL—0.52025-05-12T09:001H1DAY 8 DOSE2025-05-12T08:00
YAMAA-01PC10DRUGXDrug X<0.5ng/mL<0.5—ng/mL—0.52025-05-12T12:004H4DAY 8 DOSE2025-05-12T08:00
YAMAA-01PC1DRUGXDrug X—ng/mL——ng/mLNOT DONE0.5—PREDOSE0DAY 1 DOSE2025-05-05T08:00
YAMAA-01PC2DRUGXDrug X<0.5ng/mL<0.5—ng/mL—0.52025-05-05T08:300.5H0.5DAY 1 DOSE2025-05-05T08:00
YAMAA-01PC3DRUGXDrug X2.4ng/mL2.42.4ng/mL—0.52025-05-05T09:001H1DAY 1 DOSE2025-05-05T08:00
YAMAA-01PC4DRUGXDrug X1.7ng/mL1.71.7ng/mL—0.52025-05-05T10:002H2DAY 1 DOSE2025-05-05T08:00
YAMAA-01PC5DRUGXDrug X<0.5ng/mL<0.5—ng/mL—0.52025-05-05T12:004H4DAY 1 DOSE2025-05-05T08:00
YAMAA-01PC6DRUGXDrug X—ng/mL——ng/mLNOT DONE0.5—12H12DAY 1 DOSE2025-05-05T08:00
YAMAA-01PC7DRUGXDrug X<0.5ng/mL<0.5—ng/mL—0.52025-05-06T08:0024H24DAY 1 DOSE2025-05-05T08:00
YAMAA-01PC8DRUGXDrug X<0.5ng/mL<0.5—ng/mL—0.52025-05-12T07:50PREDOSE0DAY 8 DOSE2025-05-12T08:00
YAMAA-01PC9DRUGXDrug X<0.5ng/mL<0.5—ng/mL—0.52025-05-12T09:001H1DAY 8 DOSE2025-05-12T08:00
YAMAA-01PC10DRUGXDrug X<0.5ng/mL<0.5—ng/mL—0.52025-05-12T12:004H4DAY 8 DOSE2025-05-12T08:00

Expected output

Under review
20 expected rows

expected/adpc.csv

Edit
20 rows
expected/adpc.csv
STUDYIDUSUBJIDPCSEQPARAMCDATPTREFATPTATPTNAVALAVALU
YAMAA-011DRUGXDAY 1 DOSEPREDOSE00ng/mL
YAMAA-012DRUGXDAY 1 DOSE1H14.2ng/mL
YAMAA-013DRUGXDAY 1 DOSE2H29.6ng/mL
YAMAA-014DRUGXDAY 1 DOSE4H4—ng/mL
YAMAA-015DRUGXDAY 1 DOSE8H83.1ng/mL
YAMAA-016DRUGXDAY 1 DOSE12H120ng/mL
YAMAA-017DRUGXDAY 1 DOSE24H240ng/mL
YAMAA-018DRUGXDAY 8 DOSEPREDOSE00ng/mL
YAMAA-019DRUGXDAY 8 DOSE1H16.8ng/mL
YAMAA-0110DRUGXDAY 8 DOSE4H40ng/mL
YAMAA-011DRUGXDAY 1 DOSEPREDOSE0—ng/mL
YAMAA-012DRUGXDAY 1 DOSE0.5H0.50ng/mL
YAMAA-013DRUGXDAY 1 DOSE1H12.4ng/mL
YAMAA-014DRUGXDAY 1 DOSE2H21.7ng/mL
YAMAA-015DRUGXDAY 1 DOSE4H40ng/mL
YAMAA-016DRUGXDAY 1 DOSE12H12—ng/mL
YAMAA-017DRUGXDAY 1 DOSE24H240ng/mL
YAMAA-018DRUGXDAY 8 DOSEPREDOSE00ng/mL
YAMAA-019DRUGXDAY 8 DOSE1H10ng/mL
YAMAA-0110DRUGXDAY 8 DOSE4H40ng/mL
Shaded columns are derived— = missing value

Human review and assessment

Your review

3 questions and your comments · View benchmark discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Dataset NameVariable NameVariable LabelTypeCodelist/Controlled TermsCoreComputational MethodOrigin
ADPCSTUDYIDStudy IdentifiercharacterCopy value from PC.STUDYID.Predecessor
ADPCUSUBJIDUnique Subject IdentifiercharacterCopy value from PC.USUBJID.Predecessor
ADPCPCSEQSequence NumbernumericCopy value from PC.PCSEQ.Predecessor
ADPCPARAMCDParameter CodecharacterCopy value from PC.PCTESTCD.Predecessor
ADPCATPTREFAnalysis Timepoint ReferencecharacterCopy value from PC.PCTPTREF.Predecessor
ADPCATPTAnalysis TimepointcharacterCopy value from PC.PCTPT.Predecessor
ADPCATPTNAnalysis Timepoint (N)numericCopy value from PC.PCTPTNUM.Predecessor
ADPCAVALAnalysis ValuenumericIf PC.PCSTRESN IS NOT NULL then "PC.PCSTRESN"; If PC.PCSTRESC LIKE '<%' AND (FIRSTQ IS NULL OR ATPTN < FIRSTQ OR ATPTN > LASTQ) then "0.0"; otherwise blank.Derived
ADPCAVALUAnalysis Value UnitcharacterCopy value from PC.PCSTRESU.Predecessor
VersionDateAuthorDescriptionReviewerSign-off
1.72026-10-20yamaa docs buildRead a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default
1.62026-10-17yamaa docs buildDescribe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight
1.52026-10-17yamaa docs buildName collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part
1.42026-10-15yamaa docs buildState datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/adam-adpc-blq-imputation/full.md

Edit
22 lines

Following CDISC ADaM standards, use the provided PC dataset to create an ADPC dataset with one record per sample.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, PCSEQ, PARAMCD, ATPTREF, ATPT, ATPTN, AVAL, AVALU

PCSEQ is the sample's sequence number and PARAMCD its analyte, as in the input. ATPTREF is the dose the sample follows, ATPT its planned time point, and ATPTN that time point numbered in hours, as in the input. A profile is the samples of one subject and analyte after one dose, in planned time order. AVAL is the concentration in ng/mL. For a quantified result, AVAL is the result as recorded. A result below the lower limit of quantification is reported as text starting with "<", with no number. AVAL is 0 for a result below the limit when it comes before the first quantified result of its profile or after the last one, and has no value between them. In a profile with no quantified result, every result below the limit is 0. A sample not done has no result, and AVAL has no value. A sample not done neither starts nor ends the quantified span of its profile. AVALU is "ng/mL".

Read the source datasets from /app/input and save the completed dataset as /app/output/adpc.csv.

prompts/adam-adpc-blq-imputation/conventions.md

Edit
8 lines

Following CDISC ADaM standards, use the provided PC dataset to create an ADPC dataset with one record per sample.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, PCSEQ, PARAMCD, ATPTREF, ATPT, ATPTN, AVAL, AVALU

Read the source datasets from /app/input and save the completed dataset as /app/output/adpc.csv.

prompts/adam-adpc-blq-imputation/brief.md

Edit
8 lines

Following CDISC ADaM standards, use the provided PC dataset to create an ADPC dataset with one record per sample.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, PCSEQ, PARAMCD, ATPTREF, ATPT, ATPTN, AVAL, AVALU

Read the source datasets from /app/input and save the completed dataset as /app/output/adpc.csv.

yamaa code

Public calls generated from benchmark.yaml; results must match the expected output

R

Edit
3 lines
library(yamaa)yamaa_spec("spec.yaml")$run()$output

Python

Edit
3 lines
import yamaayamaa.spec("spec.yaml").run().output

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/adam-adpc-blq-imputation/result.R

Edit
59 lines
# Reference solution for the yamaa benchmark adam-adpc-blq-imputation (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)pc <- read_csv(  "/app/input/pc.csv",  col_types = cols(    PCSEQ = col_integer(),    PCSTRESN = col_double(),    PCTPTNUM = col_double(),    .default = col_character()  ),  na = "")# The first and last quantified time points of each profile. A result# below the limit and a sample not done have no number, so neither moves# the quantified span.span <- pc |>  filter(!is.na(PCSTRESN)) |>  group_by(STUDYID, USUBJID, PCTESTCD, PCTPTREF) |>  summarise(    FIRSTQ = min(PCTPTNUM),    LASTQ = max(PCTPTNUM),    .groups = "drop"  )based <- pc |>  left_join(    span,    by = c("STUDYID", "USUBJID", "PCTESTCD", "PCTPTREF")  )# A quantified result as recorded; a below-limit result is 0 before the# first or after the last quantified result of its profile (and in a# profile with no quantified result), and has no value between them. A# sample not done has no value.adpc <- based |>  mutate(    AVAL = case_when(      !is.na(PCSTRESN) ~ PCSTRESN,      !is.na(PCSTRESC) & startsWith(PCSTRESC, "<") &        (is.na(FIRSTQ) | PCTPTNUM < FIRSTQ | PCTPTNUM > LASTQ) ~ 0,      .default = NA_real_    )  ) |>  select(    STUDYID, USUBJID, PCSEQ,    PARAMCD = PCTESTCD, ATPTREF = PCTPTREF, ATPT = PCTPT,    ATPTN = PCTPTNUM, AVAL, AVALU = PCSTRESU  )dir.create("/app/output", showWarnings = FALSE)write_csv(adpc, "/app/output/adpc.csv", na = "")

solutions/adam-adpc-blq-imputation/result.py

Edit
80 lines
# Reference solution for the yamaa benchmark adam-adpc-blq-imputation (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1."""Build ADPC from PC with BLQ imputation.One record per sample. AVAL is the quantified concentration as recorded;a below-limit result ("<...") is 0 before the first or after the lastquantified result of its profile (subject, analyte, dose), and blankbetween them. A not-done sample has no AVAL and neither starts nor endsthe quantified span."""import csvfrom pathlib import PathINPUT = Path("/app/input")OUTPUT = Path("/app/output")def main() -> None:    with (INPUT / "pc.csv").open(newline="") as f:        rows = list(csv.DictReader(f))    # Classify each sample.    for r in rows:        stat = (r.get("PCSTAT") or "").strip()        stresc = (r.get("PCSTRESC") or "").strip()        if stat == "NOT DONE" or stresc == "":            r["_kind"] = "notdone"        elif stresc.startswith("<"):            r["_kind"] = "blq"        else:            r["_kind"] = "quant"    # Profiles: one subject, one analyte, one dose, in planned time order.    profiles = {}    for r in rows:        key = (r["USUBJID"], r["PCTESTCD"], r["PCTPTREF"])        profiles.setdefault(key, []).append(r)    for members in profiles.values():        members.sort(key=lambda r: (float(r["PCTPTNUM"]), int(r["PCSEQ"])))        quant_pos = [i for i, r in enumerate(members) if r["_kind"] == "quant"]        for i, r in enumerate(members):            if r["_kind"] == "quant":                r["_aval"] = (r.get("PCSTRESN") or "").strip()            elif r["_kind"] == "blq":                if not quant_pos or i < quant_pos[0] or i > quant_pos[-1]:                    r["_aval"] = "0"                else:                    r["_aval"] = ""            else:                r["_aval"] = ""    out_fields = [        "STUDYID", "USUBJID", "PCSEQ", "PARAMCD", "ATPTREF",        "ATPT", "ATPTN", "AVAL", "AVALU",    ]    OUTPUT.mkdir(parents=True, exist_ok=True)    with (OUTPUT / "adpc.csv").open("w", newline="") as f:        w = csv.DictWriter(f, fieldnames=out_fields)        w.writeheader()        for r in rows:            w.writerow({                "STUDYID": r["STUDYID"],                "USUBJID": r["USUBJID"],                "PCSEQ": r["PCSEQ"],                "PARAMCD": r["PCTESTCD"],                "ATPTREF": r["PCTPTREF"],                "ATPT": r["PCTPT"],                "ATPTN": r["PCTPTNUM"],                "AVAL": r["_aval"],                "AVALU": "ng/mL",            })if __name__ == "__main__":    main()

solutions/adam-adpc-blq-imputation/result.sas

Edit
53 lines
/* opensas reference solution for the yamaa benchmark adam-adpc-blq-imputation.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data pc;  length STUDYID DOMAIN USUBJID PCTESTCD PCTEST PCORRES PCORRESU PCSTRESC PCSTRESU PCSTAT PCDTC PCTPT PCTPTREF PCRFTDTC $1024;  infile "/app/input/pc.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. DOMAIN : $char1024. USUBJID : $char1024. PCSEQ PCTESTCD : $char1024. PCTEST : $char1024. PCORRES : $char1024. PCORRESU : $char1024. PCSTRESC : $char1024. PCSTRESN PCSTRESU : $char1024. PCSTAT : $char1024. PCLLOQ PCDTC : $char1024. PCTPT : $char1024. PCTPTNUM PCTPTREF : $char1024. PCRFTDTC : $char1024.;run;/* The quantified samples only: a result below the limit and a sample not   done have no number, so neither moves the quantified span. */data quant;  set pc;  if not missing(PCSTRESN);  keep STUDYID USUBJID PCTESTCD PCTPTREF PCTPTNUM;run;proc sql;  create table span as    select STUDYID, USUBJID, PCTESTCD, PCTPTREF,           min(PCTPTNUM) as FIRSTQ, max(PCTPTNUM) as LASTQ    from quant    group by STUDYID, USUBJID, PCTESTCD, PCTPTREF;quit;proc sql;  create table joined as    select a.*, s.FIRSTQ, s.LASTQ    from pc as a left join span as s      on a.STUDYID = s.STUDYID and a.USUBJID = s.USUBJID      and a.PCTESTCD = s.PCTESTCD and a.PCTPTREF = s.PCTPTREF;quit;/* A quantified result as recorded; a below-limit result is 0 before the   first or after the last quantified result of its profile (and in a   profile with no quantified result), and has no value between them. */data result;  set joined;  PARAMCD = PCTESTCD;  ATPTREF = PCTPTREF;  ATPT = PCTPT;  ATPTN = PCTPTNUM;  AVALU = PCSTRESU;  if not missing(PCSTRESN) then AVAL = PCSTRESN;  else if substr(PCSTRESC, 1, 1) = '<' and (missing(FIRSTQ) or PCTPTNUM < FIRSTQ or PCTPTNUM > LASTQ) then AVAL = 0;  else AVAL = .;run;proc sql;  create table final as select STUDYID, USUBJID, PCSEQ, PARAMCD, ATPTREF, ATPT, ATPTN, AVAL, AVALU from result;quit;proc export data=final outfile="/app/output/adpc.csv" dbms=csv replace; run;