As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.
We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.
Summary
README.mdPublic entries: Python and R. Passing coverage requires an installed inventory.
Goal: give each pharmacokinetic (PK) concentration its analysis value
(AVAL), setting a result below the lower limit of quantification by where
it falls in its concentration-time profile.
Input: Study Data Tabulation Model (SDTM) PK concentrations (PC), one
record per sample and analyte. A result below the limit is reported as text
starting with < (PCSTRESC) with no number (PCSTRESN); a sample not
done has PCSTAT "NOT DONE" and no result. Each sample has its planned
time point (PCTPT, numbered in hours in PCTPTNUM) and the dose it
follows (PCTPTREF).
Variables:
PCSEQ,PARAMCD: the sample's sequence number and its analyte (PCTESTCD), as in the input.ATPTREF,ATPT,ATPTN: the dose and the planned time point, as in the input.AVAL: the concentration in ng/mL. A quantified result is carried as recorded. A result below the limit is 0 when it comes before the first quantified result of its profile or after the last one, and has no value between them. In a profile with no quantified result, every result below the limit is 0. A sample not done has no value.AVALU: the unit, ng/mL.
Note: a profile is the samples of one subject and analyte after one dose, in planned time order; a sample not done neither starts nor ends the quantified span. The positions and their values are an authored study convention, not a recovery of the unmeasured concentration.
Inputs
Under review1 source fileinput/pc.csv
Edit20 rows| STUDYID | DOMAIN | USUBJID | PCSEQ | PCTESTCD | PCTEST | PCORRES | PCORRESU | PCSTRESC | PCSTRESN | PCSTRESU | PCSTAT | PCLLOQ | PCDTC | PCTPT | PCTPTNUM | PCTPTREF | PCRFTDTC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| YAMAA-01 | PC | 1 | DRUGX | Drug X | <0.5 | ng/mL | <0.5 | — | ng/mL | — | 0.5 | 2025-05-05T07:50 | PREDOSE | 0 | DAY 1 DOSE | 2025-05-05T08:00 | |
| YAMAA-01 | PC | 2 | DRUGX | Drug X | 4.2 | ng/mL | 4.2 | 4.2 | ng/mL | — | 0.5 | 2025-05-05T09:00 | 1H | 1 | DAY 1 DOSE | 2025-05-05T08:00 | |
| YAMAA-01 | PC | 3 | DRUGX | Drug X | 9.6 | ng/mL | 9.6 | 9.6 | ng/mL | — | 0.5 | 2025-05-05T10:00 | 2H | 2 | DAY 1 DOSE | 2025-05-05T08:00 | |
| YAMAA-01 | PC | 4 | DRUGX | Drug X | <0.5 | ng/mL | <0.5 | — | ng/mL | — | 0.5 | 2025-05-05T12:00 | 4H | 4 | DAY 1 DOSE | 2025-05-05T08:00 | |
| YAMAA-01 | PC | 5 | DRUGX | Drug X | 3.1 | ng/mL | 3.1 | 3.1 | ng/mL | — | 0.5 | 2025-05-05T16:00 | 8H | 8 | DAY 1 DOSE | 2025-05-05T08:00 | |
| YAMAA-01 | PC | 6 | DRUGX | Drug X | <0.5 | ng/mL | <0.5 | — | ng/mL | — | 0.5 | 2025-05-05T20:00 | 12H | 12 | DAY 1 DOSE | 2025-05-05T08:00 | |
| YAMAA-01 | PC | 7 | DRUGX | Drug X | <0.5 | ng/mL | <0.5 | — | ng/mL | — | 0.5 | 2025-05-06T08:00 | 24H | 24 | DAY 1 DOSE | 2025-05-05T08:00 | |
| YAMAA-01 | PC | 8 | DRUGX | Drug X | <0.5 | ng/mL | <0.5 | — | ng/mL | — | 0.5 | 2025-05-12T07:50 | PREDOSE | 0 | DAY 8 DOSE | 2025-05-12T08:00 | |
| YAMAA-01 | PC | 9 | DRUGX | Drug X | 6.8 | ng/mL | 6.8 | 6.8 | ng/mL | — | 0.5 | 2025-05-12T09:00 | 1H | 1 | DAY 8 DOSE | 2025-05-12T08:00 | |
| YAMAA-01 | PC | 10 | DRUGX | Drug X | <0.5 | ng/mL | <0.5 | — | ng/mL | — | 0.5 | 2025-05-12T12:00 | 4H | 4 | DAY 8 DOSE | 2025-05-12T08:00 | |
| YAMAA-01 | PC | 1 | DRUGX | Drug X | — | ng/mL | — | — | ng/mL | NOT DONE | 0.5 | — | PREDOSE | 0 | DAY 1 DOSE | 2025-05-05T08:00 | |
| YAMAA-01 | PC | 2 | DRUGX | Drug X | <0.5 | ng/mL | <0.5 | — | ng/mL | — | 0.5 | 2025-05-05T08:30 | 0.5H | 0.5 | DAY 1 DOSE | 2025-05-05T08:00 | |
| YAMAA-01 | PC | 3 | DRUGX | Drug X | 2.4 | ng/mL | 2.4 | 2.4 | ng/mL | — | 0.5 | 2025-05-05T09:00 | 1H | 1 | DAY 1 DOSE | 2025-05-05T08:00 | |
| YAMAA-01 | PC | 4 | DRUGX | Drug X | 1.7 | ng/mL | 1.7 | 1.7 | ng/mL | — | 0.5 | 2025-05-05T10:00 | 2H | 2 | DAY 1 DOSE | 2025-05-05T08:00 | |
| YAMAA-01 | PC | 5 | DRUGX | Drug X | <0.5 | ng/mL | <0.5 | — | ng/mL | — | 0.5 | 2025-05-05T12:00 | 4H | 4 | DAY 1 DOSE | 2025-05-05T08:00 | |
| YAMAA-01 | PC | 6 | DRUGX | Drug X | — | ng/mL | — | — | ng/mL | NOT DONE | 0.5 | — | 12H | 12 | DAY 1 DOSE | 2025-05-05T08:00 | |
| YAMAA-01 | PC | 7 | DRUGX | Drug X | <0.5 | ng/mL | <0.5 | — | ng/mL | — | 0.5 | 2025-05-06T08:00 | 24H | 24 | DAY 1 DOSE | 2025-05-05T08:00 | |
| YAMAA-01 | PC | 8 | DRUGX | Drug X | <0.5 | ng/mL | <0.5 | — | ng/mL | — | 0.5 | 2025-05-12T07:50 | PREDOSE | 0 | DAY 8 DOSE | 2025-05-12T08:00 | |
| YAMAA-01 | PC | 9 | DRUGX | Drug X | <0.5 | ng/mL | <0.5 | — | ng/mL | — | 0.5 | 2025-05-12T09:00 | 1H | 1 | DAY 8 DOSE | 2025-05-12T08:00 | |
| YAMAA-01 | PC | 10 | DRUGX | Drug X | <0.5 | ng/mL | <0.5 | — | ng/mL | — | 0.5 | 2025-05-12T12:00 | 4H | 4 | DAY 8 DOSE | 2025-05-12T08:00 |
Expected output
Under review20 expected rowsexpected/adpc.csv
Edit20 rows| STUDYID | USUBJID | PCSEQ | PARAMCD | ATPTREF | ATPT | ATPTN | AVAL | AVALU |
|---|---|---|---|---|---|---|---|---|
| YAMAA-01 | 1 | DRUGX | DAY 1 DOSE | PREDOSE | 0 | 0 | ng/mL | |
| YAMAA-01 | 2 | DRUGX | DAY 1 DOSE | 1H | 1 | 4.2 | ng/mL | |
| YAMAA-01 | 3 | DRUGX | DAY 1 DOSE | 2H | 2 | 9.6 | ng/mL | |
| YAMAA-01 | 4 | DRUGX | DAY 1 DOSE | 4H | 4 | — | ng/mL | |
| YAMAA-01 | 5 | DRUGX | DAY 1 DOSE | 8H | 8 | 3.1 | ng/mL | |
| YAMAA-01 | 6 | DRUGX | DAY 1 DOSE | 12H | 12 | 0 | ng/mL | |
| YAMAA-01 | 7 | DRUGX | DAY 1 DOSE | 24H | 24 | 0 | ng/mL | |
| YAMAA-01 | 8 | DRUGX | DAY 8 DOSE | PREDOSE | 0 | 0 | ng/mL | |
| YAMAA-01 | 9 | DRUGX | DAY 8 DOSE | 1H | 1 | 6.8 | ng/mL | |
| YAMAA-01 | 10 | DRUGX | DAY 8 DOSE | 4H | 4 | 0 | ng/mL | |
| YAMAA-01 | 1 | DRUGX | DAY 1 DOSE | PREDOSE | 0 | — | ng/mL | |
| YAMAA-01 | 2 | DRUGX | DAY 1 DOSE | 0.5H | 0.5 | 0 | ng/mL | |
| YAMAA-01 | 3 | DRUGX | DAY 1 DOSE | 1H | 1 | 2.4 | ng/mL | |
| YAMAA-01 | 4 | DRUGX | DAY 1 DOSE | 2H | 2 | 1.7 | ng/mL | |
| YAMAA-01 | 5 | DRUGX | DAY 1 DOSE | 4H | 4 | 0 | ng/mL | |
| YAMAA-01 | 6 | DRUGX | DAY 1 DOSE | 12H | 12 | — | ng/mL | |
| YAMAA-01 | 7 | DRUGX | DAY 1 DOSE | 24H | 24 | 0 | ng/mL | |
| YAMAA-01 | 8 | DRUGX | DAY 8 DOSE | PREDOSE | 0 | 0 | ng/mL | |
| YAMAA-01 | 9 | DRUGX | DAY 8 DOSE | 1H | 1 | 0 | ng/mL | |
| YAMAA-01 | 10 | DRUGX | DAY 8 DOSE | 4H | 4 | 0 | ng/mL |
Human review and assessment
| Dataset Name | Variable Name | Variable Label | Type | Codelist/Controlled Terms | Core | Computational Method | Origin |
|---|---|---|---|---|---|---|---|
| ADPC | STUDYID | Study Identifier | character | Copy value from PC.STUDYID. | Predecessor | ||
| ADPC | USUBJID | Unique Subject Identifier | character | Copy value from PC.USUBJID. | Predecessor | ||
| ADPC | PCSEQ | Sequence Number | numeric | Copy value from PC.PCSEQ. | Predecessor | ||
| ADPC | PARAMCD | Parameter Code | character | Copy value from PC.PCTESTCD. | Predecessor | ||
| ADPC | ATPTREF | Analysis Timepoint Reference | character | Copy value from PC.PCTPTREF. | Predecessor | ||
| ADPC | ATPT | Analysis Timepoint | character | Copy value from PC.PCTPT. | Predecessor | ||
| ADPC | ATPTN | Analysis Timepoint (N) | numeric | Copy value from PC.PCTPTNUM. | Predecessor | ||
| ADPC | AVAL | Analysis Value | numeric | If PC.PCSTRESN IS NOT NULL then "PC.PCSTRESN"; If PC.PCSTRESC LIKE '<%' AND (FIRSTQ IS NULL OR ATPTN < FIRSTQ OR ATPTN > LASTQ) then "0.0"; otherwise blank. | Derived | ||
| ADPC | AVALU | Analysis Value Unit | character | Copy value from PC.PCSTRESU. | Predecessor |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.7 | 2026-10-20 | yamaa docs build | Read a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default | ||
| 1.6 | 2026-10-17 | yamaa docs build | Describe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight | ||
| 1.5 | 2026-10-17 | yamaa docs build | Name collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part | ||
| 1.4 | 2026-10-15 | yamaa docs build | State datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks | ||
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/adam-adpc-blq-imputation/full.md
Edit22 linesFollowing CDISC ADaM standards, use the provided PC dataset to create an ADPC dataset with one record per sample.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, PCSEQ, PARAMCD, ATPTREF, ATPT, ATPTN, AVAL, AVALU
PCSEQ is the sample's sequence number and PARAMCD its analyte, as in the input. ATPTREF is the dose the sample follows, ATPT its planned time point, and ATPTN that time point numbered in hours, as in the input. A profile is the samples of one subject and analyte after one dose, in planned time order. AVAL is the concentration in ng/mL. For a quantified result, AVAL is the result as recorded. A result below the lower limit of quantification is reported as text starting with "<", with no number. AVAL is 0 for a result below the limit when it comes before the first quantified result of its profile or after the last one, and has no value between them. In a profile with no quantified result, every result below the limit is 0. A sample not done has no result, and AVAL has no value. A sample not done neither starts nor ends the quantified span of its profile. AVALU is "ng/mL".
Read the source datasets from /app/input and save the completed dataset as /app/output/adpc.csv.
prompts/adam-adpc-blq-imputation/conventions.md
Edit8 linesFollowing CDISC ADaM standards, use the provided PC dataset to create an ADPC dataset with one record per sample.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, PCSEQ, PARAMCD, ATPTREF, ATPT, ATPTN, AVAL, AVALU
Read the source datasets from /app/input and save the completed dataset as /app/output/adpc.csv.
prompts/adam-adpc-blq-imputation/brief.md
Edit8 linesFollowing CDISC ADaM standards, use the provided PC dataset to create an ADPC dataset with one record per sample.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, PCSEQ, PARAMCD, ATPTREF, ATPT, ATPTN, AVAL, AVALU
Read the source datasets from /app/input and save the completed dataset as /app/output/adpc.csv.
R
Edit3 lineslibrary(yamaa)yamaa_spec("spec.yaml")$run()$outputPython
Edit3 linesimport yamaayamaa.spec("spec.yaml").run().outputsolutions/adam-adpc-blq-imputation/result.R
Edit59 lines# Reference solution for the yamaa benchmark adam-adpc-blq-imputation (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)pc <- read_csv( "/app/input/pc.csv", col_types = cols( PCSEQ = col_integer(), PCSTRESN = col_double(), PCTPTNUM = col_double(), .default = col_character() ), na = "")# The first and last quantified time points of each profile. A result# below the limit and a sample not done have no number, so neither moves# the quantified span.span <- pc |> filter(!is.na(PCSTRESN)) |> group_by(STUDYID, USUBJID, PCTESTCD, PCTPTREF) |> summarise( FIRSTQ = min(PCTPTNUM), LASTQ = max(PCTPTNUM), .groups = "drop" )based <- pc |> left_join( span, by = c("STUDYID", "USUBJID", "PCTESTCD", "PCTPTREF") )# A quantified result as recorded; a below-limit result is 0 before the# first or after the last quantified result of its profile (and in a# profile with no quantified result), and has no value between them. A# sample not done has no value.adpc <- based |> mutate( AVAL = case_when( !is.na(PCSTRESN) ~ PCSTRESN, !is.na(PCSTRESC) & startsWith(PCSTRESC, "<") & (is.na(FIRSTQ) | PCTPTNUM < FIRSTQ | PCTPTNUM > LASTQ) ~ 0, .default = NA_real_ ) ) |> select( STUDYID, USUBJID, PCSEQ, PARAMCD = PCTESTCD, ATPTREF = PCTPTREF, ATPT = PCTPT, ATPTN = PCTPTNUM, AVAL, AVALU = PCSTRESU )dir.create("/app/output", showWarnings = FALSE)write_csv(adpc, "/app/output/adpc.csv", na = "")solutions/adam-adpc-blq-imputation/result.py
Edit80 lines# Reference solution for the yamaa benchmark adam-adpc-blq-imputation (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1."""Build ADPC from PC with BLQ imputation.One record per sample. AVAL is the quantified concentration as recorded;a below-limit result ("<...") is 0 before the first or after the lastquantified result of its profile (subject, analyte, dose), and blankbetween them. A not-done sample has no AVAL and neither starts nor endsthe quantified span."""import csvfrom pathlib import PathINPUT = Path("/app/input")OUTPUT = Path("/app/output")def main() -> None: with (INPUT / "pc.csv").open(newline="") as f: rows = list(csv.DictReader(f)) # Classify each sample. for r in rows: stat = (r.get("PCSTAT") or "").strip() stresc = (r.get("PCSTRESC") or "").strip() if stat == "NOT DONE" or stresc == "": r["_kind"] = "notdone" elif stresc.startswith("<"): r["_kind"] = "blq" else: r["_kind"] = "quant" # Profiles: one subject, one analyte, one dose, in planned time order. profiles = {} for r in rows: key = (r["USUBJID"], r["PCTESTCD"], r["PCTPTREF"]) profiles.setdefault(key, []).append(r) for members in profiles.values(): members.sort(key=lambda r: (float(r["PCTPTNUM"]), int(r["PCSEQ"]))) quant_pos = [i for i, r in enumerate(members) if r["_kind"] == "quant"] for i, r in enumerate(members): if r["_kind"] == "quant": r["_aval"] = (r.get("PCSTRESN") or "").strip() elif r["_kind"] == "blq": if not quant_pos or i < quant_pos[0] or i > quant_pos[-1]: r["_aval"] = "0" else: r["_aval"] = "" else: r["_aval"] = "" out_fields = [ "STUDYID", "USUBJID", "PCSEQ", "PARAMCD", "ATPTREF", "ATPT", "ATPTN", "AVAL", "AVALU", ] OUTPUT.mkdir(parents=True, exist_ok=True) with (OUTPUT / "adpc.csv").open("w", newline="") as f: w = csv.DictWriter(f, fieldnames=out_fields) w.writeheader() for r in rows: w.writerow({ "STUDYID": r["STUDYID"], "USUBJID": r["USUBJID"], "PCSEQ": r["PCSEQ"], "PARAMCD": r["PCTESTCD"], "ATPTREF": r["PCTPTREF"], "ATPT": r["PCTPT"], "ATPTN": r["PCTPTNUM"], "AVAL": r["_aval"], "AVALU": "ng/mL", })if __name__ == "__main__": main()solutions/adam-adpc-blq-imputation/result.sas
Edit53 lines/* opensas reference solution for the yamaa benchmark adam-adpc-blq-imputation. Reads the benchmark inputs and derives the requested output. Execute with the existing opensas runtime: opensas /app/output/result.sas. */data pc; length STUDYID DOMAIN USUBJID PCTESTCD PCTEST PCORRES PCORRESU PCSTRESC PCSTRESU PCSTAT PCDTC PCTPT PCTPTREF PCRFTDTC $1024; infile "/app/input/pc.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. DOMAIN : $char1024. USUBJID : $char1024. PCSEQ PCTESTCD : $char1024. PCTEST : $char1024. PCORRES : $char1024. PCORRESU : $char1024. PCSTRESC : $char1024. PCSTRESN PCSTRESU : $char1024. PCSTAT : $char1024. PCLLOQ PCDTC : $char1024. PCTPT : $char1024. PCTPTNUM PCTPTREF : $char1024. PCRFTDTC : $char1024.;run;/* The quantified samples only: a result below the limit and a sample not done have no number, so neither moves the quantified span. */data quant; set pc; if not missing(PCSTRESN); keep STUDYID USUBJID PCTESTCD PCTPTREF PCTPTNUM;run;proc sql; create table span as select STUDYID, USUBJID, PCTESTCD, PCTPTREF, min(PCTPTNUM) as FIRSTQ, max(PCTPTNUM) as LASTQ from quant group by STUDYID, USUBJID, PCTESTCD, PCTPTREF;quit;proc sql; create table joined as select a.*, s.FIRSTQ, s.LASTQ from pc as a left join span as s on a.STUDYID = s.STUDYID and a.USUBJID = s.USUBJID and a.PCTESTCD = s.PCTESTCD and a.PCTPTREF = s.PCTPTREF;quit;/* A quantified result as recorded; a below-limit result is 0 before the first or after the last quantified result of its profile (and in a profile with no quantified result), and has no value between them. */data result; set joined; PARAMCD = PCTESTCD; ATPTREF = PCTPTREF; ATPT = PCTPT; ATPTN = PCTPTNUM; AVALU = PCSTRESU; if not missing(PCSTRESN) then AVAL = PCSTRESN; else if substr(PCSTRESC, 1, 1) = '<' and (missing(FIRSTQ) or PCTPTNUM < FIRSTQ or PCTPTNUM > LASTQ) then AVAL = 0; else AVAL = .;run;proc sql; create table final as select STUDYID, USUBJID, PCSEQ, PARAMCD, ATPTREF, ATPT, ATPTN, AVAL, AVALU from result;quit;proc export data=final outfile="/app/output/adpc.csv" dbms=csv replace; run;
yamaa