Review ask

As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.

We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.

Summary

Lifecycle: draftEditUnder review
What this benchmark means
README.md

Public entries: Python and R. Passing coverage requires an installed inventory.

Goal: fill each missing vital signs result with the last value observed before it (last observation carried forward), in AVAL.

Input: one planned vital signs assessment per subject, parameter, and analysis visit, holding either a collected value or a gap.

Variables:

  • AVISITN identifies the planned analysis visit.
  • AVAL is the collected value when there is one; otherwise it is the value from the closest earlier visit, in visit-number order, that has one, for the same subject and parameter. A gap before the first collected value stays missing, and zero is a collected value that is carried like any other.

Inputs

Under review
1 source file

input/vs.csv

Edit
18 rows
input/vs.csv
USUBJIDPARAMCDAVISITNAVALCOL
SYSBP1118
SYSBP2—
SYSBP3—
SYSBP4142
DIABP1—
DIABP278
SYSBP1—
SYSBP20
SYSBP3—
SYSBP1132
SYSBP2—
SYSBP3128
SYSBP4—
SYSBP5—
SYSBP1121
DIABP1—
DIABP2—
DIABP3—

Expected output

Under review
18 expected rows

expected/advs.csv

Edit
18 rows
expected/advs.csv
USUBJIDPARAMCDAVISITNAVAL
SYSBP1118
SYSBP2118
SYSBP3118
SYSBP4142
DIABP1—
DIABP278
SYSBP1—
SYSBP20
SYSBP30
SYSBP1132
SYSBP2132
SYSBP3128
SYSBP4128
SYSBP5128
SYSBP1121
DIABP1—
DIABP2—
DIABP3—
Shaded columns are derived— = missing value

Human review and assessment

Your review

3 questions and your comments · View benchmark discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Dataset NameVariable NameVariable LabelTypeCodelist/Controlled TermsCoreComputational MethodOrigin
ADVSUSUBJIDUnique Subject IdentifiercharacterCopy value from VS.USUBJID.Predecessor
ADVSPARAMCDParameter CodecharacterCopy value from VS.PARAMCD.Predecessor
ADVSAVISITNAnalysis Visit NumbernumericCopy value from VS.AVISITN.Predecessor
ADVSAVALAnalysis ValuenumericLast observation carried forward from AVALCOL within each group of USUBJID, PARAMCD, ordered by AVISITN.Derived
VersionDateAuthorDescriptionReviewerSign-off
1.72026-10-20yamaa docs buildRead a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default
1.62026-10-17yamaa docs buildDescribe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight
1.52026-10-17yamaa docs buildName collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part
1.42026-10-15yamaa docs buildState datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/adam-advs-locf/full.md

Edit
13 lines

Following CDISC ADaM standards, use the provided VS dataset to create an ADVS dataset with one record per subject per parameter per analysis visit.

The output dataset should contain the following columns in this order: USUBJID, PARAMCD, AVISITN, AVAL

AVAL is the collected value when the record has one; otherwise it is the value from the closest earlier visit, in visit-number order, that has one, for the same subject and parameter. A gap before the first collected value has no value. A collected value of zero is carried like any other value.

Read the source datasets from /app/input and save the completed dataset as /app/output/advs.csv.

prompts/adam-advs-locf/conventions.md

Edit
8 lines

Following CDISC ADaM standards, use the provided VS dataset to create an ADVS dataset with one record per subject per parameter per analysis visit.

The output dataset should contain the following columns in this order: USUBJID, PARAMCD, AVISITN, AVAL

Read the source datasets from /app/input and save the completed dataset as /app/output/advs.csv.

prompts/adam-advs-locf/brief.md

Edit
8 lines

Following CDISC ADaM standards, use the provided VS dataset to create an ADVS dataset with one record per subject per parameter per analysis visit.

The output dataset should contain the following columns in this order: USUBJID, PARAMCD, AVISITN, AVAL

Read the source datasets from /app/input and save the completed dataset as /app/output/advs.csv.

yamaa code

Public calls generated from benchmark.yaml; results must match the expected output

R

Edit
3 lines
library(yamaa)yamaa_spec("spec.yaml")$run()$output

Python

Edit
3 lines
import yamaayamaa.spec("spec.yaml").run().output

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/adam-advs-locf/result.R

Edit
27 lines
# Reference solution for the yamaa benchmark adam-advs-locf (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)library(tidyr)vs <- read_csv(  "/app/input/vs.csv",  col_types = cols(AVISITN = col_integer(), AVALCOL = col_double(), .default = col_character()))# The collected value, otherwise the closest earlier visit with one, in# visit-number order; a gap before the first stays missing.advs <- vs |>  arrange(USUBJID, PARAMCD, AVISITN) |>  group_by(USUBJID, PARAMCD) |>  fill(AVALCOL, .direction = "down") |>  mutate(AVAL = AVALCOL) |>  ungroup() |>  select(USUBJID, PARAMCD, AVISITN, AVAL)dir.create("/app/output", showWarnings = FALSE)write_csv(advs, "/app/output/advs.csv", na = "")

solutions/adam-advs-locf/result.py

Edit
61 lines
# Reference solution for the yamaa benchmark adam-advs-locf (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1."""Create the ADVS dataset from the VS dataset.One record per subject per parameter per analysis visit. AVAL is thecollected value when the record has one; otherwise it is the value fromthe closest earlier visit, in visit-number order, that has one, for thesame subject and parameter (last observation carried forward). A gapbefore the first collected value has no value. A collected value ofzero is carried like any other value."""import csvINPUT_PATH = "/app/input/vs.csv"OUTPUT_PATH = "/app/output/advs.csv"def main() -> None:    with open(INPUT_PATH, newline="") as f:        rows = list(csv.DictReader(f))    # Walk the records in subject / parameter / visit-number order so    # each record can see the closest earlier collected value of its    # group, then map the results back to the original record order.    order = sorted(        range(len(rows)),        key=lambda i: (            rows[i]["USUBJID"],            rows[i]["PARAMCD"],            int(rows[i]["AVISITN"]),        ),    )    last_value = {}    aval = [""] * len(rows)    for i in order:        row = rows[i]        key = (row["USUBJID"], row["PARAMCD"])        collected = row["AVALCOL"]        if collected != "":            last_value[key] = collected            aval[i] = collected        else:            aval[i] = last_value.get(key, "")    with open(OUTPUT_PATH, "w", newline="") as f:        writer = csv.writer(f)        writer.writerow(["USUBJID", "PARAMCD", "AVISITN", "AVAL"])        for i, row in enumerate(rows):            writer.writerow(                [row["USUBJID"], row["PARAMCD"], row["AVISITN"], aval[i]]            )if __name__ == "__main__":    main()

solutions/adam-advs-locf/result.sas

Edit
20 lines
/* opensas reference solution for the yamaa benchmark adam-advs-locf.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data vs;  length USUBJID PARAMCD $1024;  infile "/app/input/vs.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input USUBJID : $char1024. PARAMCD : $char1024. AVISITN AVALCOL;run;proc sort data=vs out=ordered; by USUBJID PARAMCD AVISITN; run;data result;  set ordered; by USUBJID PARAMCD;  retain carried;  if first.PARAMCD then carried=.;  if not missing(AVALCOL) then carried=AVALCOL;  AVAL=carried;run;proc export data=result(keep=USUBJID PARAMCD AVISITN AVAL) outfile="/app/output/advs.csv" dbms=csv replace; run;