Review ask

As a clinical trial statistician or programmer, challenge anything in this benchmark: describe a problem in Comments, or propose a fix with the Edit button on any source file.

  1. Task: Would a real study need this derivation, specified this way? Does it follow CDISC ADaM: variable names, controlled terms, record structure? Flag any rule that is ambiguous or that a sponsor would handle differently.
  2. Inputs: Do they look like data collected in a real trial, and do they include the situations you would expect to meet in one?
  3. Expected output: Would independent QC programming reproduce every value from these inputs and rules?

The other tabs beside Comments are background for assessment, not under review.

Summary

Lifecycle: draftEditUnder review
What this benchmark means
README.md

Goal: give every subject in the efficacy population a total score record at each scheduled visit, adding a last observation carried forward record for each visit missed, with the analysis value (AVAL), the record-type flag (DTYPE), and the efficacy population flag (EFFFL).

Input: one subject-level record per subject carrying the efficacy population flag, and the collected ADAS-Cog (Alzheimer's Disease Assessment Scale, cognitive subscale) total scores, each already assigned to an analysis visit (AVISITN).

Variables:

  • AVAL: the collected score on a collected record. On a carried-forward record, the score from the closest earlier visit that has one for the same subject; empty when the subject has no earlier score.
  • DTYPE: empty on a collected record; LOCF on a record added for a missed scheduled visit.
  • EFFFL: the subject's efficacy population flag, repeated on every record.

Note: the protocol schedules Weeks 8, 16, and 24 for every subject in the efficacy population. A scheduled visit is missed when the subject has no collected score there, and a collected record with an empty score is left out, so at a scheduled visit it counts as missed. Subjects outside the efficacy population keep their collected records and gain none, and baseline is not a scheduled visit, so a missing baseline adds nothing. The input carries at most one score per subject per visit.

Inputs

Under review
2 source files

input/adsl.csv

Edit
6 rows
input/adsl.csv
STUDYIDUSUBJIDEFFFL
YAMAA-01Y
YAMAA-01Y
YAMAA-01Y
YAMAA-01Y
YAMAA-01N
YAMAA-01Y

input/qs.csv

Edit
14 rows
input/qs.csv
STUDYIDUSUBJIDQSSEQPARAMCDAVISITAVISITNAVAL
YAMAA-011ACTOTBaseline022
YAMAA-012ACTOTWeek 8824
YAMAA-013ACTOTWeek 161625
YAMAA-014ACTOTWeek 242427
YAMAA-011ACTOTBaseline030
YAMAA-012ACTOTWeek 8831
YAMAA-011ACTOTBaseline018
YAMAA-012ACTOTWeek 88—
YAMAA-013ACTOTWeek 161621
YAMAA-011ACTOTWeek 161615
YAMAA-012ACTOTWeek 242417
YAMAA-011ACTOTBaseline026
YAMAA-012ACTOTWeek 8828
YAMAA-011ACTOTBaseline00

Expected output

Under review
21 expected rows

expected/adqs.csv

Edit
21 rows
expected/adqs.csv
STUDYIDUSUBJIDPARAMCDAVISITAVISITNAVALDTYPEEFFFL
YAMAA-01ACTOTBaseline022—Y
YAMAA-01ACTOTWeek 8824—Y
YAMAA-01ACTOTWeek 161625—Y
YAMAA-01ACTOTWeek 242427—Y
YAMAA-01ACTOTBaseline030—Y
YAMAA-01ACTOTWeek 8831—Y
YAMAA-01ACTOTWeek 161631LOCFY
YAMAA-01ACTOTWeek 242431LOCFY
YAMAA-01ACTOTBaseline018—Y
YAMAA-01ACTOTWeek 8818LOCFY
YAMAA-01ACTOTWeek 161621—Y
YAMAA-01ACTOTWeek 242421LOCFY
YAMAA-01ACTOTWeek 88—LOCFY
YAMAA-01ACTOTWeek 161615—Y
YAMAA-01ACTOTWeek 242417—Y
YAMAA-01ACTOTBaseline026—N
YAMAA-01ACTOTWeek 8828—N
YAMAA-01ACTOTBaseline00—Y
YAMAA-01ACTOTWeek 880LOCFY
YAMAA-01ACTOTWeek 16160LOCFY
YAMAA-01ACTOTWeek 24240LOCFY
Shaded columns are derived— = missing value

Comments and assessment materials

Comments

Signed in with GitHub; saved in repository Discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Dataset NameVariable NameVariable LabelTypeCodelist/Controlled TermsCoreComputational MethodOrigin
ADQSSTUDYIDStudy IdentifiercharacterSet per constructed row; see the Row construction sheet.Derived
ADQSUSUBJIDUnique Subject IdentifiercharacterSet per constructed row; see the Row construction sheet.Derived
ADQSPARAMCDParameter CodecharacterSet per constructed row; see the Row construction sheet.Derived
ADQSAVISITAnalysis VisitcharacterSet per constructed row; see the Row construction sheet.Derived
ADQSAVISITNAnalysis Visit (N)numericSet per constructed row; see the Row construction sheet.Derived
ADQSDTYPEDerivation TypecharacterSet per constructed row; see the Row construction sheet.Derived
ADQSEFFFLEfficacy Population FlagcharacterSet per constructed row; see the Row construction sheet.Derived
ADQSAVALAnalysis ValuenumericLast observation carried forward from AVALCOL within each group of STUDYID, USUBJID, PARAMCD, ordered by AVISITN.Derived
TemplateInclude whenSTUDYIDUSUBJIDPARAMCDAVISITAVISITNAVALCOLDTYPEEFFFL
collectedQS.AVAL IS NOT NULLCopy QS.STUDYIDCopy QS.USUBJIDCopy QS.PARAMCDCopy QS.AVISITCopy QS.AVISITNCopy QS.AVALConstant "None"Copy ADSL.EFFFL
week_8ADSL.EFFFL = 'Y' AND COLLECTED.AVAL IS NULLCopy ADSL.STUDYIDCopy ADSL.USUBJIDConstant "ACTOT"Constant "Week 8"Constant "8"Constant "None"Constant "LOCF"Copy ADSL.EFFFL
week_16ADSL.EFFFL = 'Y' AND COLLECTED.AVAL IS NULLCopy ADSL.STUDYIDCopy ADSL.USUBJIDConstant "ACTOT"Constant "Week 16"Constant "16"Constant "None"Constant "LOCF"Copy ADSL.EFFFL
week_24ADSL.EFFFL = 'Y' AND COLLECTED.AVAL IS NULLCopy ADSL.STUDYIDCopy ADSL.USUBJIDConstant "ACTOT"Constant "Week 24"Constant "24"Constant "None"Constant "LOCF"Copy ADSL.EFFFL
VersionDateAuthorDescriptionReviewerSign-off
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/adam-adqs-missed-visit-locf/full.md

Edit
28 lines

Following CDISC ADaM standards, use the provided ADSL and QS datasets to create an ADQS dataset with one record per subject per parameter per analysis visit.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, AVISIT, AVISITN, AVAL, DTYPE, EFFFL

Use PARAMCD "ACTOT". The protocol schedules Weeks 8, 16, and 24 (AVISIT "Week 8", "Week 16", and "Week 24", with AVISITN 8, 16, and 24) for every subject in the efficacy population; baseline is not a scheduled visit, so a missing baseline adds nothing.

Keep every collected score with DTYPE empty. A collected record with an empty score is left out, so at a scheduled visit it counts as missed.

Add one record with DTYPE "LOCF" for each missed scheduled visit of a subject in the efficacy population. Its value is the score from the closest earlier visit that has one for the same subject, and a collected zero is a real score that carries forward; it has no value when the subject has no earlier score. Subjects outside the efficacy population keep their collected records and gain none.

EFFFL is the subject's efficacy population flag, repeated on every record.

Read the source datasets from /app/input and save the completed dataset as /app/output/adqs.csv.

prompts/adam-adqs-missed-visit-locf/conventions.md

Edit
14 lines

Following CDISC ADaM standards, use the provided ADSL and QS datasets to create an ADQS dataset with one record per subject per parameter per analysis visit.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, AVISIT, AVISITN, AVAL, DTYPE, EFFFL

Use PARAMCD "ACTOT". AVISIT "Week 8", "Week 16", and "Week 24" have AVISITN 8, 16, and 24.

DTYPE is "LOCF" or empty.

Read the source datasets from /app/input and save the completed dataset as /app/output/adqs.csv.

prompts/adam-adqs-missed-visit-locf/brief.md

Edit
9 lines

Following CDISC ADaM standards, use the provided ADSL and QS datasets to create an ADQS dataset with one record per subject per parameter per analysis visit.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, AVISIT, AVISITN, AVAL, DTYPE, EFFFL

Read the source datasets from /app/input and save the completed dataset as /app/output/adqs.csv.

yamaa code

Runs the yamaa specification; its result must match the expected output

run.R

Edit
4 lines
library(yamaa)adqs <- yamaa_domain("spec.yaml")$outputadqs

run.py

Edit
4 lines
import yamaaadqs = yamaa.yamaa_domain("spec.yaml").outputadqs

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/adam-adqs-missed-visit-locf/result.R

Edit
70 lines
# Reference solution for the yamaa benchmark adam-adqs-missed-visit-locf (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)library(tidyr)adsl <- read_csv(  "/app/input/adsl.csv",  col_types = cols(.default = col_character()))qs <- read_csv(  "/app/input/qs.csv",  col_types = cols(    AVISITN = col_integer(),    AVAL = col_double(),    .default = col_character()  ))# Collected scores; an empty score is left out, so it counts as missed.collected <- qs |>  filter(PARAMCD %in% "ACTOT", !is.na(AVAL)) |>  left_join(select(adsl, USUBJID, EFFFL), by = "USUBJID") |>  mutate(DTYPE = NA_character_) |>  select(STUDYID, USUBJID, PARAMCD, AVISIT, AVISITN, AVAL, DTYPE, EFFFL)# One LOCF record for each missed scheduled visit in the efficacy# population; its value is the closest earlier visit with a score, where# a collected zero carries forward and no earlier score leaves no value.scheduled <- tibble(  AVISIT = c("Week 8", "Week 16", "Week 24"),  AVISITN = c(8L, 16L, 24L))eff_subjects <- adsl |>  filter(EFFFL %in% "Y") |>  select(STUDYID, USUBJID, EFFFL)missed_keys <- eff_subjects |>  crossing(scheduled) |>  anti_join(    select(collected, USUBJID, AVISITN),    by = c("USUBJID", "AVISITN")  )carried <- missed_keys |>  left_join(    collected |> select(USUBJID, EARLIER_VISITN = AVISITN, EARLIER_AVAL = AVAL),    by = "USUBJID",    relationship = "many-to-many"  ) |>  filter(!is.na(EARLIER_AVAL), EARLIER_VISITN < AVISITN) |>  group_by(STUDYID, USUBJID, AVISIT, AVISITN, EFFFL) |>  slice_max(EARLIER_VISITN, n = 1, with_ties = FALSE) |>  ungroup() |>  transmute(USUBJID, AVISITN, CARRIED = EARLIER_AVAL)missed <- missed_keys |>  left_join(carried, by = c("USUBJID", "AVISITN")) |>  mutate(PARAMCD = "ACTOT", AVAL = CARRIED, DTYPE = "LOCF") |>  select(STUDYID, USUBJID, PARAMCD, AVISIT, AVISITN, AVAL, DTYPE, EFFFL)adqs <- bind_rows(collected, missed) |>  arrange(USUBJID, AVISITN)dir.create("/app/output", showWarnings = FALSE)write_csv(adqs, "/app/output/adqs.csv", na = "")

solutions/adam-adqs-missed-visit-locf/result.py

Edit
77 lines
# Reference solution for the yamaa benchmark adam-adqs-missed-visit-locf (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.from pathlib import Pathimport polars as pladsl = pl.read_csv("/app/input/adsl.csv", infer_schema=False)qs = pl.read_csv("/app/input/qs.csv", infer_schema=False).with_columns(    pl.col("AVISITN").cast(pl.Int64),    pl.col("AVAL").cast(pl.Float64),)# Collected scores; an empty score is left out, so it counts as missed.collected = (    qs.filter(pl.col("PARAMCD") == "ACTOT", pl.col("AVAL").is_not_null())    .join(adsl.select("USUBJID", "EFFFL"), on="USUBJID", how="left")    .with_columns(DTYPE=pl.lit(None, dtype=pl.String))    .select(        "STUDYID", "USUBJID", "PARAMCD", "AVISIT", "AVISITN", "AVAL", "DTYPE", "EFFFL"    ))# The protocol schedules Weeks 8, 16, and 24 for the efficacy population.scheduled = pl.DataFrame(    {"AVISIT": ["Week 8", "Week 16", "Week 24"], "AVISITN": [8, 16, 24]})eff_subjects = adsl.filter(pl.col("EFFFL") == "Y").select(    "STUDYID", "USUBJID", "EFFFL")planned = eff_subjects.join(scheduled, how="cross")missed_keys = planned.join(    collected.select("USUBJID", "AVISITN"), on=["USUBJID", "AVISITN"], how="anti")# Each missed visit carries the closest earlier visit with a score, where# a collected zero carries forward and no earlier score leaves no value.earlier_by_subject: dict[str, list[tuple[int, float]]] = {}for row in collected.sort(["USUBJID", "AVISITN"]).iter_rows(named=True):    earlier_by_subject.setdefault(row["USUBJID"], []).append(        (row["AVISITN"], row["AVAL"])    )missed_rows = []for row in missed_keys.sort(["USUBJID", "AVISITN"]).iter_rows(named=True):    carried = None    for visitn, aval in earlier_by_subject.get(row["USUBJID"], []):        if visitn < row["AVISITN"]:            carried = aval        else:            break    missed_rows.append({**row, "PARAMCD": "ACTOT", "AVAL": carried, "DTYPE": "LOCF"})missed = pl.DataFrame(    missed_rows,    schema={        "STUDYID": pl.String,        "USUBJID": pl.String,        "EFFFL": pl.String,        "AVISIT": pl.String,        "AVISITN": pl.Int64,        "PARAMCD": pl.String,        "AVAL": pl.Float64,        "DTYPE": pl.String,    },    strict=False,).select(    "STUDYID", "USUBJID", "PARAMCD", "AVISIT", "AVISITN", "AVAL", "DTYPE", "EFFFL")adqs = pl.concat([collected, missed], how="diagonal").sort(["USUBJID", "AVISITN"])Path("/app/output").mkdir(exist_ok=True)adqs.write_csv("/app/output/adqs.csv")

solutions/adam-adqs-missed-visit-locf/result.sas

Edit
32 lines
/* opensas reference solution for the yamaa benchmark adam-adqs-missed-visit-locf.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data adsl;  length STUDYID USUBJID EFFFL $1024;  infile "/app/input/adsl.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. EFFFL : $char1024.;run;data qs;  length STUDYID USUBJID PARAMCD AVISIT $1024;  infile "/app/input/qs.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. QSSEQ PARAMCD : $char1024. AVISIT : $char1024. AVISITN AVAL;run;proc sql;  create table collected as select a.STUDYID,a.USUBJID,a.PARAMCD,a.AVISIT,a.AVISITN,a.AVAL,'' as DTYPE,b.EFFFL from qs as a left join adsl as b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID where not missing(a.AVAL);quit;data schedule; length AVISIT PARAMCD $1024; set adsl; if EFFFL='Y'; PARAMCD='ACTOT'; do AVISITN=8,16,24; AVISIT=catx(' ','Week',strip(put(AVISITN,best32.))); output; end; run;proc sql;  create table missed as select a.* from schedule as a left join collected as b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID and a.PARAMCD=b.PARAMCD and a.AVISITN=b.AVISITN where missing(b.AVISITN);  create table candidates as select a.*,b.AVAL,b.AVISITN as sourcevisit from missed as a left join collected as b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID and a.PARAMCD=b.PARAMCD and b.AVISITN<a.AVISITN;quit;proc sort data=candidates; by STUDYID USUBJID PARAMCD AVISITN descending sourcevisit; run;data carried; length DTYPE $11; set candidates; by STUDYID USUBJID PARAMCD AVISITN; if first.AVISITN; DTYPE='LOCF'; run;data result; length DTYPE $11; set collected carried; run;proc sql;  create table final as select STUDYID, USUBJID, PARAMCD, AVISIT, AVISITN, AVAL, DTYPE, EFFFL from result;quit;proc export data=final outfile="/app/output/adqs.csv" dbms=csv replace; run;