As a clinical trial statistician or programmer, challenge anything in this benchmark: describe a problem in Comments, or propose a fix with the Edit button on any source file.
- Task: Would a real study need this derivation, specified this way? Does it follow CDISC ADaM: variable names, controlled terms, record structure? Flag any rule that is ambiguous or that a sponsor would handle differently.
- Inputs: Do they look like data collected in a real trial, and do they include the situations you would expect to meet in one?
- Expected output: Would independent QC programming reproduce every value from these inputs and rules?
The other tabs beside Comments are background for assessment, not under review.
Summary
README.mdGoal: give every subject in the efficacy population a total score
record at each scheduled visit, adding a last observation carried forward
record for each visit missed, with the analysis value (AVAL), the
record-type flag (DTYPE), and the efficacy population flag (EFFFL).
Input: one subject-level record per subject carrying the efficacy
population flag, and the collected ADAS-Cog (Alzheimer's Disease
Assessment Scale, cognitive subscale) total scores, each already assigned
to an analysis visit (AVISITN).
Variables:
AVAL: the collected score on a collected record. On a carried-forward record, the score from the closest earlier visit that has one for the same subject; empty when the subject has no earlier score.DTYPE: empty on a collected record;LOCFon a record added for a missed scheduled visit.EFFFL: the subject's efficacy population flag, repeated on every record.
Note: the protocol schedules Weeks 8, 16, and 24 for every subject in the efficacy population. A scheduled visit is missed when the subject has no collected score there, and a collected record with an empty score is left out, so at a scheduled visit it counts as missed. Subjects outside the efficacy population keep their collected records and gain none, and baseline is not a scheduled visit, so a missing baseline adds nothing. The input carries at most one score per subject per visit.
Inputs
Under review2 source filesinput/adsl.csv
Edit6 rows| STUDYID | USUBJID | EFFFL |
|---|---|---|
| YAMAA-01 | Y | |
| YAMAA-01 | Y | |
| YAMAA-01 | Y | |
| YAMAA-01 | Y | |
| YAMAA-01 | N | |
| YAMAA-01 | Y |
input/qs.csv
Edit14 rows| STUDYID | USUBJID | QSSEQ | PARAMCD | AVISIT | AVISITN | AVAL |
|---|---|---|---|---|---|---|
| YAMAA-01 | 1 | ACTOT | Baseline | 0 | 22 | |
| YAMAA-01 | 2 | ACTOT | Week 8 | 8 | 24 | |
| YAMAA-01 | 3 | ACTOT | Week 16 | 16 | 25 | |
| YAMAA-01 | 4 | ACTOT | Week 24 | 24 | 27 | |
| YAMAA-01 | 1 | ACTOT | Baseline | 0 | 30 | |
| YAMAA-01 | 2 | ACTOT | Week 8 | 8 | 31 | |
| YAMAA-01 | 1 | ACTOT | Baseline | 0 | 18 | |
| YAMAA-01 | 2 | ACTOT | Week 8 | 8 | — | |
| YAMAA-01 | 3 | ACTOT | Week 16 | 16 | 21 | |
| YAMAA-01 | 1 | ACTOT | Week 16 | 16 | 15 | |
| YAMAA-01 | 2 | ACTOT | Week 24 | 24 | 17 | |
| YAMAA-01 | 1 | ACTOT | Baseline | 0 | 26 | |
| YAMAA-01 | 2 | ACTOT | Week 8 | 8 | 28 | |
| YAMAA-01 | 1 | ACTOT | Baseline | 0 | 0 |
Expected output
Under review21 expected rowsexpected/adqs.csv
Edit21 rows| STUDYID | USUBJID | PARAMCD | AVISIT | AVISITN | AVAL | DTYPE | EFFFL |
|---|---|---|---|---|---|---|---|
| YAMAA-01 | ACTOT | Baseline | 0 | 22 | — | Y | |
| YAMAA-01 | ACTOT | Week 8 | 8 | 24 | — | Y | |
| YAMAA-01 | ACTOT | Week 16 | 16 | 25 | — | Y | |
| YAMAA-01 | ACTOT | Week 24 | 24 | 27 | — | Y | |
| YAMAA-01 | ACTOT | Baseline | 0 | 30 | — | Y | |
| YAMAA-01 | ACTOT | Week 8 | 8 | 31 | — | Y | |
| YAMAA-01 | ACTOT | Week 16 | 16 | 31 | LOCF | Y | |
| YAMAA-01 | ACTOT | Week 24 | 24 | 31 | LOCF | Y | |
| YAMAA-01 | ACTOT | Baseline | 0 | 18 | — | Y | |
| YAMAA-01 | ACTOT | Week 8 | 8 | 18 | LOCF | Y | |
| YAMAA-01 | ACTOT | Week 16 | 16 | 21 | — | Y | |
| YAMAA-01 | ACTOT | Week 24 | 24 | 21 | LOCF | Y | |
| YAMAA-01 | ACTOT | Week 8 | 8 | — | LOCF | Y | |
| YAMAA-01 | ACTOT | Week 16 | 16 | 15 | — | Y | |
| YAMAA-01 | ACTOT | Week 24 | 24 | 17 | — | Y | |
| YAMAA-01 | ACTOT | Baseline | 0 | 26 | — | N | |
| YAMAA-01 | ACTOT | Week 8 | 8 | 28 | — | N | |
| YAMAA-01 | ACTOT | Baseline | 0 | 0 | — | Y | |
| YAMAA-01 | ACTOT | Week 8 | 8 | 0 | LOCF | Y | |
| YAMAA-01 | ACTOT | Week 16 | 16 | 0 | LOCF | Y | |
| YAMAA-01 | ACTOT | Week 24 | 24 | 0 | LOCF | Y |
Comments and assessment materials
| Dataset Name | Variable Name | Variable Label | Type | Codelist/Controlled Terms | Core | Computational Method | Origin |
|---|---|---|---|---|---|---|---|
| ADQS | STUDYID | Study Identifier | character | Set per constructed row; see the Row construction sheet. | Derived | ||
| ADQS | USUBJID | Unique Subject Identifier | character | Set per constructed row; see the Row construction sheet. | Derived | ||
| ADQS | PARAMCD | Parameter Code | character | Set per constructed row; see the Row construction sheet. | Derived | ||
| ADQS | AVISIT | Analysis Visit | character | Set per constructed row; see the Row construction sheet. | Derived | ||
| ADQS | AVISITN | Analysis Visit (N) | numeric | Set per constructed row; see the Row construction sheet. | Derived | ||
| ADQS | DTYPE | Derivation Type | character | Set per constructed row; see the Row construction sheet. | Derived | ||
| ADQS | EFFFL | Efficacy Population Flag | character | Set per constructed row; see the Row construction sheet. | Derived | ||
| ADQS | AVAL | Analysis Value | numeric | Last observation carried forward from AVALCOL within each group of STUDYID, USUBJID, PARAMCD, ordered by AVISITN. | Derived |
| Template | Include when | STUDYID | USUBJID | PARAMCD | AVISIT | AVISITN | AVALCOL | DTYPE | EFFFL |
|---|---|---|---|---|---|---|---|---|---|
| collected | QS.AVAL IS NOT NULL | Copy QS.STUDYID | Copy QS.USUBJID | Copy QS.PARAMCD | Copy QS.AVISIT | Copy QS.AVISITN | Copy QS.AVAL | Constant "None" | Copy ADSL.EFFFL |
| week_8 | ADSL.EFFFL = 'Y' AND COLLECTED.AVAL IS NULL | Copy ADSL.STUDYID | Copy ADSL.USUBJID | Constant "ACTOT" | Constant "Week 8" | Constant "8" | Constant "None" | Constant "LOCF" | Copy ADSL.EFFFL |
| week_16 | ADSL.EFFFL = 'Y' AND COLLECTED.AVAL IS NULL | Copy ADSL.STUDYID | Copy ADSL.USUBJID | Constant "ACTOT" | Constant "Week 16" | Constant "16" | Constant "None" | Constant "LOCF" | Copy ADSL.EFFFL |
| week_24 | ADSL.EFFFL = 'Y' AND COLLECTED.AVAL IS NULL | Copy ADSL.STUDYID | Copy ADSL.USUBJID | Constant "ACTOT" | Constant "Week 24" | Constant "24" | Constant "None" | Constant "LOCF" | Copy ADSL.EFFFL |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/adam-adqs-missed-visit-locf/full.md
Edit28 linesFollowing CDISC ADaM standards, use the provided ADSL and QS datasets to create an ADQS dataset with one record per subject per parameter per analysis visit.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, AVISIT, AVISITN, AVAL, DTYPE, EFFFL
Use PARAMCD "ACTOT". The protocol schedules Weeks 8, 16, and 24 (AVISIT "Week 8", "Week 16", and "Week 24", with AVISITN 8, 16, and 24) for every subject in the efficacy population; baseline is not a scheduled visit, so a missing baseline adds nothing.
Keep every collected score with DTYPE empty. A collected record with an empty score is left out, so at a scheduled visit it counts as missed.
Add one record with DTYPE "LOCF" for each missed scheduled visit of a subject in the efficacy population. Its value is the score from the closest earlier visit that has one for the same subject, and a collected zero is a real score that carries forward; it has no value when the subject has no earlier score. Subjects outside the efficacy population keep their collected records and gain none.
EFFFL is the subject's efficacy population flag, repeated on every record.
Read the source datasets from /app/input and save the completed dataset as /app/output/adqs.csv.
prompts/adam-adqs-missed-visit-locf/conventions.md
Edit14 linesFollowing CDISC ADaM standards, use the provided ADSL and QS datasets to create an ADQS dataset with one record per subject per parameter per analysis visit.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, AVISIT, AVISITN, AVAL, DTYPE, EFFFL
Use PARAMCD "ACTOT". AVISIT "Week 8", "Week 16", and "Week 24" have AVISITN 8, 16, and 24.
DTYPE is "LOCF" or empty.
Read the source datasets from /app/input and save the completed dataset as /app/output/adqs.csv.
prompts/adam-adqs-missed-visit-locf/brief.md
Edit9 linesFollowing CDISC ADaM standards, use the provided ADSL and QS datasets to create an ADQS dataset with one record per subject per parameter per analysis visit.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, AVISIT, AVISITN, AVAL, DTYPE, EFFFL
Read the source datasets from /app/input and save the completed dataset as /app/output/adqs.csv.
run.R
Edit4 lineslibrary(yamaa)adqs <- yamaa_domain("spec.yaml")$outputadqsrun.py
Edit4 linesimport yamaaadqs = yamaa.yamaa_domain("spec.yaml").outputadqssolutions/adam-adqs-missed-visit-locf/result.R
Edit70 lines# Reference solution for the yamaa benchmark adam-adqs-missed-visit-locf (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)library(tidyr)adsl <- read_csv( "/app/input/adsl.csv", col_types = cols(.default = col_character()))qs <- read_csv( "/app/input/qs.csv", col_types = cols( AVISITN = col_integer(), AVAL = col_double(), .default = col_character() ))# Collected scores; an empty score is left out, so it counts as missed.collected <- qs |> filter(PARAMCD %in% "ACTOT", !is.na(AVAL)) |> left_join(select(adsl, USUBJID, EFFFL), by = "USUBJID") |> mutate(DTYPE = NA_character_) |> select(STUDYID, USUBJID, PARAMCD, AVISIT, AVISITN, AVAL, DTYPE, EFFFL)# One LOCF record for each missed scheduled visit in the efficacy# population; its value is the closest earlier visit with a score, where# a collected zero carries forward and no earlier score leaves no value.scheduled <- tibble( AVISIT = c("Week 8", "Week 16", "Week 24"), AVISITN = c(8L, 16L, 24L))eff_subjects <- adsl |> filter(EFFFL %in% "Y") |> select(STUDYID, USUBJID, EFFFL)missed_keys <- eff_subjects |> crossing(scheduled) |> anti_join( select(collected, USUBJID, AVISITN), by = c("USUBJID", "AVISITN") )carried <- missed_keys |> left_join( collected |> select(USUBJID, EARLIER_VISITN = AVISITN, EARLIER_AVAL = AVAL), by = "USUBJID", relationship = "many-to-many" ) |> filter(!is.na(EARLIER_AVAL), EARLIER_VISITN < AVISITN) |> group_by(STUDYID, USUBJID, AVISIT, AVISITN, EFFFL) |> slice_max(EARLIER_VISITN, n = 1, with_ties = FALSE) |> ungroup() |> transmute(USUBJID, AVISITN, CARRIED = EARLIER_AVAL)missed <- missed_keys |> left_join(carried, by = c("USUBJID", "AVISITN")) |> mutate(PARAMCD = "ACTOT", AVAL = CARRIED, DTYPE = "LOCF") |> select(STUDYID, USUBJID, PARAMCD, AVISIT, AVISITN, AVAL, DTYPE, EFFFL)adqs <- bind_rows(collected, missed) |> arrange(USUBJID, AVISITN)dir.create("/app/output", showWarnings = FALSE)write_csv(adqs, "/app/output/adqs.csv", na = "")solutions/adam-adqs-missed-visit-locf/result.py
Edit77 lines# Reference solution for the yamaa benchmark adam-adqs-missed-visit-locf (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.from pathlib import Pathimport polars as pladsl = pl.read_csv("/app/input/adsl.csv", infer_schema=False)qs = pl.read_csv("/app/input/qs.csv", infer_schema=False).with_columns( pl.col("AVISITN").cast(pl.Int64), pl.col("AVAL").cast(pl.Float64),)# Collected scores; an empty score is left out, so it counts as missed.collected = ( qs.filter(pl.col("PARAMCD") == "ACTOT", pl.col("AVAL").is_not_null()) .join(adsl.select("USUBJID", "EFFFL"), on="USUBJID", how="left") .with_columns(DTYPE=pl.lit(None, dtype=pl.String)) .select( "STUDYID", "USUBJID", "PARAMCD", "AVISIT", "AVISITN", "AVAL", "DTYPE", "EFFFL" ))# The protocol schedules Weeks 8, 16, and 24 for the efficacy population.scheduled = pl.DataFrame( {"AVISIT": ["Week 8", "Week 16", "Week 24"], "AVISITN": [8, 16, 24]})eff_subjects = adsl.filter(pl.col("EFFFL") == "Y").select( "STUDYID", "USUBJID", "EFFFL")planned = eff_subjects.join(scheduled, how="cross")missed_keys = planned.join( collected.select("USUBJID", "AVISITN"), on=["USUBJID", "AVISITN"], how="anti")# Each missed visit carries the closest earlier visit with a score, where# a collected zero carries forward and no earlier score leaves no value.earlier_by_subject: dict[str, list[tuple[int, float]]] = {}for row in collected.sort(["USUBJID", "AVISITN"]).iter_rows(named=True): earlier_by_subject.setdefault(row["USUBJID"], []).append( (row["AVISITN"], row["AVAL"]) )missed_rows = []for row in missed_keys.sort(["USUBJID", "AVISITN"]).iter_rows(named=True): carried = None for visitn, aval in earlier_by_subject.get(row["USUBJID"], []): if visitn < row["AVISITN"]: carried = aval else: break missed_rows.append({**row, "PARAMCD": "ACTOT", "AVAL": carried, "DTYPE": "LOCF"})missed = pl.DataFrame( missed_rows, schema={ "STUDYID": pl.String, "USUBJID": pl.String, "EFFFL": pl.String, "AVISIT": pl.String, "AVISITN": pl.Int64, "PARAMCD": pl.String, "AVAL": pl.Float64, "DTYPE": pl.String, }, strict=False,).select( "STUDYID", "USUBJID", "PARAMCD", "AVISIT", "AVISITN", "AVAL", "DTYPE", "EFFFL")adqs = pl.concat([collected, missed], how="diagonal").sort(["USUBJID", "AVISITN"])Path("/app/output").mkdir(exist_ok=True)adqs.write_csv("/app/output/adqs.csv")solutions/adam-adqs-missed-visit-locf/result.sas
Edit32 lines/* opensas reference solution for the yamaa benchmark adam-adqs-missed-visit-locf. Reads the benchmark inputs and derives the requested output. Execute with the existing opensas runtime: opensas /app/output/result.sas. */data adsl; length STUDYID USUBJID EFFFL $1024; infile "/app/input/adsl.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. EFFFL : $char1024.;run;data qs; length STUDYID USUBJID PARAMCD AVISIT $1024; infile "/app/input/qs.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. QSSEQ PARAMCD : $char1024. AVISIT : $char1024. AVISITN AVAL;run;proc sql; create table collected as select a.STUDYID,a.USUBJID,a.PARAMCD,a.AVISIT,a.AVISITN,a.AVAL,'' as DTYPE,b.EFFFL from qs as a left join adsl as b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID where not missing(a.AVAL);quit;data schedule; length AVISIT PARAMCD $1024; set adsl; if EFFFL='Y'; PARAMCD='ACTOT'; do AVISITN=8,16,24; AVISIT=catx(' ','Week',strip(put(AVISITN,best32.))); output; end; run;proc sql; create table missed as select a.* from schedule as a left join collected as b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID and a.PARAMCD=b.PARAMCD and a.AVISITN=b.AVISITN where missing(b.AVISITN); create table candidates as select a.*,b.AVAL,b.AVISITN as sourcevisit from missed as a left join collected as b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID and a.PARAMCD=b.PARAMCD and b.AVISITN<a.AVISITN;quit;proc sort data=candidates; by STUDYID USUBJID PARAMCD AVISITN descending sourcevisit; run;data carried; length DTYPE $11; set candidates; by STUDYID USUBJID PARAMCD AVISITN; if first.AVISITN; DTYPE='LOCF'; run;data result; length DTYPE $11; set collected carried; run;proc sql; create table final as select STUDYID, USUBJID, PARAMCD, AVISIT, AVISITN, AVAL, DTYPE, EFFFL from result;quit;proc export data=final outfile="/app/output/adqs.csv" dbms=csv replace; run;
yamaa
Comments
Signed in with GitHub; saved in repository Discussions