Review ask

As a clinical trial statistician or programmer, challenge anything in this benchmark: describe a problem in Comments, or propose a fix with the Edit button on any source file.

  1. Task: Would a real study need this derivation, specified this way? Does it follow CDISC SDTM: variable names, controlled terms, record structure? Flag any rule that is ambiguous or that a sponsor would handle differently.
  2. Inputs: Do they look like data collected in a real trial, and do they include the situations you would expect to meet in one?
  3. Expected output: Would independent QC programming reproduce every value from these inputs and rules?

The other tabs beside Comments are background for assessment, not under review.

Summary

Lifecycle: reviewedEditUnder review
What this benchmark means
README.md

Goal: carry each collected vital-signs result with its test, result, and collection date, and add the visit number, the trial period, and the study day: VSTESTCD, VSORRES, VSDTC, VISITNUM, EPOCH, and VSDY.

Input: collected vital-signs rows with test, result, collection date, and visit label; demographics rows carrying the reference start date; a trial-visits table carrying the planned number per visit label; and a subject-elements table carrying each subject's elements with their start and end dates.

Variables:

  • VISITNUM is the planned visit number from the trial-visits table for the collected visit label. An unplanned label such as UNSCHEDULED names no planned visit, so it takes the sponsor-assigned number: the latest planned visit number collected before the collection date plus 0.01, so an unscheduled visit after visit 2 reads 2.01.
  • EPOCH is the subject's element whose start and end dates, both ends included, contain the collection date: SCREENING, TREATMENT, or FOLLOW-UP. A day shared by two elements belongs to the later element. Blank when no collection date or no element contains it.
  • VSDY is the study day of the collection date, counted from the subject's reference start date: that date is day 1, there is no day zero, and dates before it count back from -1; missing when the result has no date or the subject has no reference start date.

Note: the visit number follows the visit label while the epoch follows the collection date, so an unscheduled visit still falls in an epoch, and a planned visit with no collection date has a visit number but no epoch.

Inputs

Under review
4 source files

input/dm.csv

Edit
3 rows
input/dm.csv
STUDYIDUSUBJIDRFSTDTC
CATH2025-01-08
CATH2025-02-08
CATH2025-03-01

input/se.csv

Edit
9 rows
input/se.csv
STUDYIDUSUBJIDEPOCHSESTDTCSEENDTC
CATHSCREENING2025-01-012025-01-07
CATHTREATMENT2025-01-082025-03-04
CATHFOLLOW-UP2025-03-052025-12-31
CATHSCREENING2025-02-052025-02-08
CATHTREATMENT2025-02-082025-04-05
CATHFOLLOW-UP2025-04-062025-12-31
CATHSCREENING2025-02-202025-02-28
CATHTREATMENT2025-03-012025-04-26
CATHFOLLOW-UP2025-04-272025-12-31

input/tv.csv

Edit
3 rows
input/tv.csv
VISITVISITNUM
SCREENING1
BASELINE2
WEEK 43

input/vs_raw.csv

Edit
7 rows
input/vs_raw.csv
STUDYIDUSUBJIDVSSEQVISITVSTESTCDVSORRESVSDTC
CATH1SCREENINGSYSBP1282025-01-02
CATH2BASELINESYSBP1322025-01-08
CATH3UNSCHEDULEDSYSBP1452025-01-15
CATH4WEEK 4SYSBP1302025-02-05
CATH1BASELINESYSBP1182025-02-08
CATH2WEEK 4SYSBP122—
CATH1BASELINESYSBP1242025-03-01

Expected output

Under review
7 expected rows

expected/vs.csv

Edit
7 rows
expected/vs.csv
DOMAINSTUDYIDUSUBJIDVSSEQVSTESTCDVSORRESVSDTCVISITVISITNUMEPOCHVSDY
VSCATH1SYSBP1282025-01-02SCREENING1SCREENING-6
VSCATH2SYSBP1322025-01-08BASELINE2TREATMENT1
VSCATH3SYSBP1452025-01-15UNSCHEDULED2.01TREATMENT8
VSCATH4SYSBP1302025-02-05WEEK 43TREATMENT29
VSCATH1SYSBP1182025-02-08BASELINE2TREATMENT1
VSCATH2SYSBP122—WEEK 43——
VSCATH1SYSBP1242025-03-01BASELINE2TREATMENT1
Shaded columns are derived— = missing value

Comments and assessment materials

Comments

Signed in with GitHub; saved in repository Discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Variable NameVariable LabelTypeLengthControlled Terms or FormatOriginCoreConversion DefinitionVariable TypeVariable OrderComments for Define
DOMAINDomain AbbreviationCharAssignedSet constant value "VS".SDTM1
STUDYIDStudy IdentifierCharCollectedCopy value from VS_RAW.STUDYID.SDTM2
USUBJIDUnique Subject IdentifierCharCollectedCopy value from VS_RAW.USUBJID.SDTM3
VSSEQSequence NumberNumCollectedCopy value from VS_RAW.VSSEQ.SDTM4
VSTESTCDVital Signs Test Short NameCharCollectedCopy value from VS_RAW.VSTESTCD.SDTM5
VSORRESResult or Finding in Original UnitsCharCollectedCopy value from VS_RAW.VSORRES.SDTM6
VSDTCDate/Time of MeasurementsCharISO 8601CollectedCopy value from VS_RAW.VSDTC.SDTM7
VISITVisit NameCharCollectedCopy value from VS_RAW.VISIT.SDTM8
VISITNUMVisit NumberNumDerivedFirst non-missing of PLANNEDNUM and SPONSORNUM.SDTM9
VSDYStudy Day of Vital SignsNumDerivedCDISC study day of VSDTC against RFSTDTC as day 1; there is no day zero.SDTM10
EPOCHEpochCharSCREENING, TREATMENT, FOLLOW-UPDerivedTake EPOCH from the last SE record matching on USUBJID, where VSDTC falls within SESTDTC to SEENDTC, ordered by SE.SESTDTC, and no match yields blank.SDTM11
VariablePermitted values
EPOCHSCREENING, TREATMENT, FOLLOW-UP
VersionDateAuthorDescriptionReviewerSign-off
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/sdtm-vs-study-day/full.md

Edit
25 lines

Following CDISC SDTM standards, use the provided VS_RAW, DM, TV, and SE datasets to create a VS dataset with one record per collected vital-signs result.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, VSSEQ, VSTESTCD, VSORRES, VSDTC, VISIT, VISITNUM, EPOCH, VSDY

VISITNUM is the planned visit number from the trial-visits table for the collected visit label. An unplanned label such as UNSCHEDULED names no planned visit, so it takes a sponsor-assigned number based on the latest planned visit number collected before the collection date, and an unscheduled visit after visit 2 reads 2.01. EPOCH is the subject's element whose start and end dates, both ends included, contain the collection date: SCREENING, TREATMENT, or FOLLOW-UP. A day shared by two elements belongs to the later element. EPOCH has no value when there is no collection date or no element contains it. VSDY is the study day of the collection date relative to the subject's reference start date.

The visit number follows the visit label while the epoch follows the collection date, so an unscheduled visit still falls in an epoch, and a planned visit with no collection date has a visit number but no epoch.

Read the source datasets from /app/input and save the completed dataset as /app/output/vs.csv.

prompts/sdtm-vs-study-day/conventions.md

Edit
13 lines

Following CDISC SDTM standards, use the provided VS_RAW, DM, TV, and SE datasets to create a VS dataset with one record per collected vital-signs result.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, VSSEQ, VSTESTCD, VSORRES, VSDTC, VISIT, VISITNUM, EPOCH, VSDY

VISITNUM for an unscheduled visit after visit 2 reads 2.01. EPOCH is SCREENING, TREATMENT, or FOLLOW-UP, or has no value.

Read the source datasets from /app/input and save the completed dataset as /app/output/vs.csv.

prompts/sdtm-vs-study-day/brief.md

Edit
10 lines

Following CDISC SDTM standards, use the provided VS_RAW, DM, TV, and SE datasets to create a VS dataset with one record per collected vital-signs result.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, VSSEQ, VSTESTCD, VSORRES, VSDTC, VISIT, VISITNUM, EPOCH, VSDY

Read the source datasets from /app/input and save the completed dataset as /app/output/vs.csv.

yamaa code

Runs the yamaa specification; its result must match the expected output

run.R

Edit
4 lines
library(yamaa)vs <- yamaa_domain("spec.yaml")$outputvs

run.py

Edit
4 lines
import yamaavs = yamaa.yamaa_domain("spec.yaml").outputvs

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/sdtm-vs-study-day/result.R

Edit
100 lines
# Reference solution for the yamaa benchmark sdtm-vs-study-day (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(lubridate)library(purrr)library(readr)study_day <- function(ref, coll) {  if (is.na(ref) || is.na(coll)) {    return(NA_integer_)  }  delta <- as.integer(coll - ref)  if (delta >= 0L) delta + 1L else delta}dm <- read_csv(  "/app/input/dm.csv",  col_types = cols(.default = col_character()))se <- read_csv(  "/app/input/se.csv",  col_types = cols(.default = col_character()))tv <- read_csv(  "/app/input/tv.csv",  col_types = cols(.default = col_character()))vs_raw <- read_csv(  "/app/input/vs_raw.csv",  col_types = cols(VSSEQ = col_integer(), .default = col_character()))tv_map <- setNames(suppressWarnings(as.numeric(tv$VISITNUM)), tv$VISIT)ref_map <- setNames(suppressWarnings(ymd(dm$RFSTDTC)), dm$USUBJID)vs <- vs_raw |>  mutate(    coll = suppressWarnings(ymd(VSDTC)),    ref = ref_map[USUBJID],    VSDY = map2_int(ref, coll, study_day, .progress = FALSE),    DOMAIN = "VS"  )vs$VISITNUM <- pmap_dbl(  list(vs$USUBJID, vs$VISIT, vs$VSDTC),  function(usubjid, visit, vsdtc) {    if (visit %in% names(tv_map)) {      return(tv_map[[visit]])    }    day <- suppressWarnings(ymd(vsdtc))    if (is.na(day)) {      return(NA_real_)    }    prior <- vs_raw |>      filter(        USUBJID == usubjid,        VISIT %in% names(tv_map),        !is.na(suppressWarnings(ymd(VSDTC))),        suppressWarnings(ymd(VSDTC)) < day      )    if (nrow(prior) == 0) {      return(NA_real_)    }    max(tv_map[prior$VISIT], na.rm = TRUE) + 0.01  },  .progress = FALSE)vs$EPOCH <- pmap_chr(  list(vs$USUBJID, vs$VSDTC),  function(usubjid, vsdtc) {    day <- suppressWarnings(ymd(vsdtc))    if (is.na(day)) {      return(NA_character_)    }    elements <- se |> filter(USUBJID == usubjid)    hits <- elements |>      filter(        suppressWarnings(ymd(SESTDTC)) <= day,        day <= suppressWarnings(ymd(SEENDTC))      ) |>      arrange(suppressWarnings(ymd(SESTDTC)))    if (nrow(hits) == 0) {      return(NA_character_)    }    hits$EPOCH[nrow(hits)]  },  .progress = FALSE)vs <- vs |>  arrange(USUBJID, VSSEQ) |>  select(    DOMAIN, STUDYID, USUBJID, VSSEQ, VSTESTCD, VSORRES, VSDTC, VISIT,    VISITNUM, EPOCH, VSDY  )dir.create("/app/output", showWarnings = FALSE)write_csv(vs, "/app/output/vs.csv", na = "")

solutions/sdtm-vs-study-day/result.py

Edit
121 lines
# Reference solution for the yamaa benchmark sdtm-vs-study-day (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.from datetime import datefrom pathlib import Pathimport polars as pldef parse_date(text: str | None) -> date | None:    if text is None or text == "":        return None    try:        # Collection and element dates are ISO dates without times.        return date.fromisoformat(text[:10])    except ValueError:        return Nonedef study_day(ref: date | None, coll: date | None) -> int | None:    if ref is None or coll is None:        return None    delta = (coll - ref).days    return delta + 1 if delta >= 0 else deltadm = pl.read_csv("/app/input/dm.csv", infer_schema=False)se = pl.read_csv("/app/input/se.csv", infer_schema=False)tv = pl.read_csv("/app/input/tv.csv", infer_schema=False)vs_raw = pl.read_csv("/app/input/vs_raw.csv", infer_schema=False).with_columns(    pl.col("VSSEQ").cast(pl.Int64))tv_map = {row["VISIT"]: row["VISITNUM"] for row in tv.to_dicts()}ref_map = {    row["USUBJID"]: parse_date(row["RFSTDTC"]) for row in dm.to_dicts()}se_by_subject: dict[str, list[dict]] = {}for row in se.to_dicts():    se_by_subject.setdefault(row["USUBJID"], []).append(        {            "EPOCH": row["EPOCH"],            "start": parse_date(row["SESTDTC"]),            "end": parse_date(row["SEENDTC"]),        }    )# Latest planned visit number collected before each date, per subject, for# unplanned labels such as UNSCHEDULED.planned_by_subject: dict[str, list[tuple[date, float]]] = {}for row in vs_raw.to_dicts():    if row["VISIT"] in tv_map and row["VSDTC"] not in (None, ""):        coll = parse_date(row["VSDTC"])        if coll is not None:            planned_by_subject.setdefault(row["USUBJID"], []).append(                (coll, float(tv_map[row["VISIT"]]))            )rows = []for row in vs_raw.to_dicts():    usubjid = row["USUBJID"]    coll = parse_date(row["VSDTC"])    # Planned visit number, or the latest planned number before the    # collection date plus .01 for an unplanned label.    if row["VISIT"] in tv_map:        visitnum: float | None = float(tv_map[row["VISIT"]])    else:        candidates = [            num for day, num in planned_by_subject.get(usubjid, []) if day < coll        ] if coll is not None else []        visitnum = max(candidates) + 0.01 if candidates else None    # Element containing the collection date; a shared day belongs to the    # later element.    epoch = None    if coll is not None:        hits = [            e            for e in se_by_subject.get(usubjid, [])            if e["start"] is not None            and e["end"] is not None            and e["start"] <= coll <= e["end"]        ]        if hits:            hits.sort(key=lambda e: e["start"])            epoch = hits[-1]["EPOCH"]    vsdy = study_day(ref_map.get(usubjid), coll)    rows.append(        {            "DOMAIN": "VS",            "STUDYID": row["STUDYID"],            "USUBJID": usubjid,            "VSSEQ": row["VSSEQ"],            "VSTESTCD": row["VSTESTCD"],            "VSORRES": row["VSORRES"],            "VSDTC": row["VSDTC"],            "VISIT": row["VISIT"],            "VISITNUM": visitnum,            "EPOCH": epoch,            "VSDY": vsdy,        }    )vs = pl.DataFrame(rows).sort(["USUBJID", "VSSEQ"]).select(    "DOMAIN",    "STUDYID",    "USUBJID",    "VSSEQ",    "VSTESTCD",    "VSORRES",    "VSDTC",    "VISIT",    "VISITNUM",    "EPOCH",    "VSDY",)Path("/app/output").mkdir(exist_ok=True)vs.write_csv("/app/output/vs.csv")

solutions/sdtm-vs-study-day/result.sas

Edit
39 lines
/* opensas reference solution for the yamaa benchmark sdtm-vs-study-day.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data dm;  length STUDYID USUBJID RFSTDTC $1024;  infile "/app/input/dm.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. RFSTDTC : $char1024.;run;data se;  length STUDYID USUBJID EPOCH SESTDTC SEENDTC $1024;  infile "/app/input/se.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. EPOCH : $char1024. SESTDTC : $char1024. SEENDTC : $char1024.;run;data tv;  length VISIT $1024;  infile "/app/input/tv.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input VISIT : $char1024. VISITNUM;run;data vs_raw;  length STUDYID USUBJID VISIT VSTESTCD VSORRES VSDTC $1024;  infile "/app/input/vs_raw.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. VSSEQ VISIT : $char1024. VSTESTCD : $char1024. VSORRES : $char1024. VSDTC : $char1024.;run;proc sql; create table collected as select a.*,b.VISITNUM,c.RFSTDTC from vs_raw a left join tv b on a.VISIT=b.VISIT left join dm c on a.STUDYID=c.STUDYID and a.USUBJID=c.USUBJID; quit;data dated; set collected; NODATE=missing(VSDTC); run;proc sort data=dated; by STUDYID USUBJID NODATE VSDTC VSSEQ; run;data numbered; set dated; by STUDYID USUBJID; retain PRIORNUM; if first.USUBJID then PRIORNUM=.; if not missing(VISITNUM) and not missing(VSDTC) then PRIORNUM=VISITNUM; else if missing(VISITNUM) and not missing(VSDTC) and not missing(PRIORNUM) then VISITNUM=PRIORNUM+0.01; run;proc sql; create table candidates as select a.*,b.EPOCH,b.SESTDTC from numbered a left join se b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID and a.VSDTC>=b.SESTDTC and a.VSDTC<=b.SEENDTC and not missing(a.VSDTC); quit;proc sort data=candidates; by STUDYID USUBJID VSSEQ descending SESTDTC; run;data result; length DOMAIN $2; set candidates; by STUDYID USUBJID VSSEQ; if first.VSSEQ; DOMAIN='VS'; VSDY=.; if lengthn(VSDTC)>=10 and lengthn(RFSTDTC)>=10 then do; VSDY=input(substr(VSDTC,1,10),yymmdd10.)-input(substr(RFSTDTC,1,10),yymmdd10.); if VSDY>=0 then VSDY+1; end; run;proc sql;  create table final as select DOMAIN, STUDYID, USUBJID, VSSEQ, VSTESTCD, VSORRES, VSDTC, VISIT, VISITNUM, EPOCH, VSDY from result;quit;proc export data=final outfile="/app/output/vs.csv" dbms=csv replace; run;