Review ask

As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.

We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.

Summary

Lifecycle: reviewedEditUnder review
What this benchmark means
README.md

Public entries: Python and R. Passing coverage requires an installed inventory.

Goal: carry each collected vital-signs result with its test, result, and collection date, and add the visit number, the trial period, and the study day: VSTESTCD, VSORRES, VSDTC, VISITNUM, EPOCH, and VSDY.

Input: collected vital-signs rows with test, result, collection date, and visit label; demographics rows carrying the reference start date; a trial-visits table carrying the planned number per visit label; and a subject-elements table carrying each subject's elements with their start and end dates.

Variables:

  • VISITNUM is the planned visit number from the trial-visits table for the collected visit label. An unplanned label such as UNSCHEDULED names no planned visit, so it takes the sponsor-assigned number: the latest planned visit number collected before the collection date plus 0.01, so an unscheduled visit after visit 2 reads 2.01.
  • EPOCH is the subject's element whose start and end dates, both ends included, contain the collection date: SCREENING, TREATMENT, or FOLLOW-UP. A day shared by two elements belongs to the later element. Blank when no collection date or no element contains it.
  • VSDY is the study day of the collection date, counted from the subject's reference start date: that date is day 1, there is no day zero, and dates before it count back from -1; missing when the result has no date or the subject has no reference start date.

Note: the visit number follows the visit label while the epoch follows the collection date, so an unscheduled visit still falls in an epoch, and a planned visit with no collection date has a visit number but no epoch.

Inputs

Under review
4 source files

input/dm.csv

Edit
3 rows
input/dm.csv
STUDYIDUSUBJIDRFSTDTC
CATH2025-01-08
CATH2025-02-08
CATH2025-03-01

input/se.csv

Edit
9 rows
input/se.csv
STUDYIDUSUBJIDEPOCHSESTDTCSEENDTC
CATHSCREENING2025-01-012025-01-07
CATHTREATMENT2025-01-082025-03-04
CATHFOLLOW-UP2025-03-052025-12-31
CATHSCREENING2025-02-052025-02-08
CATHTREATMENT2025-02-082025-04-05
CATHFOLLOW-UP2025-04-062025-12-31
CATHSCREENING2025-02-202025-02-28
CATHTREATMENT2025-03-012025-04-26
CATHFOLLOW-UP2025-04-272025-12-31

input/tv.csv

Edit
3 rows
input/tv.csv
VISITVISITNUM
SCREENING1
BASELINE2
WEEK 43

input/vs_raw.csv

Edit
7 rows
input/vs_raw.csv
STUDYIDUSUBJIDVSSEQVISITVSTESTCDVSORRESVSDTC
CATH1SCREENINGSYSBP1282025-01-02
CATH2BASELINESYSBP1322025-01-08
CATH3UNSCHEDULEDSYSBP1452025-01-15
CATH4WEEK 4SYSBP1302025-02-05
CATH1BASELINESYSBP1182025-02-08
CATH2WEEK 4SYSBP122—
CATH1BASELINESYSBP1242025-03-01

Expected output

Under review
7 expected rows

expected/vs.csv

Edit
7 rows
expected/vs.csv
DOMAINSTUDYIDUSUBJIDVSSEQVSTESTCDVSORRESVSDTCVISITVISITNUMEPOCHVSDY
VSCATH1SYSBP1282025-01-02SCREENING1SCREENING-6
VSCATH2SYSBP1322025-01-08BASELINE2TREATMENT1
VSCATH3SYSBP1452025-01-15UNSCHEDULED2.01TREATMENT8
VSCATH4SYSBP1302025-02-05WEEK 43TREATMENT29
VSCATH1SYSBP1182025-02-08BASELINE2TREATMENT1
VSCATH2SYSBP122—WEEK 43——
VSCATH1SYSBP1242025-03-01BASELINE2TREATMENT1
Shaded columns are derived— = missing value

Human review and assessment

Your review

3 questions and your comments · View benchmark discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Variable NameVariable LabelTypeLengthControlled Terms or FormatOriginCoreConversion DefinitionVariable TypeVariable OrderComments for Define
DOMAINDomain AbbreviationCharAssignedSet constant value "VS".SDTM1
STUDYIDStudy IdentifierCharCollectedCopy value from VS_RAW.STUDYID.SDTM2
USUBJIDUnique Subject IdentifierCharCollectedCopy value from VS_RAW.USUBJID.SDTM3
VSSEQSequence NumberNumCollectedCopy value from VS_RAW.VSSEQ.SDTM4
VSTESTCDVital Signs Test Short NameCharCollectedCopy value from VS_RAW.VSTESTCD.SDTM5
VSORRESResult or Finding in Original UnitsCharCollectedCopy value from VS_RAW.VSORRES.SDTM6
VSDTCDate/Time of MeasurementsCharISO 8601CollectedCopy value from VS_RAW.VSDTC.SDTM7
VISITVisit NameCharCollectedCopy value from VS_RAW.VISIT.SDTM8
VISITNUMVisit NumberNumDerivedFirst non-missing of PLANNEDNUM and SPONSORNUM.SDTM9
VSDYStudy Day of Vital SignsNumDerivedCDISC study day of VSDTC against RFSTDTC as day 1; there is no day zero.SDTM10
EPOCHEpochCharSCREENING, TREATMENT, FOLLOW-UPDerivedTake EPOCH from the last SE record matching on USUBJID, where VSDTC falls within SESTDTC to SEENDTC, ordered by SE.SESTDTC, and no match yields blank.SDTM11
VariablePermitted values
EPOCHSCREENING, TREATMENT, FOLLOW-UP
VersionDateAuthorDescriptionReviewerSign-off
1.72026-10-20yamaa docs buildRead a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default
1.62026-10-17yamaa docs buildDescribe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight
1.52026-10-17yamaa docs buildName collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part
1.42026-10-15yamaa docs buildState datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/sdtm-vs-study-day/full.md

Edit
29 lines

Following CDISC SDTM standards, use the provided DM, SE, TV, and VS_RAW datasets to create a VS dataset with one record per collected vital-signs result.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, VSSEQ, VSTESTCD, VSORRES, VSDTC, VISIT, VISITNUM, EPOCH, VSDY

DOMAIN is "VS". VSSEQ is the sequence number as collected. VISITNUM is the planned visit number in TV for the record's visit label. An unplanned visit label that names no planned visit in TV, such as "UNSCHEDULED", takes the sponsor-assigned number: the latest planned visit number collected for the subject before the collection date, plus 0.01, so an unscheduled visit after visit 2 reads 2.01. EPOCH is the epoch of the subject's element in SE whose start and end dates, both ends included, contain the collection date: "SCREENING", "TREATMENT", or "FOLLOW-UP". A day shared by two elements belongs to the later element. EPOCH has no value when the record has no collection date or when no element contains the collection date. VSDY is the study day of the collection date, counted from the subject's reference start date in DM. VSDY has no value when the record has no collection date or the subject has no reference start date in DM. The visit number follows the visit label while the epoch follows the collection date, so an unscheduled visit still falls in an epoch, and a planned visit with no collection date has a visit number but no epoch.

Read the source datasets from /app/input and save the completed dataset as /app/output/vs.csv.

prompts/sdtm-vs-study-day/conventions.md

Edit
14 lines

Following CDISC SDTM standards, use the provided DM, SE, TV, and VS_RAW datasets to create a VS dataset with one record per collected vital-signs result.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, VSSEQ, VSTESTCD, VSORRES, VSDTC, VISIT, VISITNUM, EPOCH, VSDY

DOMAIN is "VS". EPOCH is "SCREENING", "TREATMENT", or "FOLLOW-UP", or has no value. VSDY is the study day of the collection date, counted from the subject's reference start date in DM.

Read the source datasets from /app/input and save the completed dataset as /app/output/vs.csv.

prompts/sdtm-vs-study-day/brief.md

Edit
10 lines

Following CDISC SDTM standards, use the provided DM, SE, TV, and VS_RAW datasets to create a VS dataset with one record per collected vital-signs result.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, VSSEQ, VSTESTCD, VSORRES, VSDTC, VISIT, VISITNUM, EPOCH, VSDY

Read the source datasets from /app/input and save the completed dataset as /app/output/vs.csv.

yamaa code

Public calls generated from benchmark.yaml; results must match the expected output

R

Edit
3 lines
library(yamaa)yamaa_spec("spec.yaml")$run()$output

Python

Edit
3 lines
import yamaayamaa.spec("spec.yaml").run().output

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/sdtm-vs-study-day/result.R

Edit
100 lines
# Reference solution for the yamaa benchmark sdtm-vs-study-day (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(lubridate)library(purrr)library(readr)study_day <- function(ref, coll) {  if (is.na(ref) || is.na(coll)) {    return(NA_integer_)  }  delta <- as.integer(coll - ref)  if (delta >= 0L) delta + 1L else delta}dm <- read_csv(  "/app/input/dm.csv",  col_types = cols(.default = col_character()))se <- read_csv(  "/app/input/se.csv",  col_types = cols(.default = col_character()))tv <- read_csv(  "/app/input/tv.csv",  col_types = cols(.default = col_character()))vs_raw <- read_csv(  "/app/input/vs_raw.csv",  col_types = cols(VSSEQ = col_integer(), .default = col_character()))tv_map <- setNames(suppressWarnings(as.numeric(tv$VISITNUM)), tv$VISIT)ref_map <- setNames(suppressWarnings(ymd(dm$RFSTDTC)), dm$USUBJID)vs <- vs_raw |>  mutate(    coll = suppressWarnings(ymd(VSDTC)),    ref = ref_map[USUBJID],    VSDY = map2_int(ref, coll, study_day, .progress = FALSE),    DOMAIN = "VS"  )vs$VISITNUM <- pmap_dbl(  list(vs$USUBJID, vs$VISIT, vs$VSDTC),  function(usubjid, visit, vsdtc) {    if (visit %in% names(tv_map)) {      return(tv_map[[visit]])    }    day <- suppressWarnings(ymd(vsdtc))    if (is.na(day)) {      return(NA_real_)    }    prior <- vs_raw |>      filter(        USUBJID == usubjid,        VISIT %in% names(tv_map),        !is.na(suppressWarnings(ymd(VSDTC))),        suppressWarnings(ymd(VSDTC)) < day      )    if (nrow(prior) == 0) {      return(NA_real_)    }    max(tv_map[prior$VISIT], na.rm = TRUE) + 0.01  },  .progress = FALSE)vs$EPOCH <- pmap_chr(  list(vs$USUBJID, vs$VSDTC),  function(usubjid, vsdtc) {    day <- suppressWarnings(ymd(vsdtc))    if (is.na(day)) {      return(NA_character_)    }    elements <- se |> filter(USUBJID == usubjid)    hits <- elements |>      filter(        suppressWarnings(ymd(SESTDTC)) <= day,        day <= suppressWarnings(ymd(SEENDTC))      ) |>      arrange(suppressWarnings(ymd(SESTDTC)))    if (nrow(hits) == 0) {      return(NA_character_)    }    hits$EPOCH[nrow(hits)]  },  .progress = FALSE)vs <- vs |>  arrange(USUBJID, VSSEQ) |>  select(    DOMAIN, STUDYID, USUBJID, VSSEQ, VSTESTCD, VSORRES, VSDTC, VISIT,    VISITNUM, EPOCH, VSDY  )dir.create("/app/output", showWarnings = FALSE)write_csv(vs, "/app/output/vs.csv", na = "")

solutions/sdtm-vs-study-day/result.py

Edit
120 lines
# Reference solution for the yamaa benchmark sdtm-vs-study-day (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.import csvfrom datetime import dateINPUT_DIR = "/app/input"OUTPUT_DIR = "/app/output"def read_csv(path):    with open(path, newline="", encoding="utf-8") as f:        return list(csv.DictReader(f))def parse_date(value):    value = (value or "").strip()    if not value:        return None    return date.fromisoformat(value)def format_number(value):    if value is None:        return ""    if float(value).is_integer():        return str(int(value))    return f"{value:.2f}".rstrip("0").rstrip(".")def main():    dm = read_csv(f"{INPUT_DIR}/dm.csv")    se = read_csv(f"{INPUT_DIR}/se.csv")    tv = read_csv(f"{INPUT_DIR}/tv.csv")    vs_raw = read_csv(f"{INPUT_DIR}/vs_raw.csv")    ref_start = {}    for row in dm:        ref_start[row["USUBJID"]] = parse_date(row["RFSTDTC"])    elements = {}    for row in se:        elements.setdefault(row["USUBJID"], []).append(            (row["EPOCH"], parse_date(row["SESTDTC"]), parse_date(row["SEENDTC"]))        )    planned = {}    for row in tv:        planned[row["VISIT"]] = float(row["VISITNUM"])    def unplanned_visitnum(subject, coll_date):        # Latest planned visit number collected for the subject before        # the record's collection date, plus 0.01.        if coll_date is None:            return None        best = None        for row in vs_raw:            if row["USUBJID"] != subject or row["VISIT"] not in planned:                continue            other_date = parse_date(row["VSDTC"])            if other_date is not None and other_date < coll_date:                num = planned[row["VISIT"]]                if best is None or num > best:                    best = num        return None if best is None else best + 0.01    out_rows = []    for row in vs_raw:        subject = row["USUBJID"]        coll_date = parse_date(row["VSDTC"])        if row["VISIT"] in planned:            visitnum = planned[row["VISIT"]]        else:            visitnum = unplanned_visitnum(subject, coll_date)        epoch = ""        if coll_date is not None:            chosen = None            for name, start, end in elements.get(subject, []):                if start is not None and end is not None and start <= coll_date <= end:                    # A day shared by two elements belongs to the later one.                    if chosen is None or start >= chosen[1]:                        chosen = (name, start)            if chosen is not None:                epoch = chosen[0]        vsdy = ""        ref = ref_start.get(subject)        if coll_date is not None and ref is not None:            diff = (coll_date - ref).days            vsdy = str(diff + 1) if diff >= 0 else str(diff)        out_rows.append({            "DOMAIN": "VS",            "STUDYID": row["STUDYID"],            "USUBJID": subject,            "VSSEQ": row["VSSEQ"],            "VSTESTCD": row["VSTESTCD"],            "VSORRES": row["VSORRES"],            "VSDTC": row["VSDTC"],            "VISIT": row["VISIT"],            "VISITNUM": format_number(visitnum),            "EPOCH": epoch,            "VSDY": vsdy,        })    columns = ["DOMAIN", "STUDYID", "USUBJID", "VSSEQ", "VSTESTCD", "VSORRES",               "VSDTC", "VISIT", "VISITNUM", "EPOCH", "VSDY"]    with open(f"{OUTPUT_DIR}/vs.csv", "w", newline="", encoding="utf-8") as f:        writer = csv.DictWriter(f, fieldnames=columns)        writer.writeheader()        writer.writerows(out_rows)if __name__ == "__main__":    main()

solutions/sdtm-vs-study-day/result.sas

Edit
39 lines
/* opensas reference solution for the yamaa benchmark sdtm-vs-study-day.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data dm;  length STUDYID USUBJID RFSTDTC $1024;  infile "/app/input/dm.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. RFSTDTC : $char1024.;run;data se;  length STUDYID USUBJID EPOCH SESTDTC SEENDTC $1024;  infile "/app/input/se.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. EPOCH : $char1024. SESTDTC : $char1024. SEENDTC : $char1024.;run;data tv;  length VISIT $1024;  infile "/app/input/tv.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input VISIT : $char1024. VISITNUM;run;data vs_raw;  length STUDYID USUBJID VISIT VSTESTCD VSORRES VSDTC $1024;  infile "/app/input/vs_raw.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. VSSEQ VISIT : $char1024. VSTESTCD : $char1024. VSORRES : $char1024. VSDTC : $char1024.;run;proc sql; create table collected as select a.*,b.VISITNUM,c.RFSTDTC from vs_raw a left join tv b on a.VISIT=b.VISIT left join dm c on a.STUDYID=c.STUDYID and a.USUBJID=c.USUBJID; quit;data dated; set collected; NODATE=missing(VSDTC); run;proc sort data=dated; by STUDYID USUBJID NODATE VSDTC VSSEQ; run;data numbered; set dated; by STUDYID USUBJID; retain PRIORNUM; if first.USUBJID then PRIORNUM=.; if not missing(VISITNUM) and not missing(VSDTC) then PRIORNUM=VISITNUM; else if missing(VISITNUM) and not missing(VSDTC) and not missing(PRIORNUM) then VISITNUM=PRIORNUM+0.01; run;proc sql; create table candidates as select a.*,b.EPOCH,b.SESTDTC from numbered a left join se b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID and a.VSDTC>=b.SESTDTC and a.VSDTC<=b.SEENDTC and not missing(a.VSDTC); quit;proc sort data=candidates; by STUDYID USUBJID VSSEQ descending SESTDTC; run;data result; length DOMAIN $2; set candidates; by STUDYID USUBJID VSSEQ; if first.VSSEQ; DOMAIN='VS'; VSDY=.; if lengthn(VSDTC)>=10 and lengthn(RFSTDTC)>=10 then do; VSDY=input(substr(VSDTC,1,10),yymmdd10.)-input(substr(RFSTDTC,1,10),yymmdd10.); if VSDY>=0 then VSDY+1; end; run;proc sql;  create table final as select DOMAIN, STUDYID, USUBJID, VSSEQ, VSTESTCD, VSORRES, VSDTC, VISIT, VISITNUM, EPOCH, VSDY from result;quit;proc export data=final outfile="/app/output/vs.csv" dbms=csv replace; run;