As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.
We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.
Summary
README.mdPublic entries: Python and R. Passing coverage requires an installed inventory.
Goal: carry each collected vital-signs result with its test,
result, and collection date, and add the visit number, the trial
period, and the study day: VSTESTCD, VSORRES, VSDTC,
VISITNUM, EPOCH, and VSDY.
Input: collected vital-signs rows with test, result, collection date, and visit label; demographics rows carrying the reference start date; a trial-visits table carrying the planned number per visit label; and a subject-elements table carrying each subject's elements with their start and end dates.
Variables:
VISITNUMis the planned visit number from the trial-visits table for the collected visit label. An unplanned label such asUNSCHEDULEDnames no planned visit, so it takes the sponsor-assigned number: the latest planned visit number collected before the collection date plus0.01, so an unscheduled visit after visit2reads2.01.EPOCHis the subject's element whose start and end dates, both ends included, contain the collection date:SCREENING,TREATMENT, orFOLLOW-UP. A day shared by two elements belongs to the later element. Blank when no collection date or no element contains it.VSDYis the study day of the collection date, counted from the subject's reference start date: that date is day 1, there is no day zero, and dates before it count back from -1; missing when the result has no date or the subject has no reference start date.
Note: the visit number follows the visit label while the epoch follows the collection date, so an unscheduled visit still falls in an epoch, and a planned visit with no collection date has a visit number but no epoch.
Inputs
Under review4 source filesinput/dm.csv
Edit3 rows| STUDYID | USUBJID | RFSTDTC |
|---|---|---|
| CATH | 2025-01-08 | |
| CATH | 2025-02-08 | |
| CATH | 2025-03-01 |
input/se.csv
Edit9 rows| STUDYID | USUBJID | EPOCH | SESTDTC | SEENDTC |
|---|---|---|---|---|
| CATH | SCREENING | 2025-01-01 | 2025-01-07 | |
| CATH | TREATMENT | 2025-01-08 | 2025-03-04 | |
| CATH | FOLLOW-UP | 2025-03-05 | 2025-12-31 | |
| CATH | SCREENING | 2025-02-05 | 2025-02-08 | |
| CATH | TREATMENT | 2025-02-08 | 2025-04-05 | |
| CATH | FOLLOW-UP | 2025-04-06 | 2025-12-31 | |
| CATH | SCREENING | 2025-02-20 | 2025-02-28 | |
| CATH | TREATMENT | 2025-03-01 | 2025-04-26 | |
| CATH | FOLLOW-UP | 2025-04-27 | 2025-12-31 |
input/tv.csv
Edit3 rows| VISIT | VISITNUM |
|---|---|
| SCREENING | 1 |
| BASELINE | 2 |
| WEEK 4 | 3 |
input/vs_raw.csv
Edit7 rows| STUDYID | USUBJID | VSSEQ | VISIT | VSTESTCD | VSORRES | VSDTC |
|---|---|---|---|---|---|---|
| CATH | 1 | SCREENING | SYSBP | 128 | 2025-01-02 | |
| CATH | 2 | BASELINE | SYSBP | 132 | 2025-01-08 | |
| CATH | 3 | UNSCHEDULED | SYSBP | 145 | 2025-01-15 | |
| CATH | 4 | WEEK 4 | SYSBP | 130 | 2025-02-05 | |
| CATH | 1 | BASELINE | SYSBP | 118 | 2025-02-08 | |
| CATH | 2 | WEEK 4 | SYSBP | 122 | — | |
| CATH | 1 | BASELINE | SYSBP | 124 | 2025-03-01 |
Expected output
Under review7 expected rowsexpected/vs.csv
Edit7 rows| DOMAIN | STUDYID | USUBJID | VSSEQ | VSTESTCD | VSORRES | VSDTC | VISIT | VISITNUM | EPOCH | VSDY |
|---|---|---|---|---|---|---|---|---|---|---|
| VS | CATH | 1 | SYSBP | 128 | 2025-01-02 | SCREENING | 1 | SCREENING | -6 | |
| VS | CATH | 2 | SYSBP | 132 | 2025-01-08 | BASELINE | 2 | TREATMENT | 1 | |
| VS | CATH | 3 | SYSBP | 145 | 2025-01-15 | UNSCHEDULED | 2.01 | TREATMENT | 8 | |
| VS | CATH | 4 | SYSBP | 130 | 2025-02-05 | WEEK 4 | 3 | TREATMENT | 29 | |
| VS | CATH | 1 | SYSBP | 118 | 2025-02-08 | BASELINE | 2 | TREATMENT | 1 | |
| VS | CATH | 2 | SYSBP | 122 | — | WEEK 4 | 3 | — | — | |
| VS | CATH | 1 | SYSBP | 124 | 2025-03-01 | BASELINE | 2 | TREATMENT | 1 |
Human review and assessment
| Variable Name | Variable Label | Type | Length | Controlled Terms or Format | Origin | Core | Conversion Definition | Variable Type | Variable Order | Comments for Define |
|---|---|---|---|---|---|---|---|---|---|---|
| DOMAIN | Domain Abbreviation | Char | Assigned | Set constant value "VS". | SDTM | 1 | ||||
| STUDYID | Study Identifier | Char | Collected | Copy value from VS_RAW.STUDYID. | SDTM | 2 | ||||
| USUBJID | Unique Subject Identifier | Char | Collected | Copy value from VS_RAW.USUBJID. | SDTM | 3 | ||||
| VSSEQ | Sequence Number | Num | Collected | Copy value from VS_RAW.VSSEQ. | SDTM | 4 | ||||
| VSTESTCD | Vital Signs Test Short Name | Char | Collected | Copy value from VS_RAW.VSTESTCD. | SDTM | 5 | ||||
| VSORRES | Result or Finding in Original Units | Char | Collected | Copy value from VS_RAW.VSORRES. | SDTM | 6 | ||||
| VSDTC | Date/Time of Measurements | Char | ISO 8601 | Collected | Copy value from VS_RAW.VSDTC. | SDTM | 7 | |||
| VISIT | Visit Name | Char | Collected | Copy value from VS_RAW.VISIT. | SDTM | 8 | ||||
| VISITNUM | Visit Number | Num | Derived | First non-missing of PLANNEDNUM and SPONSORNUM. | SDTM | 9 | ||||
| VSDY | Study Day of Vital Signs | Num | Derived | CDISC study day of VSDTC against RFSTDTC as day 1; there is no day zero. | SDTM | 10 | ||||
| EPOCH | Epoch | Char | SCREENING, TREATMENT, FOLLOW-UP | Derived | Take EPOCH from the last SE record matching on USUBJID, where VSDTC falls within SESTDTC to SEENDTC, ordered by SE.SESTDTC, and no match yields blank. | SDTM | 11 |
| Variable | Permitted values |
|---|---|
| EPOCH | SCREENING, TREATMENT, FOLLOW-UP |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.7 | 2026-10-20 | yamaa docs build | Read a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default | ||
| 1.6 | 2026-10-17 | yamaa docs build | Describe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight | ||
| 1.5 | 2026-10-17 | yamaa docs build | Name collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part | ||
| 1.4 | 2026-10-15 | yamaa docs build | State datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks | ||
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/sdtm-vs-study-day/full.md
Edit29 linesFollowing CDISC SDTM standards, use the provided DM, SE, TV, and VS_RAW datasets to create a VS dataset with one record per collected vital-signs result.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, VSSEQ, VSTESTCD, VSORRES, VSDTC, VISIT, VISITNUM, EPOCH, VSDY
DOMAIN is "VS". VSSEQ is the sequence number as collected. VISITNUM is the planned visit number in TV for the record's visit label. An unplanned visit label that names no planned visit in TV, such as "UNSCHEDULED", takes the sponsor-assigned number: the latest planned visit number collected for the subject before the collection date, plus 0.01, so an unscheduled visit after visit 2 reads 2.01. EPOCH is the epoch of the subject's element in SE whose start and end dates, both ends included, contain the collection date: "SCREENING", "TREATMENT", or "FOLLOW-UP". A day shared by two elements belongs to the later element. EPOCH has no value when the record has no collection date or when no element contains the collection date. VSDY is the study day of the collection date, counted from the subject's reference start date in DM. VSDY has no value when the record has no collection date or the subject has no reference start date in DM. The visit number follows the visit label while the epoch follows the collection date, so an unscheduled visit still falls in an epoch, and a planned visit with no collection date has a visit number but no epoch.
Read the source datasets from /app/input and save the completed dataset as /app/output/vs.csv.
prompts/sdtm-vs-study-day/conventions.md
Edit14 linesFollowing CDISC SDTM standards, use the provided DM, SE, TV, and VS_RAW datasets to create a VS dataset with one record per collected vital-signs result.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, VSSEQ, VSTESTCD, VSORRES, VSDTC, VISIT, VISITNUM, EPOCH, VSDY
DOMAIN is "VS". EPOCH is "SCREENING", "TREATMENT", or "FOLLOW-UP", or has no value. VSDY is the study day of the collection date, counted from the subject's reference start date in DM.
Read the source datasets from /app/input and save the completed dataset as /app/output/vs.csv.
prompts/sdtm-vs-study-day/brief.md
Edit10 linesFollowing CDISC SDTM standards, use the provided DM, SE, TV, and VS_RAW datasets to create a VS dataset with one record per collected vital-signs result.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, VSSEQ, VSTESTCD, VSORRES, VSDTC, VISIT, VISITNUM, EPOCH, VSDY
Read the source datasets from /app/input and save the completed dataset as /app/output/vs.csv.
R
Edit3 lineslibrary(yamaa)yamaa_spec("spec.yaml")$run()$outputPython
Edit3 linesimport yamaayamaa.spec("spec.yaml").run().outputsolutions/sdtm-vs-study-day/result.R
Edit100 lines# Reference solution for the yamaa benchmark sdtm-vs-study-day (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(lubridate)library(purrr)library(readr)study_day <- function(ref, coll) { if (is.na(ref) || is.na(coll)) { return(NA_integer_) } delta <- as.integer(coll - ref) if (delta >= 0L) delta + 1L else delta}dm <- read_csv( "/app/input/dm.csv", col_types = cols(.default = col_character()))se <- read_csv( "/app/input/se.csv", col_types = cols(.default = col_character()))tv <- read_csv( "/app/input/tv.csv", col_types = cols(.default = col_character()))vs_raw <- read_csv( "/app/input/vs_raw.csv", col_types = cols(VSSEQ = col_integer(), .default = col_character()))tv_map <- setNames(suppressWarnings(as.numeric(tv$VISITNUM)), tv$VISIT)ref_map <- setNames(suppressWarnings(ymd(dm$RFSTDTC)), dm$USUBJID)vs <- vs_raw |> mutate( coll = suppressWarnings(ymd(VSDTC)), ref = ref_map[USUBJID], VSDY = map2_int(ref, coll, study_day, .progress = FALSE), DOMAIN = "VS" )vs$VISITNUM <- pmap_dbl( list(vs$USUBJID, vs$VISIT, vs$VSDTC), function(usubjid, visit, vsdtc) { if (visit %in% names(tv_map)) { return(tv_map[[visit]]) } day <- suppressWarnings(ymd(vsdtc)) if (is.na(day)) { return(NA_real_) } prior <- vs_raw |> filter( USUBJID == usubjid, VISIT %in% names(tv_map), !is.na(suppressWarnings(ymd(VSDTC))), suppressWarnings(ymd(VSDTC)) < day ) if (nrow(prior) == 0) { return(NA_real_) } max(tv_map[prior$VISIT], na.rm = TRUE) + 0.01 }, .progress = FALSE)vs$EPOCH <- pmap_chr( list(vs$USUBJID, vs$VSDTC), function(usubjid, vsdtc) { day <- suppressWarnings(ymd(vsdtc)) if (is.na(day)) { return(NA_character_) } elements <- se |> filter(USUBJID == usubjid) hits <- elements |> filter( suppressWarnings(ymd(SESTDTC)) <= day, day <= suppressWarnings(ymd(SEENDTC)) ) |> arrange(suppressWarnings(ymd(SESTDTC))) if (nrow(hits) == 0) { return(NA_character_) } hits$EPOCH[nrow(hits)] }, .progress = FALSE)vs <- vs |> arrange(USUBJID, VSSEQ) |> select( DOMAIN, STUDYID, USUBJID, VSSEQ, VSTESTCD, VSORRES, VSDTC, VISIT, VISITNUM, EPOCH, VSDY )dir.create("/app/output", showWarnings = FALSE)write_csv(vs, "/app/output/vs.csv", na = "")solutions/sdtm-vs-study-day/result.py
Edit120 lines# Reference solution for the yamaa benchmark sdtm-vs-study-day (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.import csvfrom datetime import dateINPUT_DIR = "/app/input"OUTPUT_DIR = "/app/output"def read_csv(path): with open(path, newline="", encoding="utf-8") as f: return list(csv.DictReader(f))def parse_date(value): value = (value or "").strip() if not value: return None return date.fromisoformat(value)def format_number(value): if value is None: return "" if float(value).is_integer(): return str(int(value)) return f"{value:.2f}".rstrip("0").rstrip(".")def main(): dm = read_csv(f"{INPUT_DIR}/dm.csv") se = read_csv(f"{INPUT_DIR}/se.csv") tv = read_csv(f"{INPUT_DIR}/tv.csv") vs_raw = read_csv(f"{INPUT_DIR}/vs_raw.csv") ref_start = {} for row in dm: ref_start[row["USUBJID"]] = parse_date(row["RFSTDTC"]) elements = {} for row in se: elements.setdefault(row["USUBJID"], []).append( (row["EPOCH"], parse_date(row["SESTDTC"]), parse_date(row["SEENDTC"])) ) planned = {} for row in tv: planned[row["VISIT"]] = float(row["VISITNUM"]) def unplanned_visitnum(subject, coll_date): # Latest planned visit number collected for the subject before # the record's collection date, plus 0.01. if coll_date is None: return None best = None for row in vs_raw: if row["USUBJID"] != subject or row["VISIT"] not in planned: continue other_date = parse_date(row["VSDTC"]) if other_date is not None and other_date < coll_date: num = planned[row["VISIT"]] if best is None or num > best: best = num return None if best is None else best + 0.01 out_rows = [] for row in vs_raw: subject = row["USUBJID"] coll_date = parse_date(row["VSDTC"]) if row["VISIT"] in planned: visitnum = planned[row["VISIT"]] else: visitnum = unplanned_visitnum(subject, coll_date) epoch = "" if coll_date is not None: chosen = None for name, start, end in elements.get(subject, []): if start is not None and end is not None and start <= coll_date <= end: # A day shared by two elements belongs to the later one. if chosen is None or start >= chosen[1]: chosen = (name, start) if chosen is not None: epoch = chosen[0] vsdy = "" ref = ref_start.get(subject) if coll_date is not None and ref is not None: diff = (coll_date - ref).days vsdy = str(diff + 1) if diff >= 0 else str(diff) out_rows.append({ "DOMAIN": "VS", "STUDYID": row["STUDYID"], "USUBJID": subject, "VSSEQ": row["VSSEQ"], "VSTESTCD": row["VSTESTCD"], "VSORRES": row["VSORRES"], "VSDTC": row["VSDTC"], "VISIT": row["VISIT"], "VISITNUM": format_number(visitnum), "EPOCH": epoch, "VSDY": vsdy, }) columns = ["DOMAIN", "STUDYID", "USUBJID", "VSSEQ", "VSTESTCD", "VSORRES", "VSDTC", "VISIT", "VISITNUM", "EPOCH", "VSDY"] with open(f"{OUTPUT_DIR}/vs.csv", "w", newline="", encoding="utf-8") as f: writer = csv.DictWriter(f, fieldnames=columns) writer.writeheader() writer.writerows(out_rows)if __name__ == "__main__": main()solutions/sdtm-vs-study-day/result.sas
Edit39 lines/* opensas reference solution for the yamaa benchmark sdtm-vs-study-day. Reads the benchmark inputs and derives the requested output. Execute with the existing opensas runtime: opensas /app/output/result.sas. */data dm; length STUDYID USUBJID RFSTDTC $1024; infile "/app/input/dm.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. RFSTDTC : $char1024.;run;data se; length STUDYID USUBJID EPOCH SESTDTC SEENDTC $1024; infile "/app/input/se.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. EPOCH : $char1024. SESTDTC : $char1024. SEENDTC : $char1024.;run;data tv; length VISIT $1024; infile "/app/input/tv.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input VISIT : $char1024. VISITNUM;run;data vs_raw; length STUDYID USUBJID VISIT VSTESTCD VSORRES VSDTC $1024; infile "/app/input/vs_raw.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. VSSEQ VISIT : $char1024. VSTESTCD : $char1024. VSORRES : $char1024. VSDTC : $char1024.;run;proc sql; create table collected as select a.*,b.VISITNUM,c.RFSTDTC from vs_raw a left join tv b on a.VISIT=b.VISIT left join dm c on a.STUDYID=c.STUDYID and a.USUBJID=c.USUBJID; quit;data dated; set collected; NODATE=missing(VSDTC); run;proc sort data=dated; by STUDYID USUBJID NODATE VSDTC VSSEQ; run;data numbered; set dated; by STUDYID USUBJID; retain PRIORNUM; if first.USUBJID then PRIORNUM=.; if not missing(VISITNUM) and not missing(VSDTC) then PRIORNUM=VISITNUM; else if missing(VISITNUM) and not missing(VSDTC) and not missing(PRIORNUM) then VISITNUM=PRIORNUM+0.01; run;proc sql; create table candidates as select a.*,b.EPOCH,b.SESTDTC from numbered a left join se b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID and a.VSDTC>=b.SESTDTC and a.VSDTC<=b.SEENDTC and not missing(a.VSDTC); quit;proc sort data=candidates; by STUDYID USUBJID VSSEQ descending SESTDTC; run;data result; length DOMAIN $2; set candidates; by STUDYID USUBJID VSSEQ; if first.VSSEQ; DOMAIN='VS'; VSDY=.; if lengthn(VSDTC)>=10 and lengthn(RFSTDTC)>=10 then do; VSDY=input(substr(VSDTC,1,10),yymmdd10.)-input(substr(RFSTDTC,1,10),yymmdd10.); if VSDY>=0 then VSDY+1; end; run;proc sql; create table final as select DOMAIN, STUDYID, USUBJID, VSSEQ, VSTESTCD, VSORRES, VSDTC, VISIT, VISITNUM, EPOCH, VSDY from result;quit;proc export data=final outfile="/app/output/vs.csv" dbms=csv replace; run;
yamaa