Review ask

As a clinical trial statistician or programmer, challenge anything in this benchmark: describe a problem in Comments, or propose a fix with the Edit button on any source file.

  1. Task: Would a real study need this derivation, specified this way? Does it follow CDISC SDTM: variable names, controlled terms, record structure? Flag any rule that is ambiguous or that a sponsor would handle differently.
  2. Inputs: Do they look like data collected in a real trial, and do they include the situations you would expect to meet in one?
  3. Expected output: Would independent QC programming reproduce every value from these inputs and rules?

The other tabs beside Comments are background for assessment, not under review.

Summary

Lifecycle: reviewedEditUnder review
What this benchmark means
README.md

Goal: turn every collected vital-signs measurement into one record that keeps the collected test, result, and unit, and adds the result in the test's standard unit (as a number and as text) plus a completion status: VSTESTCD, VSTEST, VSORRES, VSORRESU, VSSTRESN, VSSTRESC, VSSTRESU, and VSSTAT.

Input: collected vital-signs rows, each with a test code and name, the result as collected with its unit, and study, subject, sequence, and visit identifiers.

Variables:

  • VSTESTCD is the test code as collected: HEIGHT, WEIGHT, or TEMP.
  • VSTEST is the test name as collected: Height, Weight, or Temperature.
  • VSORRES is the collected result written back as text; reading it as a number drops trailing zeros (a collected 72.50 reads 72.5), but the value itself is never converted.
  • VSORRESU is the unit exactly as collected: cm, kg, C, LB, or F. Blank when no result was collected.
  • VSSTRESN is the result as a number in the test's standard unit: height passes through in cm; weight in LB is multiplied by 0.45359237 to reach kg; temperature in F becomes (VSORRES - 32) * 5 / 9 to reach C; a result already in the standard unit passes through unchanged; a result never collected stays missing.
  • VSSTRESC is the same standardized value written as text, so it always agrees with the numeric result. Blank when no result was collected.
  • VSSTRESU is the test's standard unit: cm for height, kg for weight, C for temperature. A unit that does not belong to the test (a weight in cm) stops the run instead of being carried through as the standard unit. Blank when no result was collected.
  • VSSTAT is NOT DONE when no result was collected, blank otherwise.

Note: records are grouped by test rather than kept in collection order: all heights first, then all weights, then all temperatures, each in collection order. A record whose result was never collected carries VSSTAT NOT DONE, no original unit, and no standardized value.

Inputs

Under review
1 source file

input/vs_raw.csv

Edit
10 rows
input/vs_raw.csv
STUDYIDUSUBJIDVSSEQVISITVSTESTCDVSTESTVSORRESVSORRESU
YAMAA-011SCREENINGHEIGHTHeight168.2cm
YAMAA-012SCREENINGWEIGHTWeight72.5kg
YAMAA-013SCREENINGTEMPTemperature36.8C
YAMAA-014WEEK 2TEMPTemperature98.6F
YAMAA-015WEEK 2TEMPTemperature37.0C
YAMAA-011SCREENINGWEIGHTWeight175LB
YAMAA-012SCREENINGTEMPTemperature100.4F
YAMAA-013WEEK 2TEMPTemperature—F
YAMAA-014WEEK 2HEIGHTHeight165.5cm
YAMAA-011BASELINEHEIGHTHeight172cm

Expected output

Under review
10 expected rows

expected/vs.csv

Edit
10 rows
expected/vs.csv
DOMAINSTUDYIDUSUBJIDVSSEQVISITVSTESTCDVSTESTVSORRESVSORRESUVSSTRESNVSSTRESCVSSTRESUVSSTAT
VSYAMAA-011SCREENINGHEIGHTHeight168.2cm168.2168.2cm—
VSYAMAA-014WEEK 2HEIGHTHeight165.5cm165.5165.5cm—
VSYAMAA-011BASELINEHEIGHTHeight172cm172172cm—
VSYAMAA-012SCREENINGWEIGHTWeight72.5kg72.572.5kg—
VSYAMAA-011SCREENINGWEIGHTWeight175LB79.3786647579.37866475kg—
VSYAMAA-013SCREENINGTEMPTemperature36.8C36.836.8C—
VSYAMAA-014WEEK 2TEMPTemperature98.6F3737C—
VSYAMAA-015WEEK 2TEMPTemperature37C3737C—
VSYAMAA-012SCREENINGTEMPTemperature100.4F3838C—
VSYAMAA-013WEEK 2TEMPTemperature—————NOT DONE
Shaded columns are derived— = missing value

Comments and assessment materials

Comments

Signed in with GitHub; saved in repository Discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Variable NameVariable LabelTypeLengthControlled Terms or FormatOriginCoreConversion DefinitionVariable TypeVariable OrderComments for Define
DOMAINDomain AbbreviationCharAssignedSet constant value "VS".SDTM1
STUDYIDStudy IdentifierCharCollectedCopy value from VS_RAW.STUDYID.SDTM2
USUBJIDUnique Subject IdentifierCharCollectedCopy value from VS_RAW.USUBJID.SDTM3
VSSEQSequence NumberNumCollectedCopy value from VS_RAW.VSSEQ.SDTM4
VISITVisit NameCharCollectedCopy value from VS_RAW.VISIT.SDTM5
VSTESTCDVital Signs Test Short NameCharHEIGHT, WEIGHT, TEMPCollectedCopy value from VS_RAW.VSTESTCD.SDTM6
VSTESTVital Signs Test NameCharCollectedCopy value from VS_RAW.VSTEST.SDTM7
VSORRESResult or Finding in Original UnitsCharCollectedCopy value from VS_RAW.VSORRES.SDTM8
VSORRESUOriginal UnitsCharcm, kg, C, LB, FDerivedIf VS_RAW.VSORRES IS NOT NULL then "VS_RAW.VSORRESU"; otherwise blank.SDTM9
VSSTRESNNumeric Result/Finding in Standard UnitsNumDerivedSet per constructed row; see the Row construction sheet.SDTM10
VSSTRESCCharacter Result/Finding in Std FormatCharDerivedCopy value from VSSTRESN.SDTM11
VSSTRESUStandard UnitsCharcm, kg, CDerivedSet per constructed row; see the Row construction sheet.SDTM12
VSSTATCompletion StatusCharDerivedIf VS_RAW.VSORRES IS NULL then "NOT DONE"; otherwise blank.SDTM13
TemplateInclude whenVSSTRESNVSSTRESU
heightVS_RAW.VSTESTCD = 'HEIGHT'If VS_RAW.VSORRESU = 'cm' then "VS_RAW.VSORRES"; otherwise blankIf VS_RAW.VSORRES IS NOT NULL AND VS_RAW.VSORRESU = 'cm' then "cm"; otherwise blank
weightVS_RAW.VSTESTCD = 'WEIGHT'If VS_RAW.VSORRESU = 'LB' then compute VS_RAW.VSORRES * 0.45359237; If VS_RAW.VSORRESU = 'kg' then "VS_RAW.VSORRES"; otherwise blankIf VS_RAW.VSORRES IS NOT NULL AND (VS_RAW.VSORRESU = 'kg' OR VS_RAW.VSORRESU = 'LB') then "kg"; otherwise blank
tempVS_RAW.VSTESTCD = 'TEMP'If VS_RAW.VSORRESU = 'F' then compute (VS_RAW.VSORRES - 32) * 5 / 9; If VS_RAW.VSORRESU = 'C' then "VS_RAW.VSORRES"; otherwise blankIf VS_RAW.VSORRES IS NOT NULL AND (VS_RAW.VSORRESU = 'C' OR VS_RAW.VSORRESU = 'F') then "C"; otherwise blank
VariablePermitted values
VSTESTCDHEIGHT, WEIGHT, TEMP
VSORRESUcm, kg, C, LB, F
VSSTRESUcm, kg, C
VersionDateAuthorDescriptionReviewerSign-off
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/sdtm-vs-units/full.md

Edit
33 lines

Following CDISC SDTM standards, use the provided VS_RAW dataset to create a VS dataset with one record per collected vital-signs measurement.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, VSSEQ, VISIT, VSTESTCD, VSTEST, VSORRES, VSORRESU, VSSTRESN, VSSTRESC, VSSTRESU, VSSTAT

VSSEQ is the sequence number as collected. VSTESTCD is the test code as collected: HEIGHT, WEIGHT, or TEMP. VSTEST is the test name as collected: Height, Weight, or Temperature.

VSORRES is the collected result read as a number and written back as text, so trailing zeros after the decimal point are dropped and a whole number has no decimal point; the value itself is never converted. VSORRESU is the unit exactly as collected: cm, kg, C, LB, or F, and has no value when no result was collected. VSSTRESN is the result as a number in the test's standard unit, not rounded: height passes through in cm, weight collected in LB is multiplied by 0.45359237 to give kg, and temperature collected in F becomes (F - 32) * 5 / 9 in C; a result already in the standard unit passes through unchanged, and a result never collected has no value. VSSTRESC is the same standardized value written as text with at most 15 significant digits and no trailing zeros, so a whole number has no decimal point; it has no value when no result was collected. VSSTRESU is the test's standard unit: cm for height, kg for weight, C for temperature; it has no value when no result was collected. VSSTAT is NOT DONE when no result was collected, and has no value otherwise.

A record whose result was never collected carries VSSTAT NOT DONE, no original unit, and no standardized value.

Read the source datasets from /app/input and save the completed dataset as /app/output/vs.csv.

prompts/sdtm-vs-units/conventions.md

Edit
19 lines

Following CDISC SDTM standards, use the provided VS_RAW dataset to create a VS dataset with one record per collected vital-signs measurement.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, VSSEQ, VISIT, VSTESTCD, VSTEST, VSORRES, VSORRESU, VSSTRESN, VSSTRESC, VSSTRESU, VSSTAT

VSTESTCD is HEIGHT, WEIGHT, or TEMP. VSTEST is Height, Weight, or Temperature.

VSORRES is the collected result read as a number and written back as text, so trailing zeros after the decimal point are dropped and a whole number has no decimal point. VSSTRESN is not rounded. VSSTRESC is the same standardized value written as text with at most 15 significant digits and no trailing zeros, so a whole number has no decimal point. VSSTAT is NOT DONE or has no value.

Read the source datasets from /app/input and save the completed dataset as /app/output/vs.csv.

prompts/sdtm-vs-units/brief.md

Edit
9 lines

Following CDISC SDTM standards, use the provided VS_RAW dataset to create a VS dataset with one record per collected vital-signs measurement.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, VSSEQ, VISIT, VSTESTCD, VSTEST, VSORRES, VSORRESU, VSSTRESN, VSSTRESC, VSSTRESU, VSSTAT

Read the source datasets from /app/input and save the completed dataset as /app/output/vs.csv.

yamaa code

Runs the yamaa specification; its result must match the expected output

run.R

Edit
4 lines
library(yamaa)vs <- yamaa_domain("spec.yaml")$outputvs

run.py

Edit
4 lines
import yamaavs = yamaa.yamaa_domain("spec.yaml").outputvs

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/sdtm-vs-units/result.R

Edit
73 lines
# Reference solution for the yamaa benchmark sdtm-vs-units (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)fmt_text <- function(x) {  if (is.na(x)) {    return(NA_character_)  }  if (x == floor(x)) {    formatC(x, format = "f", digits = 0)  } else {    format(x, scientific = FALSE, trim = TRUE, digits = 15)  }}fmt_orres <- function(x) {  if (is.na(x) || x == "") {    return(NA_character_)  }  num <- suppressWarnings(as.numeric(x))  if (is.na(num)) {    return(x)  }  fmt_text(num)}raw <- read_csv(  "/app/input/vs_raw.csv",  col_types = cols(VSSEQ = col_integer(), .default = col_character()))vs <- raw |>  mutate(    has_result = !is.na(VSORRES) & VSORRES != "",    VSORRES = vapply(VSORRES, fmt_orres, character(1)),    value = suppressWarnings(as.numeric(raw$VSORRES)),    VSSTRESN = case_when(      !has_result ~ NA_real_,      VSTESTCD == "HEIGHT" ~ value,      VSTESTCD == "WEIGHT" & VSORRESU == "LB" ~ value * 0.45359237,      VSTESTCD == "WEIGHT" ~ value,      VSORRESU == "F" ~ (value - 32) * 5 / 9,      .default = value    ),    VSSTRESC = vapply(VSSTRESN, fmt_text, character(1)),    VSSTRESU = case_when(      !has_result ~ NA_character_,      VSTESTCD == "HEIGHT" ~ "cm",      VSTESTCD == "WEIGHT" ~ "kg",      .default = "C"    ),    VSORRESU = if_else(has_result, VSORRESU, NA_character_),    VSSTAT = if_else(has_result, NA_character_, "NOT DONE"),    DOMAIN = "VS",    test_ord = case_when(      VSTESTCD == "HEIGHT" ~ 1L,      VSTESTCD == "WEIGHT" ~ 2L,      .default = 3L    )  ) |>  arrange(test_ord, USUBJID, VSSEQ) |>  select(    DOMAIN, STUDYID, USUBJID, VSSEQ, VISIT, VSTESTCD, VSTEST, VSORRES,    VSORRESU, VSSTRESN, VSSTRESC, VSSTRESU, VSSTAT  )dir.create("/app/output", showWarnings = FALSE)write_csv(vs, "/app/output/vs.csv", na = "")

solutions/sdtm-vs-units/result.py

Edit
101 lines
# Reference solution for the yamaa benchmark sdtm-vs-units (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.from pathlib import Pathimport polars as pldef fmt_text(x: float | None) -> str | None:    if x is None:        return None    import math    if isinstance(x, float) and (math.isnan(x) or math.isinf(x)):        return None    fx = float(x)    if fx.is_integer():        return str(int(fx))    text = f"{fx:.15g}"    return textdef fmt_orres(x: str | None) -> str | None:    if x is None or x == "":        return None    try:        return fmt_text(float(x))    except ValueError:        return xraw = pl.read_csv("/app/input/vs_raw.csv", infer_schema=False).with_columns(    pl.col("VSSEQ").cast(pl.Int64),)rows = raw.to_dicts()for row in rows:    orres = row["VSORRES"]    has_result = orres is not None and orres != ""    test = row["VSTESTCD"]    unit = row["VSORRESU"]    if not has_result:        row["VSORRES"] = None        row["VSORRESU"] = None        row["VSSTRESN"] = None        row["VSSTRESC"] = None        row["VSSTRESU"] = None        row["VSSTAT"] = "NOT DONE"        continue    row["VSORRES"] = fmt_orres(orres)    value = float(orres)    if test == "HEIGHT":        std = value        std_unit = "cm"    elif test == "WEIGHT":        std = value * 0.45359237 if unit == "LB" else value        std_unit = "kg"    else:        std = (value - 32) * 5 / 9 if unit == "F" else value        std_unit = "C"    row["VSSTRESN"] = std    row["VSSTRESC"] = fmt_text(std)    row["VSSTRESU"] = std_unit    # Original unit is empty when no result was collected; otherwise as    # collected.    row["VSORRESU"] = unit    row["VSSTAT"] = Nonevs = (    pl.DataFrame(rows)    .with_columns(        DOMAIN=pl.lit("VS"),        _ord=pl.col("VSTESTCD").replace_strict(            {"HEIGHT": 1, "WEIGHT": 2, "TEMP": 3},            default=9,            return_dtype=pl.Int64,        ),    )    .sort(["_ord", "USUBJID", "VSSEQ"])    .select(        "DOMAIN",        "STUDYID",        "USUBJID",        "VSSEQ",        "VISIT",        "VSTESTCD",        "VSTEST",        "VSORRES",        "VSORRESU",        "VSSTRESN",        "VSSTRESC",        "VSSTRESU",        "VSSTAT",    ))Path("/app/output").mkdir(exist_ok=True)vs.write_csv("/app/output/vs.csv")

solutions/sdtm-vs-units/result.sas

Edit
31 lines
/* opensas reference solution for the yamaa benchmark sdtm-vs-units.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data vs_raw;  length STUDYID USUBJID VISIT VSTESTCD VSTEST VSORRES VSORRESU $1024;  infile "/app/input/vs_raw.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. VSSEQ VISIT : $char1024. VSTESTCD : $char1024. VSTEST : $char1024. VSORRES : $char1024. VSORRESU : $char1024.;run;data result; set vs_raw; length DOMAIN $2 VSSTRESC VSSTRESU VSSTAT $1024; DOMAIN='VS'; VSSTRESN=.; VSSTRESC=''; VSSTRESU=''; VSSTAT='';if missing(VSORRES) then do; VSSTAT='NOT DONE'; VSORRESU=''; end;else do; VALUE=input(VSORRES,best32.); VSORRES=strip(put(VALUE,best15.)); VSSTRESN=VALUE;if VSTESTCD='HEIGHT' then VSSTRESU='cm'; else if VSTESTCD='WEIGHT' then do; VSSTRESU='kg'; if VSORRESU='LB' then VSSTRESN=VALUE*0.45359237; end;else do; VSSTRESU='C'; if VSORRESU='F' then VSSTRESN=(VALUE-32)*5/9; end;/* Fifteen significant digits; field width alone counts the decimal point.   Explicit decimal precision avoids BEST32 exposing binary rounding noise. */if VSSTRESN=0 then VSSTRESC='0';else do;  DECIMALS=max(0,14-floor(log10(abs(VSSTRESN))));  VSSTRESC=strip(putn(VSSTRESN,cats('32.',put(DECIMALS,best12.))));  if index(VSSTRESC,'.')>0 then do;    VSSTRESC=prxchange('s/0+$//',1,VSSTRESC);    VSSTRESC=prxchange('s/\.$//',1,VSSTRESC);  end;end;end; run;proc sql;  create table final as select DOMAIN, STUDYID, USUBJID, VSSEQ, VISIT, VSTESTCD, VSTEST, VSORRES, VSORRESU, VSSTRESN, VSSTRESC, VSSTRESU, VSSTAT from result;quit;proc export data=final outfile="/app/output/vs.csv" dbms=csv replace; run;