Review ask

As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.

We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.

Summary

Lifecycle: draftEditUnder review
What this benchmark means
README.md

Public entries: Python and R. Passing coverage requires an installed inventory.

Goal: write the supplemental records MHTERM1 and MHTERM2 that continue a reported medical history term longer than 200 characters, carrying IDVARVAL, QLABEL, QVAL, QORIG, and QEVAL.

Input: a medical history extract with one record per condition, carrying the parent record sequence and the reported term as collected.

Variables:

  • IDVARVAL is the parent record sequence, written as text.
  • QNAM is MHTERM1 for the first continuation and MHTERM2 for the second.
  • QLABEL is Reported Term for the Medical History 1 for MHTERM1 and Reported Term for the Medical History 2 for MHTERM2.
  • QVAL is the piece of the reported term that the record continues.
  • QORIG records the collected origin as Collected.
  • QEVAL is blank, since a collected value is not an assessment.

Note: the reported term is split into pieces of at most 200 characters, each ending after the last whole word that fits, so no word is broken across two pieces; the space at each break belongs to neither piece. The first piece stays on the medical history record itself. A term of 200 characters or fewer has no supplemental record, a term with two pieces has MHTERM1 only, and a term with three has both; a term that needs a fourth piece stops the run instead of losing its end.

Inputs

Under review
1 source file

input/mh_raw.csv

Edit
5 rows
input/mh_raw.csv
STUDYIDUSUBJIDMHSEQMHTERM
YAMAA-011Seasonal allergic rhinitis
YAMAA-012Recurrent migraine without aura since adolescence, with attacks two to three times a month lasting up to two days, accompanied by nausea and sensitivity to light, partially relieved by triptans and rest in a dark room
YAMAA-011Type 2 diabetes mellitus diagnosed at a routine employment screening, initially managed with diet and exercise alone, later treated with metformin and then with the addition of a sulfonylurea after glycated haemoglobin rose above target on two consecutive annual reviews; complicated by mild background retinopathy found on the first retinal screening and by peripheral sensory neuropathy affecting both feet, with no history of foot ulceration or hospital admission for hypoglycaemia
YAMAA-012Gastro-oesophageal reflux disease with heartburn after large meals and when lying flat, improved with a proton pump inhibitor taken each morning, endoscopy two years ago showing mild oesophagitis only
YAMAA-011Osteoarthritis of both knees, worse on the right, with morning stiffness of less than thirty minutes and pain on stairs, managed with paracetamol, topical diclofenac and a home exercise programme from physio

Expected output

Under review
4 expected rows

expected/suppmh.csv

Edit
4 rows
expected/suppmh.csv
STUDYIDRDOMAINUSUBJIDIDVARIDVARVALQNAMQLABELQVALQORIGQEVAL
YAMAA-01MHMHSEQ2MHTERM1Reported Term for the Medical History 1rest in a dark roomCollected—
YAMAA-01MHMHSEQ1MHTERM1Reported Term for the Medical History 1glycated haemoglobin rose above target on two consecutive annual reviews; complicated by mild background retinopathy found on the first retinal screening and by peripheral sensory neuropathy affectingCollected—
YAMAA-01MHMHSEQ1MHTERM2Reported Term for the Medical History 2both feet, with no history of foot ulceration or hospital admission for hypoglycaemiaCollected—
YAMAA-01MHMHSEQ1MHTERM1Reported Term for the Medical History 1physioCollected—
Shaded columns are derived— = missing value

Human review and assessment

Your review

3 questions and your comments · View benchmark discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Variable NameVariable LabelTypeLengthControlled Terms or FormatOriginCoreConversion DefinitionVariable TypeVariable OrderComments for Define
STUDYIDStudy IdentifierCharCollectedCopy value from MH_RAW.STUDYID.SUPP1
RDOMAINRelated Domain AbbreviationCharAssignedSet constant value "MH".SUPP2
USUBJIDUnique Subject IdentifierCharCollectedCopy value from MH_RAW.USUBJID.SUPP3
IDVARIdentifying VariableCharAssignedSet constant value "MHSEQ".SUPP4
IDVARVALIdentifying Variable ValueCharCollectedCopy value from MH_RAW.MHSEQ.SUPP5
QNAMQualifier Variable NameCharMHTERM1, MHTERM2DerivedSet per constructed row; see the Row construction sheet.SUPP6
QLABELQualifier Variable LabelCharDerivedSet per constructed row; see the Row construction sheet.SUPP7
QVALData ValueCharDerivedSet per constructed row; see the Row construction sheet.SUPP8
QORIGOriginCharAssignedSet constant value "Collected".SUPP9
QEVALEvaluatorCharAssignedSet constant value "None".SUPP10
TemplateInclude whenQNAMQLABELQVAL
second_pieceTERM2 IS NOT NULLConstant "MHTERM1"Constant "Reported Term for the Medical History 1"Copy TERM2
third_pieceTERM3 IS NOT NULLConstant "MHTERM2"Constant "Reported Term for the Medical History 2"Copy TERM3
VariablePermitted values
QNAMMHTERM1, MHTERM2
VersionDateAuthorDescriptionReviewerSign-off
1.72026-10-20yamaa docs buildRead a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default
1.62026-10-17yamaa docs buildDescribe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight
1.52026-10-17yamaa docs buildName collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part
1.42026-10-15yamaa docs buildState datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/sdtm-suppmh-long-terms/full.md

Edit
26 lines

Following CDISC SDTM standards, use the provided MH_RAW dataset to create a SUPPMH dataset with one record per medical history record per continuation of its reported term.

The output dataset should contain the following columns in this order: STUDYID, RDOMAIN, USUBJID, IDVAR, IDVARVAL, QNAM, QLABEL, QVAL, QORIG, QEVAL

The reported term MHTERM is split into pieces of at most 200 characters, each ending after the last whole word that fits, so no word is broken across two pieces; the space at each break belongs to neither piece. A word that ends exactly at the 200th character fits. The first piece stays on the MH record itself, so a term of 200 characters or fewer has no SUPPMH record. Each SUPPMH record continues one term:

  • RDOMAIN is "MH" and IDVAR is "MHSEQ".
  • IDVARVAL is the parent record's MHSEQ, written as text.
  • QNAM is "MHTERM1" for the second piece, with QLABEL "Reported Term for the Medical History 1", and "MHTERM2" for the third piece, with QLABEL "Reported Term for the Medical History 2". A term with two pieces has MHTERM1 only, and a term with three has both.
  • QVAL is the piece the record continues.
  • QORIG is "Collected".
  • QEVAL has no value, since a collected value is not an assessment.

Read the source datasets from /app/input and save the completed dataset as /app/output/suppmh.csv.

prompts/sdtm-suppmh-long-terms/conventions.md

Edit
16 lines

Following CDISC SDTM standards, use the provided MH_RAW dataset to create a SUPPMH dataset with one record per medical history record per continuation of its reported term.

The output dataset should contain the following columns in this order: STUDYID, RDOMAIN, USUBJID, IDVAR, IDVARVAL, QNAM, QLABEL, QVAL, QORIG, QEVAL

  • RDOMAIN is "MH" and IDVAR is "MHSEQ".
  • IDVARVAL is written as text.
  • QNAM is "MHTERM1" with QLABEL "Reported Term for the Medical History 1", or "MHTERM2" with QLABEL "Reported Term for the Medical History 2".
  • QORIG is "Collected".
  • QEVAL has no value.

Read the source datasets from /app/input and save the completed dataset as /app/output/suppmh.csv.

prompts/sdtm-suppmh-long-terms/brief.md

Edit
9 lines

Following CDISC SDTM standards, use the provided MH_RAW dataset to create a SUPPMH dataset with one record per medical history record per continuation of its reported term.

The output dataset should contain the following columns in this order: STUDYID, RDOMAIN, USUBJID, IDVAR, IDVARVAL, QNAM, QLABEL, QVAL, QORIG, QEVAL

Read the source datasets from /app/input and save the completed dataset as /app/output/suppmh.csv.

yamaa code

Public calls generated from benchmark.yaml; results must match the expected output

R

Edit
3 lines
library(yamaa)yamaa_spec("spec.yaml")$run()$output

Python

Edit
3 lines
import yamaayamaa.spec("spec.yaml").run().output

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/sdtm-suppmh-long-terms/result.R

Edit
68 lines
# Reference solution for the yamaa benchmark sdtm-suppmh-long-terms (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.suppressPackageStartupMessages({  library(dplyr)  library(readr)})mh <- read_csv("/app/input/mh_raw.csv", col_types = cols(.default = col_character()),               na = character(), trim_ws = FALSE)split_term <- function(x, width = 200) {  pieces <- character()  rest <- x  while (nchar(rest) > width) {    head <- substr(rest, 1, width + 1)    sp <- gregexpr(" ", head, fixed = TRUE)[[1]]    sp <- sp[sp > 1]    if (length(sp) == 0) {      # no break point: a single word longer than width      pieces <- c(pieces, substr(rest, 1, width))      rest <- substr(rest, width + 1, nchar(rest))    } else {      pos <- max(sp)      pieces <- c(pieces, substr(rest, 1, pos - 1))      rest <- substr(rest, pos + 1, nchar(rest))    }  }  c(pieces, rest)}labels <- c(MHTERM1 = "Reported Term for the Medical History 1",            MHTERM2 = "Reported Term for the Medical History 2")rows <- lapply(seq_len(nrow(mh)), function(i) {  p <- split_term(mh$MHTERM[i])  if (length(p) < 2) return(NULL)  extra <- p[-1]  qnam <- paste0("MHTERM", seq_along(extra))  tibble(    STUDYID = mh$STUDYID[i],    RDOMAIN = "MH",    USUBJID = mh$USUBJID[i],    IDVAR = "MHSEQ",    IDVARVAL = as.character(as.integer(mh$MHSEQ[i])),    QNAM = qnam,    QLABEL = unname(labels[qnam]),    QVAL = extra,    QORIG = "Collected",    QEVAL = NA_character_  )})suppmh <- bind_rows(rows)if (nrow(suppmh) == 0) {  suppmh <- tibble(STUDYID = character(), RDOMAIN = character(), USUBJID = character(),                   IDVAR = character(), IDVARVAL = character(), QNAM = character(),                   QLABEL = character(), QVAL = character(), QORIG = character(),                   QEVAL = character())}suppmh <- suppmh %>%  arrange(STUDYID, USUBJID, as.integer(IDVARVAL), QNAM) %>%  select(STUDYID, RDOMAIN, USUBJID, IDVAR, IDVARVAL, QNAM, QLABEL, QVAL, QORIG, QEVAL)write_csv(suppmh, "/app/output/suppmh.csv", na = "")

solutions/sdtm-suppmh-long-terms/result.py

Edit
66 lines
# Reference solution for the yamaa benchmark sdtm-suppmh-long-terms (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1."""Build SUPPMH from MH_RAW: continuation pieces of MHTERM beyond 200 characters."""import osimport pandas as pdINPUT = "/app/input"OUTPUT = "/app/output"LIMIT = 200COLUMNS = ["STUDYID", "RDOMAIN", "USUBJID", "IDVAR", "IDVARVAL",           "QNAM", "QLABEL", "QVAL", "QORIG", "QEVAL"]def split_term(text, limit=LIMIT):    """Split text into pieces of at most `limit` characters at word boundaries.    The space at each break belongs to neither piece; a word ending exactly    at the limit fits.    """    pieces = []    rest = text    while len(rest) > limit:        idx = rest.rfind(" ", 0, limit + 1)        if idx <= 0:            # No whole word fits: fall back to a hard break.            pieces.append(rest[:limit])            rest = rest[limit:]            continue        pieces.append(rest[:idx])        rest = rest[idx + 1:]    pieces.append(rest)    return piecesdef main():    mh = pd.read_csv(os.path.join(INPUT, "mh_raw.csv"), dtype=str,                     keep_default_na=False)    rows = []    for rec in mh.itertuples(index=False):        pieces = split_term(rec.MHTERM)        seq = str(int(float(rec.MHSEQ))) if rec.MHSEQ.strip() else ""        for n, piece in enumerate(pieces[1:], start=1):            rows.append({                "STUDYID": rec.STUDYID,                "RDOMAIN": "MH",                "USUBJID": rec.USUBJID,                "IDVAR": "MHSEQ",                "IDVARVAL": seq,                "QNAM": f"MHTERM{n}",                "QLABEL": f"Reported Term for the Medical History {n}",                "QVAL": piece,                "QORIG": "Collected",                "QEVAL": "",            })    out = pd.DataFrame(rows, columns=COLUMNS)    os.makedirs(OUTPUT, exist_ok=True)    out.to_csv(os.path.join(OUTPUT, "suppmh.csv"), index=False)if __name__ == "__main__":    main()

solutions/sdtm-suppmh-long-terms/result.sas

Edit
76 lines
/* opensas reference solution for the yamaa benchmark sdtm-suppmh-long-terms.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. *//* A quoted term can hold commas; DSD reads it as one field. */data mh_raw;  length STUDYID USUBJID MHTERM $1024;  infile "/app/input/mh_raw.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. MHSEQ MHTERM : $char1024.;run;/* The term is cut into pieces of at most 200 characters, each ending after   the last word that ends within its first 200 characters (a word ends   where a non-blank is followed by a blank); the blanks at the break belong   to neither piece. The first piece stays on the MH record, and the second   and third each give one SUPPMH record. */data result;  set mh_raw;  length RDOMAIN IDVAR IDVARVAL QNAM QLABEL QVAL QORIG QEVAL REST PIECE $1024;  RDOMAIN = 'MH';  IDVAR = 'MHSEQ';  IDVARVAL = strip(put(MHSEQ, best32.));  QORIG = 'Collected';  QEVAL = '';  REST = MHTERM;  do P = 1 to 3;    LEN = lengthn(REST);    PIECE = '';    if LEN > 0 then do;      if LEN <= 200 then do;        PIECE = REST;        REST = '';      end;      else do;        BRK = 0;        do K = 1 to 200;          if substr(REST, K, 1) ne ' ' and substr(REST, K + 1, 1) = ' ' then BRK = K;        end;        if BRK = 0 then do;          PIECE = substr(REST, 1, 200);          NXT = 201;        end;        else do;          PIECE = substr(REST, 1, BRK);          NXT = BRK + 1;          SKIP = 1;          do K = BRK + 1 to LEN;            if SKIP = 1 and substr(REST, K, 1) = ' ' then NXT = K + 1;            else SKIP = 0;          end;        end;        if NXT > LEN then REST = '';        else REST = substr(REST, NXT);      end;    end;    if P = 2 and LEN > 0 then do;      QNAM = 'MHTERM1';      QLABEL = 'Reported Term for the Medical History 1';      QVAL = PIECE;      output;    end;    if P = 3 and LEN > 0 then do;      QNAM = 'MHTERM2';      QLABEL = 'Reported Term for the Medical History 2';      QVAL = PIECE;      output;    end;  end;run;proc sql;  create table final as  select STUDYID, RDOMAIN, USUBJID, IDVAR, IDVARVAL, QNAM, QLABEL, QVAL, QORIG, QEVAL  from result;quit;proc export data=final outfile="/app/output/suppmh.csv" dbms=csv replace; run;