As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.
We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.
Summary
README.mdPublic entries: Python and R. Passing coverage requires an installed inventory.
Goal: write the supplemental records MHTERM1 and MHTERM2 that
continue a reported medical history term longer than 200 characters,
carrying IDVARVAL, QLABEL, QVAL, QORIG, and QEVAL.
Input: a medical history extract with one record per condition, carrying the parent record sequence and the reported term as collected.
Variables:
IDVARVALis the parent record sequence, written as text.QNAMisMHTERM1for the first continuation andMHTERM2for the second.QLABELisReported Term for the Medical History 1forMHTERM1andReported Term for the Medical History 2forMHTERM2.QVALis the piece of the reported term that the record continues.QORIGrecords the collected origin asCollected.QEVALis blank, since a collected value is not an assessment.
Note: the reported term is split into pieces of at most 200
characters, each ending after the last whole word that fits, so no word is
broken across two pieces; the space at each break belongs to neither piece.
The first piece stays on the medical history record itself. A term of 200
characters or fewer has no supplemental record, a term with two pieces has
MHTERM1 only, and a term with three has both; a term that needs a fourth
piece stops the run instead of losing its end.
Inputs
Under review1 source fileinput/mh_raw.csv
Edit5 rows| STUDYID | USUBJID | MHSEQ | MHTERM |
|---|---|---|---|
| YAMAA-01 | 1 | Seasonal allergic rhinitis | |
| YAMAA-01 | 2 | Recurrent migraine without aura since adolescence, with attacks two to three times a month lasting up to two days, accompanied by nausea and sensitivity to light, partially relieved by triptans and rest in a dark room | |
| YAMAA-01 | 1 | Type 2 diabetes mellitus diagnosed at a routine employment screening, initially managed with diet and exercise alone, later treated with metformin and then with the addition of a sulfonylurea after glycated haemoglobin rose above target on two consecutive annual reviews; complicated by mild background retinopathy found on the first retinal screening and by peripheral sensory neuropathy affecting both feet, with no history of foot ulceration or hospital admission for hypoglycaemia | |
| YAMAA-01 | 2 | Gastro-oesophageal reflux disease with heartburn after large meals and when lying flat, improved with a proton pump inhibitor taken each morning, endoscopy two years ago showing mild oesophagitis only | |
| YAMAA-01 | 1 | Osteoarthritis of both knees, worse on the right, with morning stiffness of less than thirty minutes and pain on stairs, managed with paracetamol, topical diclofenac and a home exercise programme from physio |
Expected output
Under review4 expected rowsexpected/suppmh.csv
Edit4 rows| STUDYID | RDOMAIN | USUBJID | IDVAR | IDVARVAL | QNAM | QLABEL | QVAL | QORIG | QEVAL |
|---|---|---|---|---|---|---|---|---|---|
| YAMAA-01 | MH | MHSEQ | 2 | MHTERM1 | Reported Term for the Medical History 1 | rest in a dark room | Collected | — | |
| YAMAA-01 | MH | MHSEQ | 1 | MHTERM1 | Reported Term for the Medical History 1 | glycated haemoglobin rose above target on two consecutive annual reviews; complicated by mild background retinopathy found on the first retinal screening and by peripheral sensory neuropathy affecting | Collected | — | |
| YAMAA-01 | MH | MHSEQ | 1 | MHTERM2 | Reported Term for the Medical History 2 | both feet, with no history of foot ulceration or hospital admission for hypoglycaemia | Collected | — | |
| YAMAA-01 | MH | MHSEQ | 1 | MHTERM1 | Reported Term for the Medical History 1 | physio | Collected | — |
Human review and assessment
| Variable Name | Variable Label | Type | Length | Controlled Terms or Format | Origin | Core | Conversion Definition | Variable Type | Variable Order | Comments for Define |
|---|---|---|---|---|---|---|---|---|---|---|
| STUDYID | Study Identifier | Char | Collected | Copy value from MH_RAW.STUDYID. | SUPP | 1 | ||||
| RDOMAIN | Related Domain Abbreviation | Char | Assigned | Set constant value "MH". | SUPP | 2 | ||||
| USUBJID | Unique Subject Identifier | Char | Collected | Copy value from MH_RAW.USUBJID. | SUPP | 3 | ||||
| IDVAR | Identifying Variable | Char | Assigned | Set constant value "MHSEQ". | SUPP | 4 | ||||
| IDVARVAL | Identifying Variable Value | Char | Collected | Copy value from MH_RAW.MHSEQ. | SUPP | 5 | ||||
| QNAM | Qualifier Variable Name | Char | MHTERM1, MHTERM2 | Derived | Set per constructed row; see the Row construction sheet. | SUPP | 6 | |||
| QLABEL | Qualifier Variable Label | Char | Derived | Set per constructed row; see the Row construction sheet. | SUPP | 7 | ||||
| QVAL | Data Value | Char | Derived | Set per constructed row; see the Row construction sheet. | SUPP | 8 | ||||
| QORIG | Origin | Char | Assigned | Set constant value "Collected". | SUPP | 9 | ||||
| QEVAL | Evaluator | Char | Assigned | Set constant value "None". | SUPP | 10 |
| Template | Include when | QNAM | QLABEL | QVAL |
|---|---|---|---|---|
| second_piece | TERM2 IS NOT NULL | Constant "MHTERM1" | Constant "Reported Term for the Medical History 1" | Copy TERM2 |
| third_piece | TERM3 IS NOT NULL | Constant "MHTERM2" | Constant "Reported Term for the Medical History 2" | Copy TERM3 |
| Variable | Permitted values |
|---|---|
| QNAM | MHTERM1, MHTERM2 |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.7 | 2026-10-20 | yamaa docs build | Read a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default | ||
| 1.6 | 2026-10-17 | yamaa docs build | Describe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight | ||
| 1.5 | 2026-10-17 | yamaa docs build | Name collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part | ||
| 1.4 | 2026-10-15 | yamaa docs build | State datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks | ||
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/sdtm-suppmh-long-terms/full.md
Edit26 linesFollowing CDISC SDTM standards, use the provided MH_RAW dataset to create a SUPPMH dataset with one record per medical history record per continuation of its reported term.
The output dataset should contain the following columns in this order: STUDYID, RDOMAIN, USUBJID, IDVAR, IDVARVAL, QNAM, QLABEL, QVAL, QORIG, QEVAL
The reported term MHTERM is split into pieces of at most 200 characters, each ending after the last whole word that fits, so no word is broken across two pieces; the space at each break belongs to neither piece. A word that ends exactly at the 200th character fits. The first piece stays on the MH record itself, so a term of 200 characters or fewer has no SUPPMH record. Each SUPPMH record continues one term:
- RDOMAIN is "MH" and IDVAR is "MHSEQ".
- IDVARVAL is the parent record's MHSEQ, written as text.
- QNAM is "MHTERM1" for the second piece, with QLABEL "Reported Term for the Medical History 1", and "MHTERM2" for the third piece, with QLABEL "Reported Term for the Medical History 2". A term with two pieces has MHTERM1 only, and a term with three has both.
- QVAL is the piece the record continues.
- QORIG is "Collected".
- QEVAL has no value, since a collected value is not an assessment.
Read the source datasets from /app/input and save the completed dataset as /app/output/suppmh.csv.
prompts/sdtm-suppmh-long-terms/conventions.md
Edit16 linesFollowing CDISC SDTM standards, use the provided MH_RAW dataset to create a SUPPMH dataset with one record per medical history record per continuation of its reported term.
The output dataset should contain the following columns in this order: STUDYID, RDOMAIN, USUBJID, IDVAR, IDVARVAL, QNAM, QLABEL, QVAL, QORIG, QEVAL
- RDOMAIN is "MH" and IDVAR is "MHSEQ".
- IDVARVAL is written as text.
- QNAM is "MHTERM1" with QLABEL "Reported Term for the Medical History 1", or "MHTERM2" with QLABEL "Reported Term for the Medical History 2".
- QORIG is "Collected".
- QEVAL has no value.
Read the source datasets from /app/input and save the completed dataset as /app/output/suppmh.csv.
prompts/sdtm-suppmh-long-terms/brief.md
Edit9 linesFollowing CDISC SDTM standards, use the provided MH_RAW dataset to create a SUPPMH dataset with one record per medical history record per continuation of its reported term.
The output dataset should contain the following columns in this order: STUDYID, RDOMAIN, USUBJID, IDVAR, IDVARVAL, QNAM, QLABEL, QVAL, QORIG, QEVAL
Read the source datasets from /app/input and save the completed dataset as /app/output/suppmh.csv.
R
Edit3 lineslibrary(yamaa)yamaa_spec("spec.yaml")$run()$outputPython
Edit3 linesimport yamaayamaa.spec("spec.yaml").run().outputsolutions/sdtm-suppmh-long-terms/result.R
Edit68 lines# Reference solution for the yamaa benchmark sdtm-suppmh-long-terms (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.suppressPackageStartupMessages({ library(dplyr) library(readr)})mh <- read_csv("/app/input/mh_raw.csv", col_types = cols(.default = col_character()), na = character(), trim_ws = FALSE)split_term <- function(x, width = 200) { pieces <- character() rest <- x while (nchar(rest) > width) { head <- substr(rest, 1, width + 1) sp <- gregexpr(" ", head, fixed = TRUE)[[1]] sp <- sp[sp > 1] if (length(sp) == 0) { # no break point: a single word longer than width pieces <- c(pieces, substr(rest, 1, width)) rest <- substr(rest, width + 1, nchar(rest)) } else { pos <- max(sp) pieces <- c(pieces, substr(rest, 1, pos - 1)) rest <- substr(rest, pos + 1, nchar(rest)) } } c(pieces, rest)}labels <- c(MHTERM1 = "Reported Term for the Medical History 1", MHTERM2 = "Reported Term for the Medical History 2")rows <- lapply(seq_len(nrow(mh)), function(i) { p <- split_term(mh$MHTERM[i]) if (length(p) < 2) return(NULL) extra <- p[-1] qnam <- paste0("MHTERM", seq_along(extra)) tibble( STUDYID = mh$STUDYID[i], RDOMAIN = "MH", USUBJID = mh$USUBJID[i], IDVAR = "MHSEQ", IDVARVAL = as.character(as.integer(mh$MHSEQ[i])), QNAM = qnam, QLABEL = unname(labels[qnam]), QVAL = extra, QORIG = "Collected", QEVAL = NA_character_ )})suppmh <- bind_rows(rows)if (nrow(suppmh) == 0) { suppmh <- tibble(STUDYID = character(), RDOMAIN = character(), USUBJID = character(), IDVAR = character(), IDVARVAL = character(), QNAM = character(), QLABEL = character(), QVAL = character(), QORIG = character(), QEVAL = character())}suppmh <- suppmh %>% arrange(STUDYID, USUBJID, as.integer(IDVARVAL), QNAM) %>% select(STUDYID, RDOMAIN, USUBJID, IDVAR, IDVARVAL, QNAM, QLABEL, QVAL, QORIG, QEVAL)write_csv(suppmh, "/app/output/suppmh.csv", na = "")solutions/sdtm-suppmh-long-terms/result.py
Edit66 lines# Reference solution for the yamaa benchmark sdtm-suppmh-long-terms (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1."""Build SUPPMH from MH_RAW: continuation pieces of MHTERM beyond 200 characters."""import osimport pandas as pdINPUT = "/app/input"OUTPUT = "/app/output"LIMIT = 200COLUMNS = ["STUDYID", "RDOMAIN", "USUBJID", "IDVAR", "IDVARVAL", "QNAM", "QLABEL", "QVAL", "QORIG", "QEVAL"]def split_term(text, limit=LIMIT): """Split text into pieces of at most `limit` characters at word boundaries. The space at each break belongs to neither piece; a word ending exactly at the limit fits. """ pieces = [] rest = text while len(rest) > limit: idx = rest.rfind(" ", 0, limit + 1) if idx <= 0: # No whole word fits: fall back to a hard break. pieces.append(rest[:limit]) rest = rest[limit:] continue pieces.append(rest[:idx]) rest = rest[idx + 1:] pieces.append(rest) return piecesdef main(): mh = pd.read_csv(os.path.join(INPUT, "mh_raw.csv"), dtype=str, keep_default_na=False) rows = [] for rec in mh.itertuples(index=False): pieces = split_term(rec.MHTERM) seq = str(int(float(rec.MHSEQ))) if rec.MHSEQ.strip() else "" for n, piece in enumerate(pieces[1:], start=1): rows.append({ "STUDYID": rec.STUDYID, "RDOMAIN": "MH", "USUBJID": rec.USUBJID, "IDVAR": "MHSEQ", "IDVARVAL": seq, "QNAM": f"MHTERM{n}", "QLABEL": f"Reported Term for the Medical History {n}", "QVAL": piece, "QORIG": "Collected", "QEVAL": "", }) out = pd.DataFrame(rows, columns=COLUMNS) os.makedirs(OUTPUT, exist_ok=True) out.to_csv(os.path.join(OUTPUT, "suppmh.csv"), index=False)if __name__ == "__main__": main()solutions/sdtm-suppmh-long-terms/result.sas
Edit76 lines/* opensas reference solution for the yamaa benchmark sdtm-suppmh-long-terms. Reads the benchmark inputs and derives the requested output. Execute with the existing opensas runtime: opensas /app/output/result.sas. *//* A quoted term can hold commas; DSD reads it as one field. */data mh_raw; length STUDYID USUBJID MHTERM $1024; infile "/app/input/mh_raw.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. MHSEQ MHTERM : $char1024.;run;/* The term is cut into pieces of at most 200 characters, each ending after the last word that ends within its first 200 characters (a word ends where a non-blank is followed by a blank); the blanks at the break belong to neither piece. The first piece stays on the MH record, and the second and third each give one SUPPMH record. */data result; set mh_raw; length RDOMAIN IDVAR IDVARVAL QNAM QLABEL QVAL QORIG QEVAL REST PIECE $1024; RDOMAIN = 'MH'; IDVAR = 'MHSEQ'; IDVARVAL = strip(put(MHSEQ, best32.)); QORIG = 'Collected'; QEVAL = ''; REST = MHTERM; do P = 1 to 3; LEN = lengthn(REST); PIECE = ''; if LEN > 0 then do; if LEN <= 200 then do; PIECE = REST; REST = ''; end; else do; BRK = 0; do K = 1 to 200; if substr(REST, K, 1) ne ' ' and substr(REST, K + 1, 1) = ' ' then BRK = K; end; if BRK = 0 then do; PIECE = substr(REST, 1, 200); NXT = 201; end; else do; PIECE = substr(REST, 1, BRK); NXT = BRK + 1; SKIP = 1; do K = BRK + 1 to LEN; if SKIP = 1 and substr(REST, K, 1) = ' ' then NXT = K + 1; else SKIP = 0; end; end; if NXT > LEN then REST = ''; else REST = substr(REST, NXT); end; end; if P = 2 and LEN > 0 then do; QNAM = 'MHTERM1'; QLABEL = 'Reported Term for the Medical History 1'; QVAL = PIECE; output; end; if P = 3 and LEN > 0 then do; QNAM = 'MHTERM2'; QLABEL = 'Reported Term for the Medical History 2'; QVAL = PIECE; output; end; end;run;proc sql; create table final as select STUDYID, RDOMAIN, USUBJID, IDVAR, IDVARVAL, QNAM, QLABEL, QVAL, QORIG, QEVAL from result;quit;proc export data=final outfile="/app/output/suppmh.csv" dbms=csv replace; run;
yamaa