yamaa Review ask
As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.
We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.
Summary
EditUnder reviewWhat this benchmark meansREADME.mdPublic entries: Python and R. Passing coverage requires an installed inventory.
Goal: derive the comment COVAL and its continuations COVAL1 and
COVAL2.
Input: collected free-text comments, one record per comment, with the
subject and the comment number (COSEQ).
Variables:
COVALis the first 200 characters of the comment, or the whole comment when it is shorter.COVAL1is characters 201 to 400 of the comment, or the rest of it when the comment ends sooner; it is absent when the comment has 200 characters or fewer.COVAL2is characters 401 to 600 of the comment, or the rest of it when the comment ends sooner; it is absent when the comment has 400 characters or fewer.
Note: the comment is cut at fixed character positions, not between words, so a word can run from one piece into the next. A comment left blank gives no comment record, and a comment longer than 600 characters stops the run, since three pieces hold no more.
Inputs
Under review1 source fileinput/co_raw.csv
Edit4 rows| STUDYID | USUBJID | COSEQ | COMMENT |
|---|---|---|---|
| YAMAA-01 | 1 | Subject missed the week 4 visit because of a family emergency; the visit was completed 3 days later, outside the window. | |
| YAMAA-01 | 1 | Subject reported dizziness on standing after the evening dose, lasting about 10 minutes each time. Blood pressure was normal at the unscheduled visit, and the dose was not changed by the investigator. | |
| YAMAA-01 | 1 | Subject called the site on day 15 to report a rash on both forearms that began 2 days after starting study drug. The rash was itchy but not painful, with no blistering, fever or mouth sores. The investigator reviewed the photos sent by the subject, advised an over-the-counter antihistamine and asked the subject to return for an unscheduled visit. At that visit the rash had faded, the subject continued study drug, and no further action was needed. | |
| YAMAA-01 | 1 | — |
Expected output
Under review3 expected rowsexpected/co.csv
Edit3 rows| DOMAIN | STUDYID | USUBJID | COSEQ | COVAL | COVAL1 | COVAL2 |
|---|---|---|---|---|---|---|
| CO | YAMAA-01 | 1 | Subject missed the week 4 visit because of a family emergency; the visit was completed 3 days later, outside the window. | — | — | |
| CO | YAMAA-01 | 1 | Subject reported dizziness on standing after the evening dose, lasting about 10 minutes each time. Blood pressure was normal at the unscheduled visit, and the dose was not changed by the investigator. | — | — | |
| CO | YAMAA-01 | 1 | Subject called the site on day 15 to report a rash on both forearms that began 2 days after starting study drug. The rash was itchy but not painful, with no blistering, fever or mouth sores. The inves | tigator reviewed the photos sent by the subject, advised an over-the-counter antihistamine and asked the subject to return for an unscheduled visit. At that visit the rash had faded, the subject conti | nued study drug, and no further action was needed. |
Shaded columns are derived— = missing value
Human review and assessment
Not under review
| Variable Name | Variable Label | Type | Length | Controlled Terms or Format | Origin | Core | Conversion Definition | Variable Type | Variable Order | Comments for Define |
|---|---|---|---|---|---|---|---|---|---|---|
| DOMAIN | Domain Abbreviation | Char | Assigned | Set constant value "CO". | SDTM | 1 | ||||
| STUDYID | Study Identifier | Char | Collected | Copy value from CO_RAW.STUDYID. | SDTM | 2 | ||||
| USUBJID | Unique Subject Identifier | Char | Collected | Copy value from CO_RAW.USUBJID. | SDTM | 3 | ||||
| COSEQ | Sequence Number | Num | Collected | Copy value from CO_RAW.COSEQ. | SDTM | 4 | ||||
| COVAL | Comment | Char | Derived | Take characters 1 to 200 of CO_RAW.COMMENT, blank when it is empty. | SDTM | 5 | ||||
| COVAL1 | Comment | Char | Derived | Take characters 201 to 400 of CO_RAW.COMMENT, blank when it has fewer than 201 characters. | SDTM | 6 | ||||
| COVAL2 | Comment | Char | Derived | Take characters 401 to 600 of CO_RAW.COMMENT, blank when it has fewer than 401 characters. | SDTM | 7 |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.7 | 2026-10-20 | yamaa docs build | Read a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default | ||
| 1.6 | 2026-10-17 | yamaa docs build | Describe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight | ||
| 1.5 | 2026-10-17 | yamaa docs build | Name collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part | ||
| 1.4 | 2026-10-15 | yamaa docs build | State datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks | ||
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/sdtm-co-long-comments/full.md
Edit23 linesFollowing CDISC SDTM standards, use the provided CO_RAW dataset to create a CO dataset with one record per subject per collected comment number.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, COSEQ, COVAL, COVAL1, COVAL2
DOMAIN is "CO". COSEQ is the comment number as collected, not renumbered. A comment left blank gives no CO record.
The comment text is held in 200-character pieces cut at fixed character positions, not between words, so a word can run from one piece into the next:
- COVAL is the first 200 characters of the comment, or the whole comment when it is shorter.
- COVAL1 is characters 201 to 400 of the comment, or the rest of it when the comment ends sooner; it has no value when the comment has 200 characters or fewer.
- COVAL2 is characters 401 to 600 of the comment, or the rest of it when the comment ends sooner; it has no value when the comment has 400 characters or fewer.
Read the source datasets from /app/input and save the completed dataset as /app/output/co.csv.
prompts/sdtm-co-long-comments/conventions.md
Edit11 linesFollowing CDISC SDTM standards, use the provided CO_RAW dataset to create a CO dataset with one record per subject per collected comment number.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, COSEQ, COVAL, COVAL1, COVAL2
DOMAIN is "CO". The comment text is held in 200-character pieces cut at fixed character positions, not between words.
Read the source datasets from /app/input and save the completed dataset as /app/output/co.csv.
prompts/sdtm-co-long-comments/brief.md
Edit8 linesFollowing CDISC SDTM standards, use the provided CO_RAW dataset to create a CO dataset with one record per subject per collected comment number.
The output dataset should contain the following columns in this order: DOMAIN, STUDYID, USUBJID, COSEQ, COVAL, COVAL1, COVAL2
Read the source datasets from /app/input and save the completed dataset as /app/output/co.csv.
R
Edit3 lineslibrary(yamaa)yamaa_spec("spec.yaml")$run()$outputPython
Edit3 linesimport yamaayamaa.spec("spec.yaml").run().outputsolutions/sdtm-co-long-comments/result.R
Edit49 lines# Reference solution for the yamaa benchmark sdtm-co-long-comments (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.# Build the SDTM CO (Comments) domain from CO_RAW.# One record per subject per collected comment number; blank comments dropped.# Comment text is split at fixed 200-character positions into COVAL,# COVAL1 and COVAL2 (characters, not bytes; no word-boundary adjustment).suppressPackageStartupMessages({ library(dplyr) library(readr)})in_path <- "/app/input/co_raw.csv"out_path <- "/app/output/co.csv"raw <- read_csv( in_path, col_types = cols(.default = col_character()), na = character(), # keep text exactly as collected trim_ws = FALSE, locale = locale(encoding = "UTF-8"))piece <- function(x, start, stop) { n <- nchar(x, type = "chars") out <- substr(x, start, stop) out[is.na(x) | n < start] <- NA_character_ out}co <- raw %>% mutate(COMMENT = ifelse(is.na(COMMENT), "", COMMENT)) %>% filter(trimws(COMMENT) != "") %>% transmute( DOMAIN = "CO", STUDYID = STUDYID, USUBJID = USUBJID, COSEQ = as.integer(COSEQ), COVAL = piece(COMMENT, 1L, 200L), COVAL1 = piece(COMMENT, 201L, 400L), COVAL2 = piece(COMMENT, 401L, 600L) ) %>% arrange(STUDYID, USUBJID, COSEQ)write_csv(co, out_path, na = "")solutions/sdtm-co-long-comments/result.py
Edit71 lines# Reference solution for the yamaa benchmark sdtm-co-long-comments (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1."""Build the SDTM CO (Comments) domain from CO_RAW.One record per subject per collected comment number. The comment text issplit into 200-character pieces at fixed character positions:COVAL = characters 1-200, COVAL1 = 201-400, COVAL2 = 401-600.A blank comment gives no record. COSEQ keeps the collected comment number."""from pathlib import Pathimport pandas as pdINPUT_DIR = Path("/app/input")OUTPUT_DIR = Path("/app/output")PIECE = 200COLUMNS = ["DOMAIN", "STUDYID", "USUBJID", "COSEQ", "COVAL", "COVAL1", "COVAL2"]def main() -> None: raw = pd.read_csv( INPUT_DIR / "co_raw.csv", dtype=str, keep_default_na=False, na_values=[], encoding="utf-8", ) records = [] for row in raw.itertuples(index=False): comment = row.COMMENT if isinstance(row.COMMENT, str) else "" # A comment left blank (empty or only whitespace) gives no CO record. if comment.strip() == "": continue pieces = [comment[i * PIECE:(i + 1) * PIECE] for i in range(3)] if len(comment) > 3 * PIECE: raise ValueError( f"Comment for {row.USUBJID} COSEQ {row.COSEQ} has " f"{len(comment)} characters, more than COVAL-COVAL2 can hold" ) records.append( { "DOMAIN": "CO", "STUDYID": row.STUDYID, "USUBJID": row.USUBJID, "COSEQ": int(str(row.COSEQ).strip()), "COVAL": pieces[0], "COVAL1": pieces[1] if pieces[1] != "" else None, "COVAL2": pieces[2] if pieces[2] != "" else None, } ) co = pd.DataFrame.from_records(records, columns=COLUMNS) co["COSEQ"] = co["COSEQ"].astype("Int64") co = co.sort_values(["STUDYID", "USUBJID", "COSEQ"], kind="stable") co = co.reset_index(drop=True) OUTPUT_DIR.mkdir(parents=True, exist_ok=True) co.to_csv(OUTPUT_DIR / "co.csv", index=False, encoding="utf-8")if __name__ == "__main__": main()solutions/sdtm-co-long-comments/result.sas
Edit33 lines/* opensas reference solution for the yamaa benchmark sdtm-co-long-comments. Reads the benchmark inputs and derives the requested output. Execute with the existing opensas runtime: opensas /app/output/result.sas. *//* A quoted comment can hold commas; DSD reads it as one field. */data co_raw; length STUDYID USUBJID $40 COMMENT $1000; infile "/app/input/co_raw.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char40. USUBJID : $char40. COSEQ COMMENT : $char1000.;run;/* A blank comment gives no record. The comment is cut into pieces of 200 characters at exact positions: COVAL from character 1, COVAL1 from 201 and COVAL2 from 401, each shorter when the comment ends inside it and empty when the comment ends before it begins. */data result; set co_raw; length DOMAIN $2 COVAL COVAL1 COVAL2 $200; if missing(COMMENT) then delete; DOMAIN = 'CO'; LEN = lengthn(COMMENT); COVAL = substr(COMMENT, 1, min(LEN, 200)); COVAL1 = ''; COVAL2 = ''; if LEN > 200 then COVAL1 = substr(COMMENT, 201, min(LEN - 200, 200)); if LEN > 400 then COVAL2 = substr(COMMENT, 401, min(LEN - 400, 200));run;proc sql; create table final as select DOMAIN, STUDYID, USUBJID, COSEQ, COVAL, COVAL1, COVAL2 from result;quit;proc export data=final outfile="/app/output/co.csv" dbms=csv replace; run;