Review ask

As a clinical trial statistician or programmer, challenge anything in this benchmark: describe a problem in Comments, or propose a fix with the Edit button on any source file.

  1. Task: Would a real study need this derivation, specified this way? Does it follow CDISC SDTM: variable names, controlled terms, record structure? Flag any rule that is ambiguous or that a sponsor would handle differently.
  2. Inputs: Do they look like data collected in a real trial, and do they include the situations you would expect to meet in one?
  3. Expected output: Would independent QC programming reproduce every value from these inputs and rules?

The other tabs beside Comments are background for assessment, not under review.

Summary

Lifecycle: reviewedEditUnder review
What this benchmark means
README.md

Goal: emit one related-records row per relationship participation, carrying IDVARVAL, RELTYPE, and RELID.

Input: the finished AE and CM datasets: each record already carries an assigned sequence number (AESEQ on AE, CMSEQ on CM) and up to two link identifiers (AELNKID1/AELNKID2 on AE, CMLNKID1/CMLNKID2 on CM). A second subject reuses link number 1, showing that link numbers are per-subject: the USUBJID key keeps the two relationships apart.

Variables:

  • IDVARVAL is the sequence number of the related record as text; always present.
  • RELTYPE is blank throughout, because each row points at one record rather than a whole dataset.
  • RELID names the relationship the row takes part in; rows sharing a value are related to one another, and every row carries one.

Note: a record with no link identifier contributes no row, while a record naming two link identifiers contributes one row per identifier - only when the two identifiers differ. A record repeating its link number in both fields produces two identical rows, which fails the run on the duplicate key; carrying a third relationship would need another link field on the collected record.

Negative case: a link number carried by only one record is a dangling relationship and fails the run. Adding the collected row CATH,CATH-UCSD-0001,4,COUGH,3, to ae.csv (AELNKID1=3, no other record carrying link 3) emits one RELREC row, and the each-relationship-has-two-or-more-rows check then fails: the group (STUDYID=CATH, USUBJID=CATH-UCSD-0001, RELID=3) holds 1 row, below the minimum of 2. That is why the passing input above carries no such row.

Inputs

Under review
2 source files

input/ae.csv

Edit
4 rows
input/ae.csv
STUDYIDUSUBJIDAESEQAETERMAELNKID1AELNKID2
CATH1HEADACHE12
CATH2NAUSEA2—
CATH3RASH——
CATH1DIZZINESS1—

input/cm.csv

Edit
4 rows
input/cm.csv
STUDYIDUSUBJIDCMSEQCMTRTCMLNKID1CMLNKID2
CATH1IBUPROFEN1—
CATH2ONDANSETRON12
CATH3EMOLLIENT——
CATH1ASPIRIN1—

Expected output

Under review
8 expected rows

expected/relrec.csv

Edit
8 rows
expected/relrec.csv
STUDYIDUSUBJIDRDOMAINIDVARIDVARVALRELTYPERELID
CATHAEAESEQ1—1
CATHCMCMSEQ1—1
CATHCMCMSEQ2—1
CATHAEAESEQ1—2
CATHAEAESEQ2—2
CATHCMCMSEQ2—2
CATHAEAESEQ1—1
CATHCMCMSEQ1—1
Shaded columns are derived— = missing value

Comments and assessment materials

Comments

Signed in with GitHub; saved in repository Discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Variable NameVariable LabelTypeLengthControlled Terms or FormatOriginCoreConversion DefinitionVariable TypeVariable OrderComments for Define
STUDYIDStudy IdentifierCharDerivedSet per constructed row; see the Row construction sheet.SDTM1
USUBJIDUnique Subject IdentifierCharDerivedSet per constructed row; see the Row construction sheet.SDTM2
RDOMAINRelated Domain AbbreviationCharDerivedSet per constructed row; see the Row construction sheet.SDTM3
IDVARIdentifying VariableCharDerivedSet per constructed row; see the Row construction sheet.SDTM4
IDVARVALIdentifying Variable ValueCharDerivedSet per constructed row; see the Row construction sheet.SDTM5
RELTYPERelationship TypeCharAssignedSet constant value "None".SDTM6
RELIDRelationship IdentifierCharDerivedSet per constructed row; see the Row construction sheet.SDTM7
TemplateInclude whenSTUDYIDUSUBJIDRDOMAINIDVARIDVARVALRELIDSEQNUM
ae_link_1AE.AELNKID1 IS NOT NULLCopy AE.STUDYIDCopy AE.USUBJIDConstant "AE"Constant "AESEQ"Copy AE.AESEQCopy AE.AELNKID1Copy AE.AESEQ
ae_link_2AE.AELNKID2 IS NOT NULLCopy AE.STUDYIDCopy AE.USUBJIDConstant "AE"Constant "AESEQ"Copy AE.AESEQCopy AE.AELNKID2Copy AE.AESEQ
cm_link_1CM.CMLNKID1 IS NOT NULLCopy CM.STUDYIDCopy CM.USUBJIDConstant "CM"Constant "CMSEQ"Copy CM.CMSEQCopy CM.CMLNKID1Copy CM.CMSEQ
cm_link_2CM.CMLNKID2 IS NOT NULLCopy CM.STUDYIDCopy CM.USUBJIDConstant "CM"Constant "CMSEQ"Copy CM.CMSEQCopy CM.CMLNKID2Copy CM.CMSEQ
VersionDateAuthorDescriptionReviewerSign-off
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/sdtm-relrec-links/full.md

Edit
20 lines

Following CDISC SDTM standards, use the provided AE and CM datasets to create a RELREC dataset with one row per relationship participation.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, RDOMAIN, IDVAR, IDVARVAL, RELTYPE, RELID

RDOMAIN names the related domain, and IDVAR its sequence variable (AESEQ on AE rows, CMSEQ on CM rows). IDVARVAL is the sequence number of the related record as text, and is always present. RELTYPE has no value throughout, because each row points at one record rather than a whole dataset. RELID names the relationship the row takes part in: it is the collected link number, written as text without a decimal point, so rows sharing a value are related to one another, and every row carries one.

A record with no link identifier contributes no row, while a record naming two link identifiers contributes one row per identifier. Link numbers are per-subject: USUBJID keeps a reused number in distinct relationships.

Read the source datasets from /app/input and save the completed dataset as /app/output/relrec.csv.

prompts/sdtm-relrec-links/conventions.md

Edit
12 lines

Following CDISC SDTM standards, use the provided AE and CM datasets to create a RELREC dataset with one row per relationship participation.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, RDOMAIN, IDVAR, IDVARVAL, RELTYPE, RELID

IDVAR is AESEQ on AE rows and CMSEQ on CM rows. IDVARVAL is the sequence number of the related record as text. RELTYPE has no value throughout. RELID is the collected link number, written as text without a decimal point.

Read the source datasets from /app/input and save the completed dataset as /app/output/relrec.csv.

prompts/sdtm-relrec-links/brief.md

Edit
8 lines

Following CDISC SDTM standards, use the provided AE and CM datasets to create a RELREC dataset with one row per relationship participation.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, RDOMAIN, IDVAR, IDVARVAL, RELTYPE, RELID

Read the source datasets from /app/input and save the completed dataset as /app/output/relrec.csv.

yamaa code

Runs the yamaa specification; its result must match the expected output

run.R

Edit
4 lines
library(yamaa)relrec <- yamaa_domain("spec.yaml")$outputrelrec

run.py

Edit
4 lines
import yamaarelrec = yamaa.yamaa_domain("spec.yaml").outputrelrec

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/sdtm-relrec-links/result.R

Edit
60 lines
# Reference solution for the yamaa benchmark sdtm-relrec-links (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)library(tidyr)ae <- read_csv(  "/app/input/ae.csv",  col_types = cols(.default = col_character()))cm <- read_csv(  "/app/input/cm.csv",  col_types = cols(.default = col_character()))# A record naming two link identifiers contributes one row per identifier.ae_rows <- ae |>  pivot_longer(    cols = c(AELNKID1, AELNKID2),    values_to = "RELID"  ) |>  filter(!is.na(RELID) & trimws(RELID) != "") |>  mutate(    RDOMAIN = "AE",    IDVAR = "AESEQ",    IDVARVAL = trimws(AESEQ),    # Link numbers as text without a decimal point.    RELID = as.character(suppressWarnings(as.integer(RELID))),    RELTYPE = NA_character_  ) |>  select(STUDYID, USUBJID, RDOMAIN, IDVAR, IDVARVAL, RELTYPE, RELID)cm_rows <- cm |>  pivot_longer(    cols = c(CMLNKID1, CMLNKID2),    values_to = "RELID"  ) |>  filter(!is.na(RELID) & trimws(RELID) != "") |>  mutate(    RDOMAIN = "CM",    IDVAR = "CMSEQ",    IDVARVAL = trimws(CMSEQ),    RELID = as.character(suppressWarnings(as.integer(RELID))),    RELTYPE = NA_character_  ) |>  select(STUDYID, USUBJID, RDOMAIN, IDVAR, IDVARVAL, RELTYPE, RELID)relrec <- bind_rows(ae_rows, cm_rows) |>  arrange(    USUBJID, suppressWarnings(as.integer(RELID)), RDOMAIN,    suppressWarnings(as.integer(IDVARVAL))  ) |>  select(STUDYID, USUBJID, RDOMAIN, IDVAR, IDVARVAL, RELTYPE, RELID)dir.create("/app/output", showWarnings = FALSE)write_csv(relrec, "/app/output/relrec.csv", na = "")

solutions/sdtm-relrec-links/result.py

Edit
90 lines
# Reference solution for the yamaa benchmark sdtm-relrec-links (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.from pathlib import Pathimport polars as pldef _present(value) -> bool:    return value is not None and str(value).strip() != ""def _link(value) -> str | None:    """A collected link number as text without a decimal point."""    if not _present(value):        return None    text = str(value).strip()    try:        number = float(text)        if number.is_integer():            return str(int(number))        return str(number)    except ValueError:        return textae = pl.read_csv("/app/input/ae.csv", infer_schema=False)cm = pl.read_csv("/app/input/cm.csv", infer_schema=False)rows = []for row in ae.to_dicts():    for link_col in ("AELNKID1", "AELNKID2"):        relid = _link(row.get(link_col))        # A record with no link identifier contributes no row.        if relid is None:            continue        rows.append(            {                "STUDYID": row["STUDYID"],                "USUBJID": row["USUBJID"],                "RDOMAIN": "AE",                "IDVAR": "AESEQ",                "IDVARVAL": str(row["AESEQ"]).strip(),                "RELTYPE": None,                "RELID": relid,            }        )for row in cm.to_dicts():    for link_col in ("CMLNKID1", "CMLNKID2"):        relid = _link(row.get(link_col))        if relid is None:            continue        rows.append(            {                "STUDYID": row["STUDYID"],                "USUBJID": row["USUBJID"],                "RDOMAIN": "CM",                "IDVAR": "CMSEQ",                "IDVARVAL": str(row["CMSEQ"]).strip(),                "RELTYPE": None,                "RELID": relid,            }        )relrec = (    pl.DataFrame(        rows,        schema={            "STUDYID": pl.String,            "USUBJID": pl.String,            "RDOMAIN": pl.String,            "IDVAR": pl.String,            "IDVARVAL": pl.String,            "RELTYPE": pl.String,            "RELID": pl.String,        },    )    .with_columns(        pl.col("IDVARVAL").cast(pl.Int64).alias("_seq"),        pl.col("RELID").cast(pl.Int64).alias("_rel"),    )    .sort(["USUBJID", "_rel", "RDOMAIN", "_seq"])    .select("STUDYID", "USUBJID", "RDOMAIN", "IDVAR", "IDVARVAL", "RELTYPE", "RELID"))Path("/app/output").mkdir(exist_ok=True)relrec.write_csv("/app/output/relrec.csv")

solutions/sdtm-relrec-links/result.sas

Edit
29 lines
/* opensas reference solution for the yamaa benchmark sdtm-relrec-links.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data ae;  length STUDYID USUBJID AETERM AELNKID1 AELNKID2 $1024;  infile "/app/input/ae.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. AESEQ AETERM : $char1024. AELNKID1 : $char1024. AELNKID2 : $char1024.;run;data cm;  length STUDYID USUBJID CMTRT CMLNKID1 CMLNKID2 $1024;  infile "/app/input/cm.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. CMSEQ CMTRT : $char1024. CMLNKID1 : $char1024. CMLNKID2 : $char1024.;run;data aelinks; set ae; length RDOMAIN IDVAR IDVARVAL RELTYPE RELID $1024; RDOMAIN='AE'; IDVAR='AESEQ'; IDVARVAL=strip(put(AESEQ,best32.)); RELTYPE='';if not missing(AELNKID1) then do; RELID=AELNKID1; output; end;if not missing(AELNKID2) then do; RELID=AELNKID2; output; end;run;data cmlinks; set cm; length RDOMAIN IDVAR IDVARVAL RELTYPE RELID $1024; RDOMAIN='CM'; IDVAR='CMSEQ'; IDVARVAL=strip(put(CMSEQ,best32.)); RELTYPE='';if not missing(CMLNKID1) then do; RELID=CMLNKID1; output; end;if not missing(CMLNKID2) then do; RELID=CMLNKID2; output; end;run;data result; set aelinks cmlinks; run;proc sql;  create table final as select STUDYID, USUBJID, RDOMAIN, IDVAR, IDVARVAL, RELTYPE, RELID from result;quit;proc export data=final outfile="/app/output/relrec.csv" dbms=csv replace; run;