yamaa / Benchmark

SDTM.TA

Trial Arms with a Sponsor-Added Epoch

input files
1
subjects
0
expected rows
6
Review ask

As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.

We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.

Summary

Lifecycle: draftEditUnder reviewWhat this benchmark means
README.md

Public entries: Python and R. Passing coverage requires an installed inventory.

Goal: build the Trial Arms dataset, one record per planned element of each arm, carrying ARM, ETCD, ELEMENT and EPOCH, and submit it as a Dataset-JSON file with its data-definition document.

Input: the protocol's planned design, one row per element of each arm in its planned order.

Variables:

  • ARMCD and ARM are the planned arm's code and description.
  • TAETORD is the element's position within its arm, starting at 1.
  • ETCD and ELEMENT are the element's code and description.
  • EPOCH is the trial period the element belongs to: SCREENING and TREATMENT from the published epoch terminology, and EXTENSION, the open-label extension period, which the sponsor added to that list.

Note: the data-definition document is prepared for a regulatory submission, so every published epoch term carries its NCI code. The sponsor's EXTENSION term has no NCI code: the document marks it as a sponsor addition to the published list instead.

Inputs

Under review1 source file

input/design.csv

Edit6 rows
STUDYIDARMCDARMTAETORDETCDELEMENTEPOCH
STUDY02PBOPlacebo1SCRNScreeningSCREENING
STUDY02PBOPlacebo2PBOPlaceboTREATMENT
STUDY02PBOPlacebo3OLEOpen-label active drugEXTENSION
STUDY02ACTActive drug1SCRNScreeningSCREENING
STUDY02ACTActive drug2ACTActive drugTREATMENT
STUDY02ACTActive drug3OLEOpen-label active drugEXTENSION

Expected output

Under review6 expected rows

expected/define.xml

EditDefine-XML

Define-XML 2.1.11: STUDY02

Example study submitting a sponsor-extended epoch; STUDY-02; SDTMIG 3.4; CDISC/NCI 2023-12-15

Datasets
DatasetDescriptionClassStructurePurposeKeysDocumentationLocation
TATrial ArmsTRIAL DESIGNOne record per planned element per armTabulationSTUDYID, ARMCD, TAETORDta.json

TA

TA variables
VariableLabel / DescriptionTypeRoleLength or Display FormatControlled Terms or ISO FormatOrigin / Source / Method / Comment
DOMAINDomain AbbreviationtextIdentifier2Assigned; Sponsor
STUDYIDStudy IdentifiertextIdentifier7Protocol; Sponsor
ARMCDPlanned Arm CodetextTopic3Protocol; Sponsor
ARMDescription of Planned ArmtextSynonym Qualifier11Protocol; Sponsor
TAETORDPlanned Order of Element within ArmintegerTiming8Protocol; Sponsor
ETCDElement CodetextRecord Qualifier4Protocol; Sponsor
ELEMENTDescription of ElementtextSynonym Qualifier22Protocol; Sponsor
EPOCHEpochtextTiming9Epoch (CL.EPOCH) SCREENING TREATMENT EXTENSIONProtocol; Sponsor
Original source
<?xml version="1.0" encoding="UTF-8"?>
<ODM xmlns="http://www.cdisc.org/ns/odm/v1.3" xmlns:def="http://www.cdisc.org/ns/def/v2.1" xmlns:xlink="http://www.w3.org/1999/xlink" ODMVersion="1.3.2" FileType="Snapshot" FileOID="STUDY02.sdtm.Define-XML_2.1" CreationDateTime="2026-09-11T00:00:00" SourceSystem="yamaa" SourceSystemVersion="1.0" def:Context="Submission">
  <Study OID="STDY.STUDY02">
    <GlobalVariables>
      <StudyName>STUDY02</StudyName>
      <StudyDescription>Example study submitting a sponsor-extended epoch</StudyDescription>
      <ProtocolName>STUDY-02</ProtocolName>
    </GlobalVariables>
    <MetaDataVersion OID="MDV.STUDY02.sdtm" Name="STUDY02 sdtm metadata" def:DefineVersion="2.1.11">
      <def:Standards>
        <def:Standard OID="STD.SDTMIG" Name="SDTMIG" Type="IG" Version="3.4" Status="Final"/>
        <def:Standard OID="STD.CT_SDTM" Name="CDISC/NCI" Type="CT" PublishingSet="SDTM" Version="2023-12-15" Status="Final"/>
      </def:Standards>
      <ItemGroupDef OID="IG.TA" Name="TA" Domain="TA" Purpose="Tabulation" SASDatasetName="TA" Repeating="No" IsReferenceData="Yes" def:Structure="One record per planned element per arm" def:ArchiveLocationID="LF.TA" def:StandardOID="STD.SDTMIG">
        <Description>
          <TranslatedText xml:lang="en">Trial Arms</TranslatedText>
        </Description>
        <ItemRef ItemOID="IT.TA.DOMAIN" OrderNumber="1" Mandatory="Yes" Role="Identifier"/>
        <ItemRef ItemOID="IT.TA.STUDYID" OrderNumber="2" Mandatory="Yes" KeySequence="1" Role="Identifier"/>
        <ItemRef ItemOID="IT.TA.ARMCD" OrderNumber="3" Mandatory="Yes" KeySequence="2" Role="Topic"/>
        <ItemRef ItemOID="IT.TA.ARM" OrderNumber="4" Mandatory="Yes" Role="Synonym Qualifier"/>
        <ItemRef ItemOID="IT.TA.TAETORD" OrderNumber="5" Mandatory="Yes" KeySequence="3" Role="Timing"/>
        <ItemRef ItemOID="IT.TA.ETCD" OrderNumber="6" Mandatory="Yes" Role="Record Qualifier"/>
        <ItemRef ItemOID="IT.TA.ELEMENT" OrderNumber="7" Mandatory="No" Role="Synonym Qualifier"/>
        <ItemRef ItemOID="IT.TA.EPOCH" OrderNumber="8" Mandatory="Yes" Role="Timing"/>
        <def:Class Name="TRIAL DESIGN"/>
        <def:leaf ID="LF.TA" xlink:href="ta.json">
          <def:title>ta.json</def:title>
        </def:leaf>
      </ItemGroupDef>
      <ItemDef OID="IT.TA.DOMAIN" Name="DOMAIN" DataType="text" Length="2" SASFieldName="DOMAIN">
        <Description>
          <TranslatedText xml:lang="en">Domain Abbreviation</TranslatedText>
        </Description>
        <def:Origin Type="Assigned" Source="Sponsor"/>
      </ItemDef>
      <ItemDef OID="IT.TA.STUDYID" Name="STUDYID" DataType="text" Length="7" SASFieldName="STUDYID">
        <Description>
          <TranslatedText xml:lang="en">Study Identifier</TranslatedText>
        </Description>
        <def:Origin Type="Protocol" Source="Sponsor"/>
      </ItemDef>
      <ItemDef OID="IT.TA.ARMCD" Name="ARMCD" DataType="text" Length="3" SASFieldName="ARMCD">
        <Description>
          <TranslatedText xml:lang="en">Planned Arm Code</TranslatedText>
        </Description>
        <def:Origin Type="Protocol" Source="Sponsor"/>
      </ItemDef>
      <ItemDef OID="IT.TA.ARM" Name="ARM" DataType="text" Length="11" SASFieldName="ARM">
        <Description>
          <TranslatedText xml:lang="en">Description of Planned Arm</TranslatedText>
        </Description>
        <def:Origin Type="Protocol" Source="Sponsor"/>
      </ItemDef>
      <ItemDef OID="IT.TA.TAETORD" Name="TAETORD" DataType="integer" Length="8" SASFieldName="TAETORD">
        <Description>
          <TranslatedText xml:lang="en">Planned Order of Element within Arm</TranslatedText>
        </Description>
        <def:Origin Type="Protocol" Source="Sponsor"/>
      </ItemDef>
      <ItemDef OID="IT.TA.ETCD" Name="ETCD" DataType="text" Length="4" SASFieldName="ETCD">
        <Description>
          <TranslatedText xml:lang="en">Element Code</TranslatedText>
        </Description>
        <def:Origin Type="Protocol" Source="Sponsor"/>
      </ItemDef>
      <ItemDef OID="IT.TA.ELEMENT" Name="ELEMENT" DataType="text" Length="22" SASFieldName="ELEMENT">
        <Description>
          <TranslatedText xml:lang="en">Description of Element</TranslatedText>
        </Description>
        <def:Origin Type="Protocol" Source="Sponsor"/>
      </ItemDef>
      <ItemDef OID="IT.TA.EPOCH" Name="EPOCH" DataType="text" Length="9" SASFieldName="EPOCH">
        <Description>
          <TranslatedText xml:lang="en">Epoch</TranslatedText>
        </Description>
        <CodeListRef CodeListOID="CL.EPOCH"/>
        <def:Origin Type="Protocol" Source="Sponsor"/>
      </ItemDef>
      <CodeList OID="CL.EPOCH" Name="Epoch" DataType="text" def:StandardOID="STD.CT_SDTM">
        <EnumeratedItem CodedValue="SCREENING">
          <Alias Context="nci:ExtCodeID" Name="C48262"/>
        </EnumeratedItem>
        <EnumeratedItem CodedValue="TREATMENT">
          <Alias Context="nci:ExtCodeID" Name="C101526"/>
        </EnumeratedItem>
        <EnumeratedItem CodedValue="EXTENSION" def:ExtendedValue="Yes"/>
        <Alias Context="nci:ExtCodeID" Name="C99079"/>
      </CodeList>
    </MetaDataVersion>
  </Study>
</ODM>

expected/ta.csv

Edit6 rows
DOMAINSTUDYIDARMCDARMTAETORDETCDELEMENTEPOCH
TASTUDY02PBOPlacebo1SCRNScreeningSCREENING
TASTUDY02PBOPlacebo2PBOPlaceboTREATMENT
TASTUDY02PBOPlacebo3OLEOpen-label active drugEXTENSION
TASTUDY02ACTActive drug1SCRNScreeningSCREENING
TASTUDY02ACTActive drug2ACTActive drugTREATMENT
TASTUDY02ACTActive drug3OLEOpen-label active drugEXTENSION

expected/ta.json

Edit
{
  "datasetJSONCreationDateTime": "2026-09-11T00:00:00",
  "datasetJSONVersion": "1.1.0",
  "fileOID": "STUDY02.sdtm.Define-XML_2.1.TA",
  "sourceSystem": {"name": "yamaa", "version": "1.0"},
  "studyOID": "STDY.STUDY02",
  "metaDataVersionOID": "MDV.STUDY02.sdtm",
  "metaDataRef": "define.xml",
  "itemGroupOID": "IG.TA",
  "records": 6,
  "name": "TA",
  "label": "Trial Arms",
  "columns": [
    {"itemOID": "IT.TA.DOMAIN", "name": "DOMAIN", "label": "Domain Abbreviation", "dataType": "string", "length": 2},
    {"itemOID": "IT.TA.STUDYID", "name": "STUDYID", "label": "Study Identifier", "dataType": "string", "length": 7, "keySequence": 1},
    {"itemOID": "IT.TA.ARMCD", "name": "ARMCD", "label": "Planned Arm Code", "dataType": "string", "length": 3, "keySequence": 2},
    {"itemOID": "IT.TA.ARM", "name": "ARM", "label": "Description of Planned Arm", "dataType": "string", "length": 11},
    {"itemOID": "IT.TA.TAETORD", "name": "TAETORD", "label": "Planned Order of Element within Arm", "dataType": "integer", "length": 8, "keySequence": 3},
    {"itemOID": "IT.TA.ETCD", "name": "ETCD", "label": "Element Code", "dataType": "string", "length": 4},
    {"itemOID": "IT.TA.ELEMENT", "name": "ELEMENT", "label": "Description of Element", "dataType": "string", "length": 22},
    {"itemOID": "IT.TA.EPOCH", "name": "EPOCH", "label": "Epoch", "dataType": "string", "length": 9}
  ],
  "rows": [
    ["TA", "STUDY02", "PBO", "Placebo", 1, "SCRN", "Screening", "SCREENING"],
    ["TA", "STUDY02", "PBO", "Placebo", 2, "PBO", "Placebo", "TREATMENT"],
    ["TA", "STUDY02", "PBO", "Placebo", 3, "OLE", "Open-label active drug", "EXTENSION"],
    ["TA", "STUDY02", "ACT", "Active drug", 1, "SCRN", "Screening", "SCREENING"],
    ["TA", "STUDY02", "ACT", "Active drug", 2, "ACT", "Active drug", "TREATMENT"],
    ["TA", "STUDY02", "ACT", "Active drug", 3, "OLE", "Open-label active drug", "EXTENSION"]
  ]
}
Shaded columns are derived— = missing value

Human review and assessment

Your review

3 questions and your comments · View benchmark discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Variable NameVariable LabelTypeLengthControlled Terms or FormatOriginCoreConversion DefinitionVariable TypeVariable OrderComments for Define
DOMAINDomain AbbreviationChar2AssignedReqSet constant value "TA".SDTM1
STUDYIDStudy IdentifierChar12ProtocolReqCopy value from DESIGN.STUDYID.SDTM2
ARMCDPlanned Arm CodeChar8ProtocolReqCopy value from DESIGN.ARMCD.SDTM3
ARMDescription of Planned ArmChar40ProtocolReqCopy value from DESIGN.ARM.SDTM4
TAETORDPlanned Order of Element within ArmNum8ProtocolReqCopy value from DESIGN.TAETORD.SDTM5
ETCDElement CodeChar8ProtocolReqCopy value from DESIGN.ETCD.SDTM6
ELEMENTDescription of ElementChar40ProtocolPermCopy value from DESIGN.ELEMENT.SDTM7
EPOCHEpochChar20EpochProtocolReqCopy value from DESIGN.EPOCH.SDTM8
Codelist IDNameValueDecodeCodelist NCI aliasValue NCI aliasExtensibleExtended valueSAS formatDictionaryVersion
EPOCHEpochSCREENINGC99079C48262Yes
EPOCHEpochTREATMENTC99079C101526Yes
EPOCHEpochEXTENSIONC99079YesYes
VersionDateAuthorDescriptionReviewerSign-off
1.72026-10-20yamaa docs buildRead a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default
1.62026-10-17yamaa docs buildDescribe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight
1.52026-10-17yamaa docs buildName collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part
1.42026-10-15yamaa docs buildState datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/sdtm-ta-sponsor-epoch/full.md

Edit13 lines

Following CDISC SDTM standards, use the provided DESIGN dataset to create a TA dataset with one record per planned element per arm.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, ARMCD, ARM, TAETORD, ETCD, ELEMENT, EPOCH

ARMCD, ARM, ETCD, ELEMENT and EPOCH are as the protocol design states them, and TAETORD is the element's planned order within its arm, starting at 1. "EXTENSION", the open-label extension period, is a sponsor addition to the published epoch terms and is kept as written.

Read the source datasets from /app/input and save the completed dataset as /app/output/ta.csv.

prompts/sdtm-ta-sponsor-epoch/conventions.md

Edit10 lines

Following CDISC SDTM standards, use the provided DESIGN dataset to create a TA dataset with one record per planned element per arm.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, ARMCD, ARM, TAETORD, ETCD, ELEMENT, EPOCH

"EXTENSION" is a sponsor epoch and is kept as written.

Read the source datasets from /app/input and save the completed dataset as /app/output/ta.csv.

prompts/sdtm-ta-sponsor-epoch/brief.md

Edit8 lines

Following CDISC SDTM standards, use the provided DESIGN dataset to create a TA dataset with one record per planned element per arm.

The output dataset should contain the following columns in this order: DOMAIN, STUDYID, ARMCD, ARM, TAETORD, ETCD, ELEMENT, EPOCH

Read the source datasets from /app/input and save the completed dataset as /app/output/ta.csv.

yamaa code

Public calls generated from benchmark.yaml; results must match the expected output

R

Edit3 lines
library(yamaa)yamaa_project("project.yaml")$spec("TA")$run()$output

Python

Edit3 lines
import yamaayamaa.project("project.yaml").spec("TA").run().output

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/sdtm-ta-sponsor-epoch/result.R

Edit23 lines
# Reference solution for the yamaa benchmark sdtm-ta-sponsor-epoch (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)design <- read_csv(  "/app/input/design.csv",  col_types = cols(.default = col_character()))ta <- design |>  mutate(    DOMAIN = "TA",    TAETORD = as.integer(TAETORD)  ) |>  select(DOMAIN, STUDYID, ARMCD, ARM, TAETORD, ETCD, ELEMENT, EPOCH)dir.create("/app/output", showWarnings = FALSE)write_csv(ta, "/app/output/ta.csv", na = "")

solutions/sdtm-ta-sponsor-epoch/result.py

Edit28 lines
# Reference solution for the yamaa benchmark sdtm-ta-sponsor-epoch (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.from pathlib import Pathimport polars as pldesign = pl.read_csv("/app/input/design.csv", infer_schema=False)ta = design.with_columns(    DOMAIN=pl.lit("TA"),    TAETORD=pl.col("TAETORD").cast(pl.Int64),).select(    "DOMAIN",    "STUDYID",    "ARMCD",    "ARM",    "TAETORD",    "ETCD",    "ELEMENT",    "EPOCH",)Path("/app/output").mkdir(exist_ok=True)ta.write_csv("/app/output/ta.csv")

solutions/sdtm-ta-sponsor-epoch/result.sas

Edit17 lines
/* opensas reference solution for the yamaa benchmark sdtm-ta-sponsor-epoch.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data design;  length STUDYID ARMCD ARM ETCD ELEMENT EPOCH $1024;  infile "/app/input/design.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. ARMCD : $char1024. ARM : $char1024. TAETORD    ETCD : $char1024. ELEMENT : $char1024. EPOCH : $char1024.;run;data result; set design; length DOMAIN $2; DOMAIN='TA'; run;proc sql;  create table final as select DOMAIN, STUDYID, ARMCD, ARM, TAETORD, ETCD, ELEMENT,    EPOCH from result;quit;proc export data=final outfile="/app/output/ta.csv" dbms=csv replace; run;