Review ask

As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.

We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.

Summary

Lifecycle: draftEditUnder review
What this benchmark means
README.md

Public entries: Python and R. Passing coverage requires an installed inventory.

Goal: derive each subject's planned treatment for the two periods of a crossover (TRT01P, TRT02P) and the planned sequence of treatments (TRTSEQP), with one record per subject.

Input: demographics (DM) listing each subject with the planned arm code (ARMCD); the trial arms design (TA) listing each arm's elements in planned order (TAETORD) with the element code (ETCD) and its epoch (EPOCH); and the trial elements (TE) giving each element's name (ELEMENT).

Variables:

  • TRT01P is the name of the first treatment element of the subject's planned arm, taking the arm's elements in planned order and counting only those in an epoch whose name begins with TREATMENT, so screening, washout and follow-up elements are skipped; empty when the subject has no planned arm code.
  • TRT02P is the name of the second treatment element of the planned arm, chosen the same way; empty when the subject has no planned arm code.
  • TRTSEQP is TRT01P and TRT02P joined by / in period order, such as Placebo/Vitamin D3; empty when the subject has no planned arm code.

Note: the planned values follow the planned arm code only, never the actual arm or the arm description, so a subject given the other order keeps the planned one, and a screen failure or a subject never assigned to an arm has all three empty.

Inputs

Under review
3 source files

input/dm.csv

Edit
4 rows
input/dm.csv
STUDYIDUSUBJIDARMCDARMACTARMCDACTARMARMNRS
YAMAA-01VITD-PBOVitamin D3 then PlaceboVITD-PBOVitamin D3 then Placebo—
YAMAA-01PBO-VITDPlacebo then Vitamin D3VITD-PBOVitamin D3 then Placebo—
YAMAA-01————SCREEN FAILURE
YAMAA-01————NOT ASSIGNED

input/ta.csv

Edit
10 rows
input/ta.csv
STUDYIDARMCDARMTAETORDETCDEPOCH
YAMAA-01PBO-VITDPlacebo then Vitamin D31SCRNSCREENING
YAMAA-01PBO-VITDPlacebo then Vitamin D32PBOTREATMENT 1
YAMAA-01PBO-VITDPlacebo then Vitamin D33WASHWASHOUT
YAMAA-01PBO-VITDPlacebo then Vitamin D34VITDTREATMENT 2
YAMAA-01PBO-VITDPlacebo then Vitamin D35FUFOLLOW-UP
YAMAA-01VITD-PBOVitamin D3 then Placebo1SCRNSCREENING
YAMAA-01VITD-PBOVitamin D3 then Placebo2VITDTREATMENT 1
YAMAA-01VITD-PBOVitamin D3 then Placebo3WASHWASHOUT
YAMAA-01VITD-PBOVitamin D3 then Placebo4PBOTREATMENT 2
YAMAA-01VITD-PBOVitamin D3 then Placebo5FUFOLLOW-UP

input/te.csv

Edit
5 rows
input/te.csv
STUDYIDETCDELEMENTTESTRLTEDUR
YAMAA-01FUFollow-upDay after the last dose of the second periodP14D
YAMAA-01PBOPlaceboFirst dose of PlaceboP14D
YAMAA-01SCRNScreeningInformed consentP14D
YAMAA-01VITDVitamin D3First dose of Vitamin D3P14D
YAMAA-01WASHWashoutDay after the last dose of the first periodP7D

Expected output

Under review
4 expected rows

expected/adsl.csv

Edit
4 rows
expected/adsl.csv
STUDYIDUSUBJIDTRT01PTRT02PTRTSEQP
YAMAA-01Vitamin D3PlaceboVitamin D3/Placebo
YAMAA-01PlaceboVitamin D3Placebo/Vitamin D3
YAMAA-01———
YAMAA-01———
Shaded columns are derived— = missing value

Human review and assessment

Your review

3 questions and your comments · View benchmark discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Dataset NameVariable NameVariable LabelTypeCodelist/Controlled TermsCoreComputational MethodOrigin
ADSLSTUDYIDStudy IdentifiercharacterCopy value from DM.STUDYID.Predecessor
ADSLUSUBJIDUnique Subject IdentifiercharacterCopy value from DM.USUBJID.Predecessor
ADSLTRT01PPlanned Treatment for Period 01characterVitamin D3, PlaceboTake ELEMENT from the TA record whose STUDYID equals DM.STUDYID and ARMCD equals DM.ARMCD, where TA.TRTNUM = 1, where ELEMENT is derived as ELEMENT from the TE record whose STUDYID equals TA.STUDYID and ETCD equals TA.ETCD, an error when unmatched, and no match yields blank.Derived
ADSLTRT02PPlanned Treatment for Period 02characterVitamin D3, PlaceboTake ELEMENT from the TA record whose STUDYID equals DM.STUDYID and ARMCD equals DM.ARMCD, where TA.TRTNUM = 2, where ELEMENT is derived as ELEMENT from the TE record whose STUDYID equals TA.STUDYID and ETCD equals TA.ETCD, an error when unmatched, and no match yields blank.Derived
ADSLTRTSEQPPlanned Sequence of TreatmentscharacterFill the template "{TRT01P}/{TRT02P}"; a missing field → blank.Derived
VariablePermitted values
TRT01PVitamin D3, Placebo
TRT02PVitamin D3, Placebo
VersionDateAuthorDescriptionReviewerSign-off
1.72026-10-20yamaa docs buildRead a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default
1.62026-10-17yamaa docs buildDescribe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight
1.52026-10-17yamaa docs buildName collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part
1.42026-10-15yamaa docs buildState datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/adam-adsl-crossover-sequence/full.md

Edit
20 lines

Following CDISC ADaM standards, use the provided DM, TA, and TE datasets to create an ADSL dataset with one record per subject.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, TRT01P, TRT02P, TRTSEQP

TRT01P is the name in TE of the first treatment element of the subject's planned arm, taking the arm's elements in TA in planned order (TAETORD) and counting only the elements in an epoch whose name begins with "TREATMENT", so screening, washout and follow-up elements do not count. TRT02P is the name in TE of the second treatment element of the planned arm, chosen the same way. TRTSEQP is TRT01P and TRT02P joined by "/" in period order, such as "Placebo/Vitamin D3". The planned values follow the planned arm code in DM only, never the actual arm or the arm description, so a subject given the other treatment order keeps the planned one. TRT01P, TRT02P and TRTSEQP have no value when the subject has no planned arm code, as for a screen failure or a subject never assigned to an arm.

Read the source datasets from /app/input and save the completed dataset as /app/output/adsl.csv.

prompts/adam-adsl-crossover-sequence/conventions.md

Edit
11 lines

Following CDISC ADaM standards, use the provided DM, TA, and TE datasets to create an ADSL dataset with one record per subject.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, TRT01P, TRT02P, TRTSEQP

TRTSEQP is TRT01P and TRT02P joined by "/" in period order, such as "Placebo/Vitamin D3".

Read the source datasets from /app/input and save the completed dataset as /app/output/adsl.csv.

prompts/adam-adsl-crossover-sequence/brief.md

Edit
8 lines

Following CDISC ADaM standards, use the provided DM, TA, and TE datasets to create an ADSL dataset with one record per subject.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, TRT01P, TRT02P, TRTSEQP

Read the source datasets from /app/input and save the completed dataset as /app/output/adsl.csv.

yamaa code

Public calls generated from benchmark.yaml; results must match the expected output

R

Edit
3 lines
library(yamaa)yamaa_spec("spec.yaml")$run()$output

Python

Edit
3 lines
import yamaayamaa.spec("spec.yaml").run().output

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/adam-adsl-crossover-sequence/result.R

Edit
57 lines
# Reference solution for the yamaa benchmark# adam-adsl-crossover-sequence (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)dm <- read_csv(  "/app/input/dm.csv",  col_types = cols(.default = col_character()))ta <- read_csv(  "/app/input/ta.csv",  col_types = cols(TAETORD = col_integer(), .default = col_character()))te <- read_csv(  "/app/input/te.csv",  col_types = cols(.default = col_character()))# The treatment elements of each arm, numbered in planned order: only the# elements in an epoch whose name begins with TREATMENT are counted.ranked <- ta |>  filter(startsWith(EPOCH, "TREATMENT")) |>  arrange(STUDYID, ARMCD, TAETORD) |>  group_by(STUDYID, ARMCD) |>  mutate(TRTNUM = row_number()) |>  ungroup() |>  left_join(    te |> select(STUDYID, ETCD, ELEMENT),    by = c("STUDYID", "ETCD")  )period1 <- ranked |>  filter(TRTNUM == 1L) |>  transmute(STUDYID, ARMCD, TRT01P = ELEMENT)period2 <- ranked |>  filter(TRTNUM == 2L) |>  transmute(STUDYID, ARMCD, TRT02P = ELEMENT)adsl <- dm |>  left_join(period1, by = c("STUDYID", "ARMCD")) |>  left_join(period2, by = c("STUDYID", "ARMCD")) |>  mutate(    TRTSEQP = if_else(      !is.na(TRT01P) & !is.na(TRT02P),      paste(TRT01P, TRT02P, sep = "/"),      NA_character_    )  ) |>  select(STUDYID, USUBJID, TRT01P, TRT02P, TRTSEQP)dir.create("/app/output", showWarnings = FALSE)write_csv(adsl, "/app/output/adsl.csv", na = "")

solutions/adam-adsl-crossover-sequence/result.py

Edit
88 lines
# Reference solution for the yamaa benchmark adam-adsl-crossover-sequence (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1."""Create ADSL planned treatment sequence from DM, TA and TE.One record per subject in DM. TRT01P/TRT02P are the names (from TE) of thefirst and second treatment elements of the subject's planned arm (DM ARMCD),taking the arm's elements from TA in TAETORD order and counting onlyelements whose epoch name begins with "TREATMENT". TRTSEQP joins the twowith "/". Subjects with no planned arm code get no values."""import csvimport osINPUT_DIR = "/app/input"OUTPUT_DIR = "/app/output"def read_csv(name):    with open(os.path.join(INPUT_DIR, name), newline="", encoding="utf-8") as f:        return list(csv.DictReader(f))def main():    dm = read_csv("dm.csv")    ta = read_csv("ta.csv")    te = read_csv("te.csv")    # Element code -> element name, per study.    element_name = {}    for row in te:        element_name[(row["STUDYID"], row["ETCD"])] = row["ELEMENT"]    # Planned arm -> ordered treatment element codes, per study.    arm_elements = {}    for row in ta:        if not row["EPOCH"].startswith("TREATMENT"):            continue        key = (row["STUDYID"], row["ARMCD"])        arm_elements.setdefault(key, []).append(            (int(row["TAETORD"]), row["ETCD"])        )    for key in arm_elements:        arm_elements[key] = [            etcd for _, etcd in sorted(arm_elements[key])        ]    out_rows = []    for subj in dm:        studyid = subj["STUDYID"]        armcd = subj.get("ARMCD", "")        trt01p = trt02p = trtseqp = ""        if armcd:            codes = arm_elements.get((studyid, armcd), [])            names = [element_name.get((studyid, c), "") for c in codes]            if len(names) >= 1:                trt01p = names[0]            if len(names) >= 2:                trt02p = names[1]            parts = [n for n in (trt01p, trt02p) if n]            if parts:                trtseqp = "/".join(parts)        out_rows.append(            {                "STUDYID": studyid,                "USUBJID": subj["USUBJID"],                "TRT01P": trt01p,                "TRT02P": trt02p,                "TRTSEQP": trtseqp,            }        )    os.makedirs(OUTPUT_DIR, exist_ok=True)    fieldnames = ["STUDYID", "USUBJID", "TRT01P", "TRT02P", "TRTSEQP"]    with open(        os.path.join(OUTPUT_DIR, "adsl.csv"), "w", newline="", encoding="utf-8"    ) as f:        writer = csv.DictWriter(f, fieldnames=fieldnames)        writer.writeheader()        writer.writerows(out_rows)if __name__ == "__main__":    main()

solutions/adam-adsl-crossover-sequence/result.sas

Edit
65 lines
/* opensas reference solution for the yamaa benchmark adam-adsl-crossover-sequence.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data dm;  length STUDYID USUBJID ARMCD ARM ACTARMCD ACTARM ARMNRS $1024;  infile "/app/input/dm.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. ARMCD : $char1024. ARM : $char1024. ACTARMCD : $char1024. ACTARM : $char1024. ARMNRS : $char1024.;run;data ta;  length STUDYID ARMCD ARM ETCD EPOCH $1024;  infile "/app/input/ta.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. ARMCD : $char1024. ARM : $char1024. TAETORD ETCD : $char1024. EPOCH : $char1024.;run;data te;  length STUDYID ETCD ELEMENT TESTRL TEDUR $1024;  infile "/app/input/te.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. ETCD : $char1024. ELEMENT : $char1024. TESTRL : $char1024. TEDUR : $char1024.;run;/* Only elements in an epoch whose name begins with TREATMENT count, taken   in planned order within each arm. */data treatment;  set ta;  where substr(EPOCH, 1, 9) = 'TREATMENT';run;proc sort data=treatment;  by STUDYID ARMCD TAETORD;run;data ranked;  set treatment;  by STUDYID ARMCD;  if first.ARMCD then TRTNUM = 0;  TRTNUM + 1;run;proc sql;  create table periods as    select r.STUDYID, r.ARMCD, r.TRTNUM, t.ELEMENT    from ranked as r    left join te as t on r.STUDYID = t.STUDYID and r.ETCD = t.ETCD;  create table period1 as    select STUDYID, ARMCD, ELEMENT as TRT01P from periods where TRTNUM = 1;  create table period2 as    select STUDYID, ARMCD, ELEMENT as TRT02P from periods where TRTNUM = 2;  create table joined as    select a.STUDYID, a.USUBJID, b.TRT01P, c.TRT02P    from dm as a    left join period1 as b on a.STUDYID = b.STUDYID and a.ARMCD = b.ARMCD    left join period2 as c on a.STUDYID = c.STUDYID and a.ARMCD = c.ARMCD;quit;data result;  length TRT01P TRT02P TRTSEQP $1024;  set joined;  if not missing(TRT01P) and not missing(TRT02P) then    TRTSEQP = trim(TRT01P) || '/' || trim(TRT02P);run;proc sql;  create table final as select STUDYID, USUBJID, TRT01P, TRT02P, TRTSEQP from result;quit;proc export data=final outfile="/app/output/adsl.csv" dbms=csv replace; run;