As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.
We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.
Summary
README.mdPublic entries: Python and R. Passing coverage requires an installed inventory.
Goal: derive each subject's planned treatment for the two periods of
a crossover (TRT01P, TRT02P) and the planned sequence of treatments
(TRTSEQP), with one record per subject.
Input: demographics (DM) listing each subject with the planned arm
code (ARMCD); the trial arms design (TA) listing each arm's elements
in planned order (TAETORD) with the element code (ETCD) and its epoch
(EPOCH); and the trial elements (TE) giving each element's name
(ELEMENT).
Variables:
TRT01Pis the name of the first treatment element of the subject's planned arm, taking the arm's elements in planned order and counting only those in an epoch whose name begins withTREATMENT, so screening, washout and follow-up elements are skipped; empty when the subject has no planned arm code.TRT02Pis the name of the second treatment element of the planned arm, chosen the same way; empty when the subject has no planned arm code.TRTSEQPisTRT01PandTRT02Pjoined by/in period order, such asPlacebo/Vitamin D3; empty when the subject has no planned arm code.
Note: the planned values follow the planned arm code only, never the actual arm or the arm description, so a subject given the other order keeps the planned one, and a screen failure or a subject never assigned to an arm has all three empty.
Inputs
Under review3 source filesinput/dm.csv
Edit4 rows| STUDYID | USUBJID | ARMCD | ARM | ACTARMCD | ACTARM | ARMNRS |
|---|---|---|---|---|---|---|
| YAMAA-01 | VITD-PBO | Vitamin D3 then Placebo | VITD-PBO | Vitamin D3 then Placebo | — | |
| YAMAA-01 | PBO-VITD | Placebo then Vitamin D3 | VITD-PBO | Vitamin D3 then Placebo | — | |
| YAMAA-01 | — | — | — | — | SCREEN FAILURE | |
| YAMAA-01 | — | — | — | — | NOT ASSIGNED |
input/ta.csv
Edit10 rows| STUDYID | ARMCD | ARM | TAETORD | ETCD | EPOCH |
|---|---|---|---|---|---|
| YAMAA-01 | PBO-VITD | Placebo then Vitamin D3 | 1 | SCRN | SCREENING |
| YAMAA-01 | PBO-VITD | Placebo then Vitamin D3 | 2 | PBO | TREATMENT 1 |
| YAMAA-01 | PBO-VITD | Placebo then Vitamin D3 | 3 | WASH | WASHOUT |
| YAMAA-01 | PBO-VITD | Placebo then Vitamin D3 | 4 | VITD | TREATMENT 2 |
| YAMAA-01 | PBO-VITD | Placebo then Vitamin D3 | 5 | FU | FOLLOW-UP |
| YAMAA-01 | VITD-PBO | Vitamin D3 then Placebo | 1 | SCRN | SCREENING |
| YAMAA-01 | VITD-PBO | Vitamin D3 then Placebo | 2 | VITD | TREATMENT 1 |
| YAMAA-01 | VITD-PBO | Vitamin D3 then Placebo | 3 | WASH | WASHOUT |
| YAMAA-01 | VITD-PBO | Vitamin D3 then Placebo | 4 | PBO | TREATMENT 2 |
| YAMAA-01 | VITD-PBO | Vitamin D3 then Placebo | 5 | FU | FOLLOW-UP |
input/te.csv
Edit5 rows| STUDYID | ETCD | ELEMENT | TESTRL | TEDUR |
|---|---|---|---|---|
| YAMAA-01 | FU | Follow-up | Day after the last dose of the second period | P14D |
| YAMAA-01 | PBO | Placebo | First dose of Placebo | P14D |
| YAMAA-01 | SCRN | Screening | Informed consent | P14D |
| YAMAA-01 | VITD | Vitamin D3 | First dose of Vitamin D3 | P14D |
| YAMAA-01 | WASH | Washout | Day after the last dose of the first period | P7D |
Expected output
Under review4 expected rowsexpected/adsl.csv
Edit4 rows| STUDYID | USUBJID | TRT01P | TRT02P | TRTSEQP |
|---|---|---|---|---|
| YAMAA-01 | Vitamin D3 | Placebo | Vitamin D3/Placebo | |
| YAMAA-01 | Placebo | Vitamin D3 | Placebo/Vitamin D3 | |
| YAMAA-01 | — | — | — | |
| YAMAA-01 | — | — | — |
Human review and assessment
| Dataset Name | Variable Name | Variable Label | Type | Codelist/Controlled Terms | Core | Computational Method | Origin |
|---|---|---|---|---|---|---|---|
| ADSL | STUDYID | Study Identifier | character | Copy value from DM.STUDYID. | Predecessor | ||
| ADSL | USUBJID | Unique Subject Identifier | character | Copy value from DM.USUBJID. | Predecessor | ||
| ADSL | TRT01P | Planned Treatment for Period 01 | character | Vitamin D3, Placebo | Take ELEMENT from the TA record whose STUDYID equals DM.STUDYID and ARMCD equals DM.ARMCD, where TA.TRTNUM = 1, where ELEMENT is derived as ELEMENT from the TE record whose STUDYID equals TA.STUDYID and ETCD equals TA.ETCD, an error when unmatched, and no match yields blank. | Derived | |
| ADSL | TRT02P | Planned Treatment for Period 02 | character | Vitamin D3, Placebo | Take ELEMENT from the TA record whose STUDYID equals DM.STUDYID and ARMCD equals DM.ARMCD, where TA.TRTNUM = 2, where ELEMENT is derived as ELEMENT from the TE record whose STUDYID equals TA.STUDYID and ETCD equals TA.ETCD, an error when unmatched, and no match yields blank. | Derived | |
| ADSL | TRTSEQP | Planned Sequence of Treatments | character | Fill the template "{TRT01P}/{TRT02P}"; a missing field → blank. | Derived |
| Variable | Permitted values |
|---|---|
| TRT01P | Vitamin D3, Placebo |
| TRT02P | Vitamin D3, Placebo |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.7 | 2026-10-20 | yamaa docs build | Read a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default | ||
| 1.6 | 2026-10-17 | yamaa docs build | Describe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight | ||
| 1.5 | 2026-10-17 | yamaa docs build | Name collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part | ||
| 1.4 | 2026-10-15 | yamaa docs build | State datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks | ||
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/adam-adsl-crossover-sequence/full.md
Edit20 linesFollowing CDISC ADaM standards, use the provided DM, TA, and TE datasets to create an ADSL dataset with one record per subject.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, TRT01P, TRT02P, TRTSEQP
TRT01P is the name in TE of the first treatment element of the subject's planned arm, taking the arm's elements in TA in planned order (TAETORD) and counting only the elements in an epoch whose name begins with "TREATMENT", so screening, washout and follow-up elements do not count. TRT02P is the name in TE of the second treatment element of the planned arm, chosen the same way. TRTSEQP is TRT01P and TRT02P joined by "/" in period order, such as "Placebo/Vitamin D3". The planned values follow the planned arm code in DM only, never the actual arm or the arm description, so a subject given the other treatment order keeps the planned one. TRT01P, TRT02P and TRTSEQP have no value when the subject has no planned arm code, as for a screen failure or a subject never assigned to an arm.
Read the source datasets from /app/input and save the completed dataset as /app/output/adsl.csv.
prompts/adam-adsl-crossover-sequence/conventions.md
Edit11 linesFollowing CDISC ADaM standards, use the provided DM, TA, and TE datasets to create an ADSL dataset with one record per subject.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, TRT01P, TRT02P, TRTSEQP
TRTSEQP is TRT01P and TRT02P joined by "/" in period order, such as "Placebo/Vitamin D3".
Read the source datasets from /app/input and save the completed dataset as /app/output/adsl.csv.
prompts/adam-adsl-crossover-sequence/brief.md
Edit8 linesFollowing CDISC ADaM standards, use the provided DM, TA, and TE datasets to create an ADSL dataset with one record per subject.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, TRT01P, TRT02P, TRTSEQP
Read the source datasets from /app/input and save the completed dataset as /app/output/adsl.csv.
R
Edit3 lineslibrary(yamaa)yamaa_spec("spec.yaml")$run()$outputPython
Edit3 linesimport yamaayamaa.spec("spec.yaml").run().outputsolutions/adam-adsl-crossover-sequence/result.R
Edit57 lines# Reference solution for the yamaa benchmark# adam-adsl-crossover-sequence (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)dm <- read_csv( "/app/input/dm.csv", col_types = cols(.default = col_character()))ta <- read_csv( "/app/input/ta.csv", col_types = cols(TAETORD = col_integer(), .default = col_character()))te <- read_csv( "/app/input/te.csv", col_types = cols(.default = col_character()))# The treatment elements of each arm, numbered in planned order: only the# elements in an epoch whose name begins with TREATMENT are counted.ranked <- ta |> filter(startsWith(EPOCH, "TREATMENT")) |> arrange(STUDYID, ARMCD, TAETORD) |> group_by(STUDYID, ARMCD) |> mutate(TRTNUM = row_number()) |> ungroup() |> left_join( te |> select(STUDYID, ETCD, ELEMENT), by = c("STUDYID", "ETCD") )period1 <- ranked |> filter(TRTNUM == 1L) |> transmute(STUDYID, ARMCD, TRT01P = ELEMENT)period2 <- ranked |> filter(TRTNUM == 2L) |> transmute(STUDYID, ARMCD, TRT02P = ELEMENT)adsl <- dm |> left_join(period1, by = c("STUDYID", "ARMCD")) |> left_join(period2, by = c("STUDYID", "ARMCD")) |> mutate( TRTSEQP = if_else( !is.na(TRT01P) & !is.na(TRT02P), paste(TRT01P, TRT02P, sep = "/"), NA_character_ ) ) |> select(STUDYID, USUBJID, TRT01P, TRT02P, TRTSEQP)dir.create("/app/output", showWarnings = FALSE)write_csv(adsl, "/app/output/adsl.csv", na = "")solutions/adam-adsl-crossover-sequence/result.py
Edit88 lines# Reference solution for the yamaa benchmark adam-adsl-crossover-sequence (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1."""Create ADSL planned treatment sequence from DM, TA and TE.One record per subject in DM. TRT01P/TRT02P are the names (from TE) of thefirst and second treatment elements of the subject's planned arm (DM ARMCD),taking the arm's elements from TA in TAETORD order and counting onlyelements whose epoch name begins with "TREATMENT". TRTSEQP joins the twowith "/". Subjects with no planned arm code get no values."""import csvimport osINPUT_DIR = "/app/input"OUTPUT_DIR = "/app/output"def read_csv(name): with open(os.path.join(INPUT_DIR, name), newline="", encoding="utf-8") as f: return list(csv.DictReader(f))def main(): dm = read_csv("dm.csv") ta = read_csv("ta.csv") te = read_csv("te.csv") # Element code -> element name, per study. element_name = {} for row in te: element_name[(row["STUDYID"], row["ETCD"])] = row["ELEMENT"] # Planned arm -> ordered treatment element codes, per study. arm_elements = {} for row in ta: if not row["EPOCH"].startswith("TREATMENT"): continue key = (row["STUDYID"], row["ARMCD"]) arm_elements.setdefault(key, []).append( (int(row["TAETORD"]), row["ETCD"]) ) for key in arm_elements: arm_elements[key] = [ etcd for _, etcd in sorted(arm_elements[key]) ] out_rows = [] for subj in dm: studyid = subj["STUDYID"] armcd = subj.get("ARMCD", "") trt01p = trt02p = trtseqp = "" if armcd: codes = arm_elements.get((studyid, armcd), []) names = [element_name.get((studyid, c), "") for c in codes] if len(names) >= 1: trt01p = names[0] if len(names) >= 2: trt02p = names[1] parts = [n for n in (trt01p, trt02p) if n] if parts: trtseqp = "/".join(parts) out_rows.append( { "STUDYID": studyid, "USUBJID": subj["USUBJID"], "TRT01P": trt01p, "TRT02P": trt02p, "TRTSEQP": trtseqp, } ) os.makedirs(OUTPUT_DIR, exist_ok=True) fieldnames = ["STUDYID", "USUBJID", "TRT01P", "TRT02P", "TRTSEQP"] with open( os.path.join(OUTPUT_DIR, "adsl.csv"), "w", newline="", encoding="utf-8" ) as f: writer = csv.DictWriter(f, fieldnames=fieldnames) writer.writeheader() writer.writerows(out_rows)if __name__ == "__main__": main()solutions/adam-adsl-crossover-sequence/result.sas
Edit65 lines/* opensas reference solution for the yamaa benchmark adam-adsl-crossover-sequence. Reads the benchmark inputs and derives the requested output. Execute with the existing opensas runtime: opensas /app/output/result.sas. */data dm; length STUDYID USUBJID ARMCD ARM ACTARMCD ACTARM ARMNRS $1024; infile "/app/input/dm.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. ARMCD : $char1024. ARM : $char1024. ACTARMCD : $char1024. ACTARM : $char1024. ARMNRS : $char1024.;run;data ta; length STUDYID ARMCD ARM ETCD EPOCH $1024; infile "/app/input/ta.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. ARMCD : $char1024. ARM : $char1024. TAETORD ETCD : $char1024. EPOCH : $char1024.;run;data te; length STUDYID ETCD ELEMENT TESTRL TEDUR $1024; infile "/app/input/te.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. ETCD : $char1024. ELEMENT : $char1024. TESTRL : $char1024. TEDUR : $char1024.;run;/* Only elements in an epoch whose name begins with TREATMENT count, taken in planned order within each arm. */data treatment; set ta; where substr(EPOCH, 1, 9) = 'TREATMENT';run;proc sort data=treatment; by STUDYID ARMCD TAETORD;run;data ranked; set treatment; by STUDYID ARMCD; if first.ARMCD then TRTNUM = 0; TRTNUM + 1;run;proc sql; create table periods as select r.STUDYID, r.ARMCD, r.TRTNUM, t.ELEMENT from ranked as r left join te as t on r.STUDYID = t.STUDYID and r.ETCD = t.ETCD; create table period1 as select STUDYID, ARMCD, ELEMENT as TRT01P from periods where TRTNUM = 1; create table period2 as select STUDYID, ARMCD, ELEMENT as TRT02P from periods where TRTNUM = 2; create table joined as select a.STUDYID, a.USUBJID, b.TRT01P, c.TRT02P from dm as a left join period1 as b on a.STUDYID = b.STUDYID and a.ARMCD = b.ARMCD left join period2 as c on a.STUDYID = c.STUDYID and a.ARMCD = c.ARMCD;quit;data result; length TRT01P TRT02P TRTSEQP $1024; set joined; if not missing(TRT01P) and not missing(TRT02P) then TRTSEQP = trim(TRT01P) || '/' || trim(TRT02P);run;proc sql; create table final as select STUDYID, USUBJID, TRT01P, TRT02P, TRTSEQP from result;quit;proc export data=final outfile="/app/output/adsl.csv" dbms=csv replace; run;
yamaa