Review ask

As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.

We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.

Summary

Lifecycle: reviewedEditUnder review
What this benchmark means
README.md

Public entries: Python and R. Passing coverage requires an installed inventory.

Goal: derive TR01SDT, TR01EDT, TR02SDT, TR02EDT, TRT01A, TRT02A, and WASHDUR to capture each treatment period and the washout between them, with one record per subject.

Input: demographics listing each subject, plus exposure records carrying the epoch (EPOCH), the treatment given (EXTRT), start and end dates (EXSTDTC, EXENDTC), and a sequence number (EXSEQ).

Variables:

  • TR01SDT is the earliest EXSTDTC among the subject's exposure records in the TREATMENT 1 epoch and a recorded start date; empty when the subject has no such record.
  • TR01EDT is the latest EXENDTC among the subject's exposure records in the TREATMENT 1 epoch and a recorded end date; empty when the subject has no such record.
  • TR02SDT is the earliest EXSTDTC among the subject's exposure records in the TREATMENT 2 epoch and a recorded start date; empty when the subject has no such record.
  • TR02EDT is the latest EXENDTC among the subject's exposure records in the TREATMENT 2 epoch and a recorded end date; empty when the subject has no such record.
  • TRT01A is the EXTRT from the subject's earliest exposure record in the TREATMENT 1 epoch, earliest by EXSTDTC with the lower EXSEQ breaking ties on the same day and a record with no start date sorting last; either VITAMIN D3 or PLACEBO, and empty when the subject has no period-one exposure.
  • TRT02A is the EXTRT from the subject's earliest exposure record in the TREATMENT 2 epoch, chosen the same way; either VITAMIN D3 or PLACEBO, and empty when the subject has no period-two exposure.
  • WASHDUR is the number of days strictly between TR01EDT and TR02SDT, counting neither endpoint, so periods that touch give zero; empty when either date is missing.

Note: every value comes only from exposure records in its own period, so period-one dates and treatment never mix with period-two records. A subject with no period-two exposure leaves TR02SDT, TR02EDT, TRT02A, and WASHDUR empty together.

Inputs

Under review
2 source files

input/dm.csv

Edit
6 rows
input/dm.csv
STUDYIDUSUBJID
YAMAA-01
YAMAA-01
YAMAA-01
YAMAA-01
YAMAA-01
YAMAA-01

input/ex.csv

Edit
17 rows
input/ex.csv
STUDYIDUSUBJIDEXSEQEPOCHEXTRTEXSTDTCEXENDTC
YAMAA-011TREATMENT 1VITAMIN D32025-01-062025-01-16
YAMAA-012TREATMENT 1VITAMIN D32025-01-172025-01-26
YAMAA-013TREATMENT 2PLACEBO2025-02-102025-03-02
YAMAA-013TREATMENT 2PLACEBO2025-02-102025-03-02
YAMAA-011TREATMENT 1PLACEBO2025-01-082025-01-28
YAMAA-012TREATMENT 2VITAMIN D32025-02-122025-02-22
YAMAA-013TREATMENT 2VITAMIN D32025-02-232025-03-04
YAMAA-011TREATMENT 1VITAMIN D32025-01-102025-01-20
YAMAA-012TREATMENT 1VITAMIN D32025-03-102025-03-15
YAMAA-015TREATMENT 1PLACEBO2025-03-102025-03-20
YAMAA-011TREATMENT 2PLACEBO2025-04-012025-04-15
YAMAA-011TREATMENT 1PLACEBO2025-04-012025-04-10
YAMAA-012TREATMENT 2VITAMIN D32025-04-112025-04-25
YAMAA-011TREATMENT 1VITAMIN D32025-05-012025-05-10
YAMAA-012TREATMENT 1PLACEBO—2025-05-12
YAMAA-013TREATMENT 2PLACEBO2025-06-01—
YAMAA-014TREATMENT 2PLACEBO2025-06-022025-06-20

Expected output

Under review
6 expected rows

expected/adsl.csv

Edit
6 rows
expected/adsl.csv
STUDYIDUSUBJIDTR01SDTTR01EDTTR02SDTTR02EDTTRT01ATRT02AWASHDUR
YAMAA-012025-01-062025-01-262025-02-102025-03-02VITAMIN D3PLACEBO14
YAMAA-012025-01-082025-01-282025-02-122025-03-04PLACEBOVITAMIN D314
YAMAA-012025-01-102025-01-20——VITAMIN D3——
YAMAA-012025-03-102025-03-202025-04-012025-04-15VITAMIN D3PLACEBO11
YAMAA-012025-04-012025-04-102025-04-112025-04-25PLACEBOVITAMIN D30
YAMAA-012025-05-012025-05-122025-06-012025-06-20VITAMIN D3PLACEBO19
Shaded columns are derived— = missing value

Human review and assessment

Your review

3 questions and your comments · View benchmark discussions

Mapping spec

Generated from the benchmark YAML; do not edit by hand

Dataset NameVariable NameVariable LabelTypeCodelist/Controlled TermsCoreComputational MethodOrigin
ADSLSTUDYIDStudy IdentifiercharacterCopy value from DM.STUDYID.Predecessor
ADSLUSUBJIDUnique Subject IdentifiercharacterCopy value from DM.USUBJID.Predecessor
ADSLTR01SDTDate of First Exposure in Period 01numericISO 8601Aggregate MIN(EX.EXSTDTC) over records matching on the shared output keys, where EX.EPOCH = 'TREATMENT 1'.Derived
ADSLTR01EDTDate of Last Exposure in Period 01numericISO 8601Aggregate MAX(EX.EXENDTC) over records matching on the shared output keys, where EX.EPOCH = 'TREATMENT 1'.Derived
ADSLTR02SDTDate of First Exposure in Period 02numericISO 8601Aggregate MIN(EX.EXSTDTC) over records matching on the shared output keys, where EX.EPOCH = 'TREATMENT 2'.Derived
ADSLTR02EDTDate of Last Exposure in Period 02numericISO 8601Aggregate MAX(EX.EXENDTC) over records matching on the shared output keys, where EX.EPOCH = 'TREATMENT 2'.Derived
ADSLTRT01AActual Treatment for Period 01characterVITAMIN D3, PLACEBOCopy value from EX.EXTRT where EX.EPOCH = 'TREATMENT 1', keeping the first ordered by EX.EXSTDTC, nulls last then EX.EXSEQ.Predecessor
ADSLTRT02AActual Treatment for Period 02characterVITAMIN D3, PLACEBOCopy value from EX.EXTRT where EX.EPOCH = 'TREATMENT 2', keeping the first ordered by EX.EXSTDTC, nulls last then EX.EXSEQ.Predecessor
ADSLWASHDURWashout Duration (Days)numericWhole days from TR01EDT to TR02SDT, counting neither endpoint.Derived
VariablePermitted values
TRT01AVITAMIN D3, PLACEBO
TRT02AVITAMIN D3, PLACEBO
VersionDateAuthorDescriptionReviewerSign-off
1.72026-10-20yamaa docs buildRead a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default
1.62026-10-17yamaa docs buildDescribe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight
1.52026-10-17yamaa docs buildName collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part
1.42026-10-15yamaa docs buildState datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks
1.32026-09-29yamaa docs buildShow study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.22026-09-29yamaa docs buildClassify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.12026-09-29yamaa docs buildResolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0yamaa docs buildInitial generation from spec.yaml

Agent prompt

The request an agent receives with the input files; each tier states no more than the one above

prompts/adam-adsl-crossover-periods/full.md

Edit
36 lines

Following CDISC ADaM standards, use the provided DM and EX datasets to create an ADSL dataset with one record per subject.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, TR01SDT, TR01EDT, TR02SDT, TR02EDT, TRT01A, TRT02A, WASHDUR

  • TR01SDT is the earliest EXSTDTC among the subject's exposure records in the "TREATMENT 1" epoch with a recorded start date, and has no value when the subject has no such record.
  • TR01EDT is the latest EXENDTC among the subject's exposure records in the "TREATMENT 1" epoch with a recorded end date, and has no value when the subject has no such record.
  • TR02SDT is the earliest EXSTDTC among the subject's exposure records in the "TREATMENT 2" epoch with a recorded start date, and has no value when the subject has no such record.
  • TR02EDT is the latest EXENDTC among the subject's exposure records in the "TREATMENT 2" epoch with a recorded end date, and has no value when the subject has no such record.
  • TRT01A is the EXTRT of the subject's earliest exposure record in the "TREATMENT 1" epoch, earliest by EXSTDTC, with the lower EXSEQ breaking ties between records that start on the same day and a record with no start date counting as later than any record with one; TRT01A is either "VITAMIN D3" or "PLACEBO", and has no value when the subject has no exposure record in the "TREATMENT 1" epoch.
  • TRT02A is the EXTRT of the subject's earliest exposure record in the "TREATMENT 2" epoch, chosen the same way as TRT01A; TRT02A is either "VITAMIN D3" or "PLACEBO", and has no value when the subject has no exposure record in the "TREATMENT 2" epoch.
  • WASHDUR is the number of days strictly between TR01EDT and TR02SDT, counting neither endpoint, so periods that touch give zero, and has no value when either date has no value.
  • A subject with no exposure record in the "TREATMENT 2" epoch has no value for TR02SDT, TR02EDT, TRT02A, and WASHDUR.

Read the source datasets from /app/input and save the completed dataset as /app/output/adsl.csv.

prompts/adam-adsl-crossover-periods/conventions.md

Edit
13 lines

Following CDISC ADaM standards, use the provided DM and EX datasets to create an ADSL dataset with one record per subject.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, TR01SDT, TR01EDT, TR02SDT, TR02EDT, TRT01A, TRT02A, WASHDUR

  • TRT01A is either "VITAMIN D3" or "PLACEBO", or has no value.
  • TRT02A is either "VITAMIN D3" or "PLACEBO", or has no value.
  • WASHDUR is the number of days strictly between TR01EDT and TR02SDT, counting neither endpoint.

Read the source datasets from /app/input and save the completed dataset as /app/output/adsl.csv.

prompts/adam-adsl-crossover-periods/brief.md

Edit
8 lines

Following CDISC ADaM standards, use the provided DM and EX datasets to create an ADSL dataset with one record per subject.

The output dataset should contain the following columns in this order: STUDYID, USUBJID, TR01SDT, TR01EDT, TR02SDT, TR02EDT, TRT01A, TRT02A, WASHDUR

Read the source datasets from /app/input and save the completed dataset as /app/output/adsl.csv.

yamaa code

Public calls generated from benchmark.yaml; results must match the expected output

R

Edit
3 lines
library(yamaa)yamaa_spec("spec.yaml")$run()$output

Python

Edit
3 lines
import yamaayamaa.spec("spec.yaml").run().output

Solution without yamaa

Written from the full prompt and the inputs alone; CI checks it reproduces the expected output

solutions/adam-adsl-crossover-periods/result.R

Edit
66 lines
# Reference solution for the yamaa benchmark adam-adsl-crossover-periods (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)dm <- read_csv(  "/app/input/dm.csv",  col_types = cols(.default = col_character()))ex <- read_csv(  "/app/input/ex.csv",  col_types = cols(    EXSEQ = col_integer(),    EXSTDTC = col_date(),    EXENDTC = col_date(),    .default = col_character()  ))ex1 <- ex |> filter(EPOCH %in% "TREATMENT 1")ex2 <- ex |> filter(EPOCH %in% "TREATMENT 2")agg1 <- ex1 |>  group_by(STUDYID, USUBJID) |>  summarise(    TR01SDT = if (all(is.na(EXSTDTC))) as.Date(NA) else min(EXSTDTC, na.rm = TRUE),    TR01EDT = if (all(is.na(EXENDTC))) as.Date(NA) else max(EXENDTC, na.rm = TRUE),    .groups = "drop"  )agg2 <- ex2 |>  group_by(STUDYID, USUBJID) |>  summarise(    TR02SDT = if (all(is.na(EXSTDTC))) as.Date(NA) else min(EXSTDTC, na.rm = TRUE),    TR02EDT = if (all(is.na(EXENDTC))) as.Date(NA) else max(EXENDTC, na.rm = TRUE),    .groups = "drop"  )# The treatment on the earliest record in each period: earliest start date# with the lower sequence number breaking ties, records without a start# date sorting last.first1 <- ex1 |>  arrange(STUDYID, USUBJID, is.na(EXSTDTC), EXSTDTC, EXSEQ) |>  distinct(STUDYID, USUBJID, .keep_all = TRUE) |>  transmute(STUDYID, USUBJID, TRT01A = EXTRT)first2 <- ex2 |>  arrange(STUDYID, USUBJID, is.na(EXSTDTC), EXSTDTC, EXSEQ) |>  distinct(STUDYID, USUBJID, .keep_all = TRUE) |>  transmute(STUDYID, USUBJID, TRT02A = EXTRT)adsl <- dm |>  left_join(agg1, by = c("STUDYID", "USUBJID")) |>  left_join(agg2, by = c("STUDYID", "USUBJID")) |>  left_join(first1, by = c("STUDYID", "USUBJID")) |>  left_join(first2, by = c("STUDYID", "USUBJID")) |>  mutate(WASHDUR = as.integer(TR02SDT - TR01EDT) - 1L) |>  select(    STUDYID, USUBJID, TR01SDT, TR01EDT, TR02SDT, TR02EDT,    TRT01A, TRT02A, WASHDUR  )dir.create("/app/output", showWarnings = FALSE)write_csv(adsl, "/app/output/adsl.csv", na = "")

solutions/adam-adsl-crossover-periods/result.py

Edit
99 lines
# Reference solution for the yamaa benchmark adam-adsl-crossover-periods (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1."""Build ADSL (one record per subject) from DM and EX.Derives the two crossover treatment periods per subject: earliest startand latest end per epoch, the treatment of the earliest record in eachepoch, and the washout duration strictly between the periods."""import csvfrom datetime import dateINPUT_DIR = "/app/input"OUTPUT_DIR = "/app/output"COLUMNS = [    "STUDYID", "USUBJID", "TR01SDT", "TR01EDT", "TR02SDT", "TR02EDT",    "TRT01A", "TRT02A", "WASHDUR",]def parse_day(text):    text = (text or "").strip()    if not text:        return None    return date.fromisoformat(text)def read_rows(path):    with open(path, newline="") as handle:        return list(csv.DictReader(handle))def period(rows):    """Return (start, end, treatment) for one subject's records in one epoch."""    starts = [d for d in (parse_day(r["EXSTDTC"]) for r in rows) if d]    ends = [d for d in (parse_day(r["EXENDTC"]) for r in rows) if d]    start = min(starts) if starts else None    end = max(ends) if ends else None    treatment = ""    if rows:        def earliest_key(row):            day = parse_day(row["EXSTDTC"])            # A record with no start date counts as later than any with one.            return (day is None, day or date.max, float(row["EXSEQ"]))        treatment = min(rows, key=earliest_key)["EXTRT"]    return start, end, treatmentdef main():    dm = read_rows(f"{INPUT_DIR}/dm.csv")    ex = read_rows(f"{INPUT_DIR}/ex.csv")    by_subject = {}    for row in ex:        by_subject.setdefault((row["STUDYID"], row["USUBJID"]), []).append(row)    out = []    for subject in dm:        key = (subject["STUDYID"], subject["USUBJID"])        records = by_subject.get(key, [])        t1 = [r for r in records if r["EPOCH"] == "TREATMENT 1"]        t2 = [r for r in records if r["EPOCH"] == "TREATMENT 2"]        tr01sdt, tr01edt, trt01a = period(t1)        tr02sdt, tr02edt, trt02a = period(t2)        washdur = ""        if tr01edt is not None and tr02sdt is not None:            # Days strictly between the two dates, counting neither endpoint.            washdur = str((tr02sdt - tr01edt).days - 1)        def fmt(day):            return day.isoformat() if day is not None else ""        out.append({            "STUDYID": subject["STUDYID"],            "USUBJID": subject["USUBJID"],            "TR01SDT": fmt(tr01sdt),            "TR01EDT": fmt(tr01edt),            "TR02SDT": fmt(tr02sdt),            "TR02EDT": fmt(tr02edt),            "TRT01A": trt01a,            "TRT02A": trt02a,            "WASHDUR": washdur,        })    with open(f"{OUTPUT_DIR}/adsl.csv", "w", newline="") as handle:        writer = csv.DictWriter(handle, fieldnames=COLUMNS)        writer.writeheader()        writer.writerows(out)if __name__ == "__main__":    main()

solutions/adam-adsl-crossover-periods/result.sas

Edit
33 lines
/* opensas reference solution for the yamaa benchmark adam-adsl-crossover-periods.   Reads the benchmark inputs and derives the requested output.   Execute with the existing opensas runtime: opensas /app/output/result.sas. */data dm;  length STUDYID USUBJID $1024;  infile "/app/input/dm.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024.;run;data ex;  length STUDYID USUBJID EPOCH EXTRT EXSTDTC EXENDTC $1024;  infile "/app/input/ex.csv" dsd dlm="," firstobs=2 truncover lrecl=32767;  input STUDYID : $char1024. USUBJID : $char1024. EXSEQ EPOCH : $char1024. EXTRT : $char1024. EXSTDTC : $char1024. EXENDTC : $char1024.;run;data exposure; set ex; NODATE=missing(EXSTDTC); run;proc sort data=exposure; by STUDYID USUBJID EPOCH NODATE EXSTDTC EXSEQ; run;data firstex; set exposure; by STUDYID USUBJID EPOCH; if first.EPOCH; run;proc sql;create table starts1 as select STUDYID,USUBJID,min(EXSTDTC) as TR01SDT from ex where EPOCH='TREATMENT 1' and not missing(EXSTDTC) group by STUDYID,USUBJID;create table ends1 as select STUDYID,USUBJID,max(EXENDTC) as TR01EDT from ex where EPOCH='TREATMENT 1' group by STUDYID,USUBJID;create table starts2 as select STUDYID,USUBJID,min(EXSTDTC) as TR02SDT from ex where EPOCH='TREATMENT 2' and not missing(EXSTDTC) group by STUDYID,USUBJID;create table ends2 as select STUDYID,USUBJID,max(EXENDTC) as TR02EDT from ex where EPOCH='TREATMENT 2' group by STUDYID,USUBJID;create table joined as select a.*,b.TR01SDT,c.TR01EDT,d.TR02SDT,e.TR02EDT,f.EXTRT as TRT01A,g.EXTRT as TRT02A from dm aleft join starts1 b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJIDleft join ends1 c on a.STUDYID=c.STUDYID and a.USUBJID=c.USUBJIDleft join starts2 d on a.STUDYID=d.STUDYID and a.USUBJID=d.USUBJIDleft join ends2 e on a.STUDYID=e.STUDYID and a.USUBJID=e.USUBJIDleft join firstex f on a.STUDYID=f.STUDYID and a.USUBJID=f.USUBJID and f.EPOCH='TREATMENT 1'left join firstex g on a.STUDYID=g.STUDYID and a.USUBJID=g.USUBJID and g.EPOCH='TREATMENT 2'; quit;data result; set joined; WASHDUR=.; if not missing(TR01EDT) and not missing(TR02SDT) then WASHDUR=max(0,input(TR02SDT,yymmdd10.)-input(TR01EDT,yymmdd10.)-1); run;proc export data=result(keep=STUDYID USUBJID TR01SDT TR01EDT TR02SDT TR02EDT TRT01A TRT02A WASHDUR) outfile="/app/output/adsl.csv" dbms=csv replace; run;