yamaa As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.
We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.
Summary
EditUnder reviewWhat this benchmark meansREADME.mdPublic entries: Python and R. Passing coverage requires an installed inventory.
Goal: flag subjects who completed the study, adding COMPLFL
and carrying TRTSDT through.
Input: subject-level records carrying first treatment date
(TRTSDT) when treated, plus disposition records carrying a
category (DSCAT), standardized outcome (DSDECOD), collection
date (DSDTC), and epoch (EPOCH).
Variables:
COMPLFL:Ywhen the subject has a disposition event in the study (FOLLOW-UP) epoch with standardized outcomeCOMPLETED;Notherwise. A subject whose records carry only other outcomes, such asADVERSE EVENT, and a subject with no disposition record at all, are bothN. A subject with two such completion records stops the run rather than being flagged.
Note: the flag answers whether an end-of-study completion
record exists, not whether its collection date was filled in or
how an earlier period ended, so a subject who completed
treatment but discontinued the study is N.
Inputs
Under review2 source filesinput/adsl_raw.csv
Edit8 rows| STUDYID | USUBJID | TRTSDT |
|---|---|---|
| YAMAA-01 | 2024-01-15 | |
| YAMAA-01 | 2024-01-20 | |
| YAMAA-01 | 2024-02-02 | |
| YAMAA-01 | 2024-02-05 | |
| YAMAA-01 | — | |
| YAMAA-01 | 2024-02-12 | |
| YAMAA-01 | 2024-02-18 | |
| YAMAA-01 | 2024-03-01 |
input/ds.csv
Edit9 rows| STUDYID | USUBJID | DSSEQ | DSCAT | DSDECOD | DSDTC | EPOCH |
|---|---|---|---|---|---|---|
| YAMAA-01 | 1 | PROTOCOL MILESTONE | RANDOMIZED | 2024-01-12 | SCREENING | |
| YAMAA-01 | 2 | DISPOSITION EVENT | COMPLETED | 2024-02-10 | TREATMENT | |
| YAMAA-01 | 3 | DISPOSITION EVENT | DISCONTINUED | 2024-03-01 | FOLLOW-UP | |
| YAMAA-01 | 1 | DISPOSITION EVENT | COMPLETED | — | FOLLOW-UP | |
| YAMAA-01 | 1 | DISPOSITION EVENT | ADVERSE EVENT | 2024-02-20 | TREATMENT | |
| YAMAA-01 | 1 | DISPOSITION EVENT | ADVERSE EVENT | 2024-01-05 | FOLLOW-UP | |
| YAMAA-01 | 1 | DISPOSITION EVENT | COMPLETED | 2024-03-10 | FOLLOW-UP | |
| YAMAA-01 | 1 | PROTOCOL MILESTONE | SCREEN FAILURE | 2024-01-08 | SCREENING | |
| YAMAA-01 | 1 | DISPOSITION EVENT | COMPLETED | 2024-04-01 | FOLLOW-UP |
Expected output
Under review8 expected rowsexpected/adsl.csv
Edit8 rows| STUDYID | USUBJID | TRTSDT | COMPLFL |
|---|---|---|---|
| YAMAA-01 | 2024-01-15 | N | |
| YAMAA-01 | 2024-01-20 | Y | |
| YAMAA-01 | 2024-02-02 | N | |
| YAMAA-01 | 2024-02-05 | N | |
| YAMAA-01 | — | N | |
| YAMAA-01 | 2024-02-12 | Y | |
| YAMAA-01 | 2024-02-18 | N | |
| YAMAA-01 | 2024-03-01 | Y |
Human review and assessment
| Dataset Name | Variable Name | Variable Label | Type | Codelist/Controlled Terms | Core | Computational Method | Origin |
|---|---|---|---|---|---|---|---|
| ADSL | STUDYID | Study Identifier | character | Copy value from ADSL_RAW.STUDYID. | Predecessor | ||
| ADSL | USUBJID | Unique Subject Identifier | character | Copy value from ADSL_RAW.USUBJID. | Predecessor | ||
| ADSL | TRTSDT | Date of First Exposure to Treatment | numeric | ISO 8601 | Copy value from ADSL_RAW.TRTSDT. | Predecessor | |
| ADSL | COMPLFL | Completers Population Flag | character | Y, N | "Y" when COMPREC IS NOT NULL; "N" otherwise; "N" when the condition is unknown. | Derived |
| Variable | Permitted values |
|---|---|
| COMPLFL | Y, N |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.7 | 2026-10-20 | yamaa docs build | Read a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default | ||
| 1.6 | 2026-10-17 | yamaa docs build | Describe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight | ||
| 1.5 | 2026-10-17 | yamaa docs build | Name collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part | ||
| 1.4 | 2026-10-15 | yamaa docs build | State datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks | ||
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/adam-adsl-completion-flag/full.md
Edit17 linesFollowing CDISC ADaM standards, use the provided ADSL_RAW and DS datasets to create an ADSL dataset with one record per subject.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, TRTSDT, COMPLFL
COMPLFL is "Y" when the subject has a disposition record with category "DISPOSITION EVENT" in the "FOLLOW-UP" epoch whose standardized outcome is "COMPLETED", and "N" otherwise. The completion record counts whether or not its collection date was recorded. A "COMPLETED" outcome in an earlier epoch does not count, so a subject who completed treatment but discontinued the study has COMPLFL "N". A subject with no disposition record at all, or whose disposition records carry only other outcomes such as "ADVERSE EVENT", has COMPLFL "N".
Read the source datasets from /app/input and save the completed dataset as /app/output/adsl.csv.
prompts/adam-adsl-completion-flag/conventions.md
Edit10 linesFollowing CDISC ADaM standards, use the provided ADSL_RAW and DS datasets to create an ADSL dataset with one record per subject.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, TRTSDT, COMPLFL
COMPLFL is "Y" or "N".
Read the source datasets from /app/input and save the completed dataset as /app/output/adsl.csv.
prompts/adam-adsl-completion-flag/brief.md
Edit8 linesFollowing CDISC ADaM standards, use the provided ADSL_RAW and DS datasets to create an ADSL dataset with one record per subject.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, TRTSDT, COMPLFL
Read the source datasets from /app/input and save the completed dataset as /app/output/adsl.csv.
R
Edit3 lineslibrary(yamaa)yamaa_spec("spec.yaml")$run()$outputPython
Edit3 linesimport yamaayamaa.spec("spec.yaml").run().outputsolutions/adam-adsl-completion-flag/result.R
Edit34 lines# Reference solution for the yamaa benchmark adam-adsl-completion-flag (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)adsl_raw <- read_csv( "/app/input/adsl_raw.csv", col_types = cols(TRTSDT = col_date(), .default = col_character()))ds <- read_csv( "/app/input/ds.csv", col_types = cols(.default = col_character()))completed <- ds |> filter( DSCAT %in% "DISPOSITION EVENT", EPOCH %in% "FOLLOW-UP", DSDECOD %in% "COMPLETED" ) |> distinct(STUDYID, USUBJID) |> mutate(.completed = "Y")adsl <- adsl_raw |> left_join(completed, by = c("STUDYID", "USUBJID")) |> mutate(COMPLFL = if_else(!is.na(.completed), "Y", "N")) |> select(STUDYID, USUBJID, TRTSDT, COMPLFL)dir.create("/app/output", showWarnings = FALSE)write_csv(adsl, "/app/output/adsl.csv", na = "")solutions/adam-adsl-completion-flag/result.py
Edit51 lines# Reference solution for the yamaa benchmark adam-adsl-completion-flag (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1."""Create ADSL with a completion flag from ADSL_RAW and DS.COMPLFL is "Y" when the subject has a disposition record with category"DISPOSITION EVENT" in the "FOLLOW-UP" epoch whose standardized outcome(DSDECOD) is "COMPLETED"; the record counts whether or not itscollection date (DSDTC) was recorded. Otherwise COMPLFL is "N"."""import csvINPUT_DIR = "/app/input"OUTPUT_PATH = "/app/output/adsl.csv"def main(): with open(f"{INPUT_DIR}/adsl_raw.csv", newline="") as f: subjects = list(csv.DictReader(f)) completed = set() with open(f"{INPUT_DIR}/ds.csv", newline="") as f: for row in csv.DictReader(f): if ( row["DSCAT"] == "DISPOSITION EVENT" and row["EPOCH"] == "FOLLOW-UP" and row["DSDECOD"] == "COMPLETED" ): completed.add((row["STUDYID"], row["USUBJID"])) with open(OUTPUT_PATH, "w", newline="") as f: writer = csv.writer(f) writer.writerow(["STUDYID", "USUBJID", "TRTSDT", "COMPLFL"]) for subject in subjects: key = (subject["STUDYID"], subject["USUBJID"]) writer.writerow( [ subject["STUDYID"], subject["USUBJID"], subject["TRTSDT"], "Y" if key in completed else "N", ] )if __name__ == "__main__": main()solutions/adam-adsl-completion-flag/result.sas
Edit20 lines/* opensas reference solution for the yamaa benchmark adam-adsl-completion-flag. Reads the benchmark inputs and derives the requested output. Execute with the existing opensas runtime: opensas /app/output/result.sas. */data adsl_raw; length STUDYID USUBJID TRTSDT $1024; infile "/app/input/adsl_raw.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. TRTSDT : $char1024.;run;data ds; length STUDYID USUBJID DSCAT DSDECOD DSDTC EPOCH $1024; infile "/app/input/ds.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. DSSEQ DSCAT : $char1024. DSDECOD : $char1024. DSDTC : $char1024. EPOCH : $char1024.;run;proc sql;create table completed as select distinct STUDYID,USUBJID from ds where EPOCH='FOLLOW-UP' and DSDECOD='COMPLETED';create table result as select a.*,case when not missing(b.USUBJID) then 'Y' else 'N' end as COMPLFL from adsl_raw a left join completed b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID; quit;proc export data=result(keep=STUDYID USUBJID TRTSDT COMPLFL) outfile="/app/output/adsl.csv" dbms=csv replace; run;