As a clinical trial statistician or programmer, answer three short questions in Your review: the task, the inputs and outputs, and the AI-assisted review. Agreeing with all three approves this benchmark; disagreeing with any needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.
We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.
Summary
README.mdPublic entries: Python and R. Passing coverage requires an installed inventory.
Goal: write one best overall response record for each subject,
carrying the randomization date (RANDDT) through and adding the
response (AVALC), its rank (AVAL), and its supporting date
(ADT).
Input: subject-level records with randomization date (RANDDT),
plus a prepared response selection in which the record with
BORSEQ equal to 1 holds the subject's winning category
(BORCAT) and date (ADT).
Variables:
AVALCis the best overall response: complete response (CR), partial response (PR), stable disease (SD), neither complete response nor progressive disease (NON-CR/NON-PD), progressive disease (PD), or not evaluable (NE); empty when the subject has no record withBORSEQequal to1.AVALranks that response as1(complete response),2(partial response),3(stable disease),4(neither complete response nor progressive disease),5(progressive disease), or6(not evaluable); empty whenAVALCis empty.ADTis the analysis date supporting the response, taken from that record's date; empty wheneverAVALCis.
Note: every subject in the subject-level records gets one
record, so a subject with no record with BORSEQ equal to 1
keeps its randomization date with response, rank, and date all
empty. The supporting date is never before the randomization
date; it may fall exactly on it.
Inputs
Under review2 source filesinput/adrs_selection.csv
Edit12 rows| STUDYID | USUBJID | ASEQ | ADT | RANDDY | AVALC | BORCAT | BORPRI | BORSEQ |
|---|---|---|---|---|---|---|---|---|
| YAMAA-01 | 1 | 2025-02-17 | 42 | PR | PR | 2 | 2 | |
| YAMAA-01 | 2 | 2025-04-07 | 91 | CR | CR | 1 | 1 | |
| YAMAA-01 | 1 | 2025-02-10 | 28 | SD | NE | 6 | 3 | |
| YAMAA-01 | 2 | 2025-04-14 | 91 | SD | SD | 3 | 1 | |
| YAMAA-01 | 3 | 2025-05-12 | 119 | SD | SD | 3 | 2 | |
| YAMAA-01 | 1 | 2025-02-10 | 21 | SD | NE | 6 | 2 | |
| YAMAA-01 | 2 | 2025-03-03 | 42 | PD | PD | 5 | 1 | |
| YAMAA-01 | 1 | 2025-02-24 | 28 | SD | NE | 6 | 1 | |
| YAMAA-01 | 2 | 2025-04-07 | 70 | NE | NE | 6 | 2 | |
| YAMAA-01 | 1 | 2025-03-31 | 49 | NON-CR/NON-PD | NON-CR/NON-PD | 4 | 1 | |
| YAMAA-01 | 1 | 2025-03-24 | 35 | PR | PR | 2 | 1 | |
| YAMAA-01 | 1 | 2025-02-24 | 1 | CR | CR | 1 | 1 |
input/adsl.csv
Edit8 rows| STUDYID | USUBJID | RANDDT |
|---|---|---|
| YAMAA-01 | 2025-01-06 | |
| YAMAA-01 | 2025-01-13 | |
| YAMAA-01 | 2025-01-20 | |
| YAMAA-01 | 2025-01-27 | |
| YAMAA-01 | 2025-02-03 | |
| YAMAA-01 | 2025-02-10 | |
| YAMAA-01 | 2025-02-17 | |
| YAMAA-01 | 2025-02-24 |
Expected output
Under review8 expected rowsexpected/adrs.csv
Edit8 rows| STUDYID | USUBJID | PARAMCD | PARAM | RANDDT | AVALC | AVAL | ADT |
|---|---|---|---|---|---|---|---|
| YAMAA-01 | BOR | Best Overall Response by Investigator | 2025-01-06 | CR | 1 | 2025-04-07 | |
| YAMAA-01 | BOR | Best Overall Response by Investigator | 2025-01-13 | SD | 3 | 2025-04-14 | |
| YAMAA-01 | BOR | Best Overall Response by Investigator | 2025-01-20 | PD | 5 | 2025-03-03 | |
| YAMAA-01 | BOR | Best Overall Response by Investigator | 2025-01-27 | NE | 6 | 2025-02-24 | |
| YAMAA-01 | BOR | Best Overall Response by Investigator | 2025-02-03 | — | — | — | |
| YAMAA-01 | BOR | Best Overall Response by Investigator | 2025-02-10 | NON-CR/NON-PD | 4 | 2025-03-31 | |
| YAMAA-01 | BOR | Best Overall Response by Investigator | 2025-02-17 | PR | 2 | 2025-03-24 | |
| YAMAA-01 | BOR | Best Overall Response by Investigator | 2025-02-24 | CR | 1 | 2025-02-24 |
Human review and assessment
| Dataset Name | Variable Name | Variable Label | Type | Codelist/Controlled Terms | Core | Computational Method | Origin |
|---|---|---|---|---|---|---|---|
| ADRS | STUDYID | Study Identifier | character | Copy value from ADSL.STUDYID. | Predecessor | ||
| ADRS | USUBJID | Unique Subject Identifier | character | Copy value from ADSL.USUBJID. | Predecessor | ||
| ADRS | PARAMCD | Parameter Code | character | Set constant value "BOR". | Assigned | ||
| ADRS | PARAM | Parameter | character | Set constant value "Best Overall Response by Investigator". | Assigned | ||
| ADRS | RANDDT | Randomization Date | numeric | ISO 8601 | Copy value from ADSL.RANDDT. | Predecessor | |
| ADRS | AVALC | Analysis Value (C) | character | CR, PR, SD, NON-CR/NON-PD, PD, NE | Take BORCAT from the ADRSSEL record matching on STUDYID and USUBJID, where ADRSSEL.BORSEQ = 1, and no match yields blank. | Derived | |
| ADRS | AVAL | Analysis Value | numeric | Recode AVALC: "CR" → "1", "PR" → "2", "SD" → "3", "NON-CR/NON-PD" → "4", "PD" → "5", "NE" → "6"; missing values stay missing; unlisted values are errors. | Derived | ||
| ADRS | ADT | Analysis Date | numeric | ISO 8601 | Take ADT from the ADRSSEL record matching on STUDYID and USUBJID, where ADRSSEL.BORSEQ = 1, and no match yields blank. | Derived |
| Variable | Permitted values |
|---|---|
| AVALC | CR, PR, SD, NON-CR/NON-PD, PD, NE |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.7 | 2026-10-20 | yamaa docs build | Read a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default | ||
| 1.6 | 2026-10-17 | yamaa docs build | Describe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight | ||
| 1.5 | 2026-10-17 | yamaa docs build | Name collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part | ||
| 1.4 | 2026-10-15 | yamaa docs build | State datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks | ||
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/adam-adrs-best-response/full.md
Edit28 linesFollowing CDISC ADaM standards, use the provided ADRS_SELECTION and ADSL datasets to create an ADRS dataset with one record per subject.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, RANDDT, AVALC, AVAL, ADT
- PARAMCD is "BOR" and PARAM is "Best Overall Response by Investigator" on every record.
- Every subject in ADSL has a record, even when the subject has no ADRS_SELECTION record with BORSEQ 1.
- AVALC is the category in BORCAT on the subject's ADRS_SELECTION record with BORSEQ 1: "CR" for complete response, "PR" for partial response, "SD" for stable disease, "NON-CR/NON-PD" for neither complete response nor progressive disease, "PD" for progressive disease, and "NE" for not evaluable.
- AVALC has no value when the subject has no ADRS_SELECTION record with BORSEQ 1.
- AVAL is 1 when AVALC is "CR", 2 when AVALC is "PR", 3 when AVALC is "SD", 4 when AVALC is "NON-CR/NON-PD", 5 when AVALC is "PD", and 6 when AVALC is "NE".
- AVAL has no value when AVALC has no value.
- ADT is the analysis date supporting the response, taken from the date of the subject's ADRS_SELECTION record with BORSEQ 1.
- ADT has no value whenever AVALC has no value.
- ADT is never before RANDDT, and it may fall exactly on RANDDT.
Read the source datasets from /app/input and save the completed dataset as /app/output/adrs.csv.
prompts/adam-adrs-best-response/conventions.md
Edit17 linesFollowing CDISC ADaM standards, use the provided ADRS_SELECTION and ADSL datasets to create an ADRS dataset with one record per subject.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, RANDDT, AVALC, AVAL, ADT
- PARAMCD is "BOR" and PARAM is "Best Overall Response by Investigator" on every record.
- AVALC is "CR" for complete response, "PR" for partial response, "SD" for stable disease, "NON-CR/NON-PD" for neither complete response nor progressive disease, "PD" for progressive disease, or "NE" for not evaluable, or has no value.
- AVAL is 1 for "CR", 2 for "PR", 3 for "SD", 4 for "NON-CR/NON-PD", 5 for "PD", and 6 for "NE", or has no value.
Read the source datasets from /app/input and save the completed dataset as /app/output/adrs.csv.
prompts/adam-adrs-best-response/brief.md
Edit8 linesFollowing CDISC ADaM standards, use the provided ADRS_SELECTION and ADSL datasets to create an ADRS dataset with one record per subject.
The output dataset should contain the following columns in this order: STUDYID, USUBJID, PARAMCD, PARAM, RANDDT, AVALC, AVAL, ADT
Read the source datasets from /app/input and save the completed dataset as /app/output/adrs.csv.
R
Edit3 lineslibrary(yamaa)yamaa_spec("spec.yaml")$run()$outputPython
Edit3 linesimport yamaayamaa.spec("spec.yaml").run().outputsolutions/adam-adrs-best-response/result.R
Edit51 lines# Reference solution for the yamaa benchmark adam-adrs-best-response (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)adsl <- read_csv( "/app/input/adsl.csv", col_types = cols(RANDDT = col_date(), .default = col_character()))selection <- read_csv( "/app/input/adrs_selection.csv", col_types = cols( ASEQ = col_integer(), ADT = col_date(), RANDDY = col_integer(), BORPRI = col_integer(), BORSEQ = col_integer(), .default = col_character() ))# The best overall response is the selection record numbered 1.best <- selection |> filter(BORSEQ == 1L) |> select(USUBJID, BORCAT, BESTDT = ADT)adrs <- adsl |> left_join(best, by = "USUBJID") |> mutate( PARAMCD = "BOR", PARAM = "Best Overall Response by Investigator", AVALC = BORCAT, AVAL = case_when( AVALC == "CR" ~ 1L, AVALC == "PR" ~ 2L, AVALC == "SD" ~ 3L, AVALC == "NON-CR/NON-PD" ~ 4L, AVALC == "PD" ~ 5L, AVALC == "NE" ~ 6L ), ADT = BESTDT ) |> select(STUDYID, USUBJID, PARAMCD, PARAM, RANDDT, AVALC, AVAL, ADT) |> arrange(USUBJID)dir.create("/app/output", showWarnings = FALSE)write_csv(adrs, "/app/output/adrs.csv", na = "")solutions/adam-adrs-best-response/result.py
Edit53 lines# Reference solution for the yamaa benchmark adam-adrs-best-response (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.import csvAVAL_BY_AVALC = { "CR": "1", "PR": "2", "SD": "3", "NON-CR/NON-PD": "4", "PD": "5", "NE": "6",}with open("/app/input/adsl.csv", newline="") as f: adsl = list(csv.DictReader(f))with open("/app/input/adrs_selection.csv", newline="") as f: selection = list(csv.DictReader(f))best = {}for row in selection: if row["BORSEQ"] == "1": best[row["USUBJID"]] = rowout = []for subj in adsl: usubjid = subj["USUBJID"] sel = best.get(usubjid) avalc = sel["BORCAT"] if sel else "" aval = AVAL_BY_AVALC[avalc] if avalc else "" adt = sel["ADT"] if sel else "" out.append( { "STUDYID": subj["STUDYID"], "USUBJID": usubjid, "PARAMCD": "BOR", "PARAM": "Best Overall Response by Investigator", "RANDDT": subj["RANDDT"], "AVALC": avalc, "AVAL": aval, "ADT": adt, } )fields = ["STUDYID", "USUBJID", "PARAMCD", "PARAM", "RANDDT", "AVALC", "AVAL", "ADT"]with open("/app/output/adrs.csv", "w", newline="") as f: writer = csv.DictWriter(f, fieldnames=fields) writer.writeheader() writer.writerows(out)solutions/adam-adrs-best-response/result.sas
Edit28 lines/* opensas reference solution for the yamaa benchmark adam-adrs-best-response. Reads the benchmark inputs and derives the requested output. Execute with the existing opensas runtime: opensas /app/output/result.sas. */data adrs_selection; length STUDYID USUBJID ADT RANDDY AVALC BORCAT BORPRI $1024; infile "/app/input/adrs_selection.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. ASEQ ADT : $char1024. RANDDY : $char1024. AVALC : $char1024. BORCAT : $char1024. BORPRI : $char1024. BORSEQ;run;data adsl; length STUDYID USUBJID RANDDT $1024; infile "/app/input/adsl.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. RANDDT : $char1024.;run;proc sql; create table joined as select a.*,b.BORCAT as AVALC,b.ADT from adsl as a left join adrs_selection as b on a.STUDYID=b.STUDYID and a.USUBJID=b.USUBJID and b.BORSEQ=1; quit;data result; length PARAMCD $8 PARAM $40; set joined; PARAMCD='BOR'; PARAM='Best Overall Response by Investigator'; select(AVALC); when('CR') AVAL=1; when('PR') AVAL=2; when('SD') AVAL=3; when('NON-CR/NON-PD') AVAL=4; when('PD') AVAL=5; when('NE') AVAL=6; otherwise AVAL=.; end; if missing(AVALC) then ADT='';run;proc sql; create table final as select STUDYID, USUBJID, PARAMCD, PARAM, RANDDT, AVALC, AVAL, ADT from result;quit;proc export data=final outfile="/app/output/adrs.csv" dbms=csv replace; run;
yamaa