As a clinical trial statistician or programmer, challenge anything in this benchmark: describe a problem in Comments, or propose a fix with the Edit button on any source file.
- Task: Would a real study need this derivation, specified this way? Does it follow CDISC standards: variable names, controlled terms, record structure? Flag any rule that is ambiguous or that a sponsor would handle differently.
- Inputs: Do they look like data collected in a real trial, and do they include the situations you would expect to meet in one?
- Expected output: Would independent QC programming reproduce every value from these inputs and rules?
The other tabs beside Comments are background for assessment, not under review.
Summary
README.mdGoal: build one record per clinical site for a Bioresearch Monitoring
(BIMO) submission, with the site identifier (SITEID), the assigned
treatment arm (ARM), the number of treated subjects (SAFPOP), and the
number of enrolled subjects (ENRLPOP).
Input: one row per subject carrying the study identifier, the unique subject identifier, the site key, the subject's treatment arm, and the treated flag.
Variables:
SITEIDis the site key exactly as collected, leading zeros kept, so a site key of007and a site key of7are different sites.ARMis the alphabetically first arm among the site's treated subjects; it has no value when the site has no treated subjects.SAFPOPis the number of treated subjects at the site; it is zero for a site whose subjects were all untreated.ENRLPOPis the number of enrolled subjects at the site, treated or not.
Note: a site appears in the dataset exactly when at least one subject row names it. A site with only untreated subjects still gets a record, with no arm value and a treated count of zero, while a site named by no subject row gets no record at all.
Inputs
Under review1 source fileinput/adsl.csv
Edit11 rows| STUDYID | USUBJID | SITENUM | ARM | SAFFL |
|---|---|---|---|---|
| YAMAA-01 | 007 | Placebo | Y | |
| YAMAA-01 | 007 | Drug A | Y | |
| YAMAA-01 | 007 | Drug A | Y | |
| YAMAA-01 | 7 | Drug A | N | |
| YAMAA-01 | 7 | Drug A | N | |
| YAMAA-01 | 012 | Drug A | Y | |
| YAMAA-01 | 012 | Drug A | Y | |
| YAMAA-01 | 3 | Placebo | Y | |
| YAMAA-01 | 104 | Drug A | N | |
| YAMAA-01 | 104 | Drug A | N | |
| YAMAA-01 | 104 | Placebo | Y |
Expected output
Under review5 expected rowsexpected/clinsite.csv
Edit5 rows| STUDYID | SITEID | ARM | SAFPOP | ENRLPOP |
|---|---|---|---|---|
| YAMAA-01 | 007 | Drug A | 3 | 3 |
| YAMAA-01 | 012 | Drug A | 2 | 2 |
| YAMAA-01 | 104 | Placebo | 1 | 3 |
| YAMAA-01 | 3 | Placebo | 1 | 1 |
| YAMAA-01 | 7 | — | 0 | 2 |
Comments and assessment materials
| Variable Name | Variable Label | Type | Length | Controlled Terms or Format | Origin | Core | Conversion Definition | Variable Type | Variable Order | Comments for Define |
|---|---|---|---|---|---|---|---|---|---|---|
| STUDYID | Study Identifier | Char | Collected | Copy value from SUBJ.STUDYID. | SDTM | 1 | ||||
| SITEID | Study Site Identifier | Char | Collected | Copy value from SUBJ.SITENUM. | SDTM | 2 | ||||
| ARM | Assigned Treatment Arm at the Site | Char | Derived | Set per constructed row; see the Row construction sheet. | SDTM | 3 | ||||
| SAFPOP | Number of Treated Subjects at the Site | Num | Derived | First non-missing of SAFPOP_RAW; all missing → "0". | SDTM | 4 | ||||
| ENRLPOP | Number of Enrolled Subjects at the Site | Num | Derived | Set per constructed row; see the Row construction sheet. | SDTM | 5 |
| Template | Include when | ARM | SAFPOP_RAW | ENRLPOP |
|---|---|---|---|---|
| site | Aggregate MIN(SUBJ.ARM) over records matching on the shared output keys, where SUBJ.SAFFL = 'Y' | Aggregate COUNT(SUBJ.USUBJID) over records matching on the shared output keys, where SUBJ.SAFFL = 'Y' | Aggregate COUNT(SUBJ.USUBJID) over records matching on the shared output keys |
| Version | Date | Author | Description | Reviewer | Sign-off |
|---|---|---|---|---|---|
| 1.3 | 2026-09-29 | yamaa docs build | Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata | ||
| 1.2 | 2026-09-29 | yamaa docs build | Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived | ||
| 1.1 | 2026-09-29 | yamaa docs build | Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML | ||
| 1.0 | yamaa docs build | Initial generation from spec.yaml |
prompts/bimo-clinsite-site-level/full.md
Edit23 linesFollowing CDISC standards, use the provided SUBJ dataset to create a CLINSITE dataset for a Bioresearch Monitoring (BIMO) submission package, with one record per clinical site.
The output dataset should contain the following columns in this order: STUDYID, SITEID, ARM, SAFPOP, ENRLPOP
SITEID is the site key (SITENUM) exactly as collected, with leading zeros kept: '007' and '7' are different sites.
ARM is the alphabetically first arm among the site's treated subjects; it has no value when the site has no treated subjects.
SAFPOP is the number of treated subjects at the site; ENRLPOP is the number of enrolled subjects at the site, treated or not. A treated subject is one whose treated flag (SAFFL) is 'Y'.
A site appears in the dataset exactly when at least one subject row names it. A site with only untreated subjects still gets a record, with no ARM value and SAFPOP of zero; a site named by no subject row gets no record.
Read the source datasets from /app/input and save the completed dataset as /app/output/clinsite.csv.
prompts/bimo-clinsite-site-level/conventions.md
Edit16 linesFollowing CDISC standards, use the provided SUBJ dataset to create a CLINSITE dataset for a Bioresearch Monitoring (BIMO) submission package, with one record per clinical site.
The output dataset should contain the following columns in this order: STUDYID, SITEID, ARM, SAFPOP, ENRLPOP
SITEID keeps leading zeros: '007' and '7' are different sites.
ARM is the alphabetically first arm among the site's treated subjects, or has no value.
SAFPOP counts treated subjects ('Y'); ENRLPOP counts all enrolled subjects.
Read the source datasets from /app/input and save the completed dataset as /app/output/clinsite.csv.
prompts/bimo-clinsite-site-level/brief.md
Edit9 linesFollowing CDISC standards, use the provided SUBJ dataset to create a CLINSITE dataset for a Bioresearch Monitoring (BIMO) submission package, with one record per clinical site.
The output dataset should contain the following columns in this order: STUDYID, SITEID, ARM, SAFPOP, ENRLPOP
Read the source datasets from /app/input and save the completed dataset as /app/output/clinsite.csv.
run.R
Edit4 lineslibrary(yamaa)clinsite <- yamaa_domain("spec.yaml")$outputclinsiterun.py
Edit4 linesimport yamaaclinsite = yamaa.yamaa_domain("spec.yaml").outputclinsitesolutions/bimo-clinsite-site-level/result.R
Edit37 lines# Reference solution for the yamaa benchmark bimo-clinsite-site-level (R track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.library(dplyr, warn.conflicts = FALSE)library(readr)subj <- read_csv( "/app/input/adsl.csv", col_types = cols(.default = col_character()))clinsite <- subj |> # One record per site; the site key stays text so leading zeros survive. group_by(STUDYID, SITENUM) |> summarise( # The alphabetically first arm among the site's treated subjects. # A blank (missing) SAFFL counts as not "Y" so it can neither enter # the treated arm list nor turn the treated count missing. ARM = { treated <- !is.na(SAFFL) & SAFFL == "Y" treated_arms <- ARM[treated] if (length(treated_arms)) min(treated_arms) else NA_character_ }, # Treated subjects, then all enrolled subjects, at the site. SAFPOP = sum(!is.na(SAFFL) & SAFFL == "Y"), ENRLPOP = n(), .groups = "drop" ) |> rename(SITEID = SITENUM) |> select(STUDYID, SITEID, ARM, SAFPOP, ENRLPOP) |> arrange(SITEID)dir.create("/app/output", showWarnings = FALSE)write_csv(clinsite, "/app/output/clinsite.csv", na = "")solutions/bimo-clinsite-site-level/result.py
Edit29 lines# Reference solution for the yamaa benchmark bimo-clinsite-site-level (Python track).## Written from the benchmark's full prompt and its inputs alone, as an agent# would write it. Harbor's oracle agent runs it to check that the prompt can# be solved and that the grader scores a correct answer 1.from pathlib import Pathimport polars as plsubj = pl.read_csv("/app/input/adsl.csv", infer_schema=False, null_values="")clinsite = ( # One record per site; the site key stays text so leading zeros survive. subj.group_by("STUDYID", "SITENUM", maintain_order=True) .agg( # The alphabetically first arm among the site's treated subjects. pl.col("ARM").filter(pl.col("SAFFL") == "Y").min().alias("ARM"), # Treated subjects, then all enrolled subjects, at the site. (pl.col("SAFFL") == "Y").sum().alias("SAFPOP"), pl.len().alias("ENRLPOP"), ) .rename({"SITENUM": "SITEID"}) .select("STUDYID", "SITEID", "ARM", "SAFPOP", "ENRLPOP") .sort("SITEID"))Path("/app/output").mkdir(parents=True, exist_ok=True)clinsite.write_csv("/app/output/clinsite.csv", null_value="")solutions/bimo-clinsite-site-level/result.sas
Edit19 lines/* opensas reference solution for the yamaa benchmark bimo-clinsite-site-level. Reads the benchmark inputs and derives the requested output. Execute with the existing opensas runtime: opensas /app/output/result.sas. */data adsl; length STUDYID USUBJID SITENUM ARM SAFFL $1024; infile "/app/input/adsl.csv" dsd dlm="," firstobs=2 truncover lrecl=32767; input STUDYID : $char1024. USUBJID : $char1024. SITENUM : $char1024. ARM : $char1024. SAFFL : $char1024.;run;proc sql;create table counts as select STUDYID,SITENUM,count(*) as ENRLPOP,sum(case when SAFFL='Y' then 1 else 0 end) as SAFPOP from adsl group by STUDYID,SITENUM;create table arms as select STUDYID,SITENUM,min(ARM) as ARM from adsl where SAFFL='Y' group by STUDYID,SITENUM;create table result as select a.STUDYID,a.SITENUM as SITEID,b.ARM,a.SAFPOP,a.ENRLPOP from counts a left join arms b on a.STUDYID=b.STUDYID and a.SITENUM=b.SITENUM;quit;proc sql; create table final as select STUDYID, SITEID, ARM, SAFPOP, ENRLPOP from result;quit;proc export data=final outfile="/app/output/clinsite.csv" dbms=csv replace; run;
yamaa
Comments
Signed in with GitHub; saved in repository Discussions