Reject a Project Routine Whose Own Test Case Fails

Review ask

As a clinical trial statistician or programmer, answer two short questions in Your review: the task and the inputs. Agreeing with both approves this benchmark; disagreeing with either needs a comment. Propose a fix with the Edit button on any source file, or add it to your comments.

We suggest spending no more than 5 minutes on a review. The other tabs beside Your review are background for assessment, not under review.

Summary

EditUnder reviewWhat this benchmark means

README.md

Public entries: Python and R. Passing coverage requires an installed inventory.

Goal: carry the source identifier through as ID and return the project routine's result as RESULT.

Input: a source file with ID and VALUE, where VALUE holds the numeric value passed to the routine.

Variables:

ID is the source identifier, carried through unchanged.
RESULT would be the result of passing the source VALUE to the project_value routine, which returns the value it is given.

Note: before any data is read, the routine is run on the test cases written beside it. Its ordinary case expects 15.0 for an input of 1.5, while the routine returns 1.5, so either the routine or its stated expectation is wrong. The run is rejected before any data is read, and no artifact is accepted.

How to fix

Decide which side is wrong. The routine is meant to return its argument unchanged, so here the expected result was mistyped; correct the test case:

- id: ordinary
  covers:
  - normal
  - numeric-comparison
  args:
    x: 1.5
  result: 1.5

When the expected result is right, fix the routine in both host packages instead. Never delete a failing test case to make the run pass.

Expected failure

Editthe run is rejected; no artifact is accepted

expected/error.yaml

phase: validation
condition: function_conformance_failed
rule: FUNC-12

expected/error.yaml

phase: validation
condition: function_conformance_failed
spec_paths:
  - tests[0]
rule: FUNC-12
context:
  function: project_value
  case: ordinary
  environment_path: functions[0]

YAML specification

1 spec file

spec.yaml

Edit27 lines

1yamaa: "1.0"2type: spec3model: ADaM4domain: TEST5keys: [ID]6input:7  SOURCE:8    path: input/source.csv9    types: {VALUE: float}1011output:12  path: test.csv13  columns: [ID, RESULT]1415columns:16  - name: ID17    type: str18    label: Identifier19    derivation: SOURCE.ID2021  - name: RESULT22    type: float23    label: Project Function Result24    derivation:25      function:26        name: project_value27        args: {x: SOURCE.VALUE}

Inputs

Under review1 source file

input/source.csv

Edit2 rows

input/source.csv
ID	VALUE
01	1.5
02	0.0

Unexpected Output

Under reviewNo artifact is produced

No fixture files in this directory.

Shaded columns are derived— = missing value

Human review and assessment

Variable Name	Variable Label	Type	Length	Controlled Terms or Format	Origin	Core	Conversion Definition	Variable Type	Variable Order	Comments for Define
ID	Identifier	Char			Collected		Copy value from SOURCE.ID.	SDTM	1
RESULT	Project Function Result	Num			Derived		Call project function project_value with x = SOURCE.VALUE.	SDTM	2

Version	Date	Author	Description
1.7	2026-10-20	yamaa docs build	Read a specification in the 1.0 surface as its 0.5 counterpart; state that missing values stay missing where a mapping declares no missing handler, and leave out a declared missing answer that restates a handler's default
1.6	2026-10-17	yamaa docs build	Describe collected date parts written as ISO 8601 date text and times of day counted in seconds since midnight
1.5	2026-10-17	yamaa docs build	Name collected precision in words instead of letter codes; state imputation fallbacks, minimum source precision, bounds ignored outside the collected period, and invalid text with no handler as an error; describe date and datetime shifts, imputation flag codes, and partial text giving no date part
1.4	2026-10-15	yamaa docs build	State datetime precision codes through the hour and minute, and complete only the time parts a collected datetime lacks
1.3	2026-09-29	yamaa docs build	Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.2	2026-09-29	yamaa docs build	Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.1	2026-09-29	yamaa docs build	Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0		yamaa docs build	Initial generation from spec.yaml

How to fix

input/source.csv

Your review

Mapping spec