Rules at a glance¶

The principles and every rule's title, generated from the rule files. Each title states its rule; the statement a reviewer approves, the detail, conditions and evidence are in the rule's file.

Principles¶

P1. Everything is declared. Columns, types, keys, joins, row construction, order and checks are written in the specification. Nothing is inferred from data, from file contents, from a same-named field or from a host default. Enforced by SPEC-4, INPUT-5, OUTPUT-5 and INPUT-1.

P2. Values never convert silently. An operation never converts its inputs. A value converts once, to its column's declared type, by one table, and a failed conversion is a named failure. Enforced by VALUE-2, VALUE-3 and INPUT-7.

P3. Missing is a value. Missing is distinct from every string, number and the empty string; no text spells it, every type admits it, and every operation states what it does with it. Enforced by VALUE-25, FORMULA-4, COLUMN-10, FORMULA-24 and INPUT-8.

P4. Ambiguity fails. Two values for one key, several matching records without a declared selection, an unmapped value or text that does not parse fails with a named condition instead of choosing. Enforced by COLUMN-2, LOOKUP-10, FORMULA-20 and NAME-13.

P5. Order is data. Stored record order, construction order and declared order terms decide every tie. No operation reorders records or chooses an order of its own. Enforced by ROW-7, VALUE-23, FORMULA-19 and INPUT-2.

P6. Values are exact. Integers are exact 64-bit values and floats are binary64 computed exactly as written. Nothing is rounded except where a specification declares it. Enforced by VALUE-7, VALUE-9, FORMULA-26 and CSV-8.

P7. Vocabularies are closed. Verbs, grammars, functions, column types, storage profiles and conditions are closed lists. Extending one is a rule change. Enforced by VALUE-1, EXPR-1, FORMULA-18 and COLUMN-15.

P8. Inheritance is deterministic. Layers compose in one fixed order, and the resolved specification is the same whatever spelling its layers used. Enforced by LAYER-3, LAYER-5 and LAYER-14.

P9. Faults are found before data. A fault that does not depend on data fails during validation, before any input record is read. Enforced by LAYER-12, FILE-11, INPUT-1 and VALUE-24.

P10. Artifacts are reproducible. The same specification and inputs produce the same artifact bytes, and an artifact is published whole or not at all. Enforced by CSV-3, OUTPUT-8 and DEFINE-23.

P11. A package is consistent and traceable. The datasets, transport files, run reports and Define-XML documents a project publishes come from one set of resolved specifications and agree on every fact they share. Every published fact names its source: an identifier derives from declared identifiers, metadata from the specification that built the data, and an origin is declared, never inferred. Enforced by JSON-5, JSON-6, DEFINE-9, SUB-19 and INPUT-11.

Specification structure¶

Declare root fields, unique identifiers, column derivation coverage, internal and copied columns, and ASCII source.

Root fields and identifiers¶

SPEC-1. A specification derives one output dataset from the root fields it declares
SPEC-2. Mapping keys, dataset identifiers, column names and row IDs are each unique

Columns¶

SPEC-3. The artifact's columns are declared, never generated from source fields
SPEC-4. Every declared column is derived, never filled from a same-named source variable
SPEC-5. A column without a column-level derivation is derived in every row template
SPEC-6. A row-local column-level derivation is a default that each row template may override
SPEC-7. A declared column outside output.columns is internal and is evaluated like an output column
SPEC-9. A copy entry declares the listed columns of a schema-linked input under their names, with the producer's type and label

Language source¶

SPEC-8. Language source is ASCII

Inheritance¶

Resolve inherited layers into one minimal, ordered specification.

Layers¶

LAYER-1. A specification is its entry file composed with the chain of parents and the fragments it reaches
LAYER-2. A parent or a fragment is one local YAML file resolved from the layer that names it
LAYER-3. Layers contribute from the root ancestor to the entry, each after its fragments, later layers taking precedence
LAYER-4. Every layer is a valid partial specification of the language version it declares

Composition¶

LAYER-5. Keyed collections compose member by member, the blocks output and submission field by field, and every other root field replaces whole
LAYER-6. A matching member merges recursively: classes by field, mappings by key, scalars and lists by replacement
LAYER-7. Requiredness and defaults apply to the composed member, never to one contribution
LAYER-8. A named window composes by name like every keyed member, and every reference uses the final definition
LAYER-9. YAML null is the missing value, and a layer clears an inherited field only by listing it in clear
LAYER-10. A path keeps its declaring layer's base and authored spelling

The resolved specification¶

LAYER-11. Declarations that cannot affect the artifact or an assertion are pruned
LAYER-12. Name and reference validation applies after pruning
LAYER-13. A layered specification's resolved columns follow dependency order with a stable tie-break
LAYER-14. The resolved specification is canonical

Name binding¶

Resolve input datasets, qualified and bare names, driver reads, and ODM item reads.

Input datasets¶

NAME-1. input declares every dataset a specification reads, each under its own identifier
NAME-2. No input dataset takes the output dataset's name

Variables¶

NAME-3. A qualified name reads a declared dataset or lookup, and a bare name reads an output column
NAME-4. A qualified read of the driver reads its current record, or in a group the group's key
NAME-5. A variable field names exactly one variable, never an expression
NAME-6. A structured source adds handlers and a record filter to one variable

ODM item reads¶

NAME-7. An odm expression reads one collected item named as DATASET.ItemOID
NAME-8. An ODM input binds its eleven fixed text fields by case-folded name, checked at validation
NAME-9. Every ODM input record carries the seven required identifiers
NAME-10. A row's ODM scope is the set of ODM input records the row was built from
NAME-11. An odm expression evaluates only for a row whose ODM scope covers the input it names
NAME-12. An odm read identifies the scope records of its item that meet its OID lists and filter
NAME-13. An odm read gives the identified record's Value, missing for none, and fails for several

Values¶

Define column types, missing values, numbers, text, dates, datetimes and times, conversion, and order.

Column types and conversion¶

VALUE-1. A column declares exactly one of six types, and every type also admits missing
VALUE-2. An operation never converts its inputs, and an input of an incompatible type fails
VALUE-3. A completed result converts to its column's type by one table, or fails
VALUE-24. A derivation whose result type can never convert to its declared type fails validation

Missing values¶

VALUE-25. Missing is one value distinct from every other, and each operation states what it does with it
VALUE-4. A non-finite float becomes missing before any consumer can see it

Numbers¶

VALUE-5. An int is a signed 64-bit integer, and leaving that range fails with integer_overflow
VALUE-6. Numeric text parses by one grammar, and every other spelling fails
VALUE-7. Conversion to int succeeds only for an exactly integral value within range
VALUE-8. An int widens to the nearest binary64, and a number converts to one canonical text
VALUE-9. Values are used unrounded, and presentation rounding never feeds back
VALUE-10. Each arithmetic operator and numeric function returns a fixed numeric type

Text¶

VALUE-11. A string is a sequence of Unicode scalar values, decoded strictly where it enters
VALUE-12. Text is never normalized, and two strings are equal only scalar for scalar

Dates, datetimes and times¶

VALUE-13. A date names one calendar day, a datetime one civil second and a time one second of the day, with no gaps
VALUE-14. A datetime carries no zone, offset or fractional second, and never shifts implicitly
VALUE-15. Text becomes a date, datetime or time only in one zero-padded extended form
VALUE-16. Every temporal value is complete and records its collected precision on one ladder
VALUE-17. A temporal value is written in one canonical text that carries its fields alone
VALUE-18. A date, a datetime and a time never convert into one another or enter arithmetic

Comparison and order¶

VALUE-19. An int and a float compare exactly; every other type compares only with itself
VALUE-20. Strings order by scalar value, position by position
VALUE-21. Values of one temporal type order chronologically, whatever their collected precision
VALUE-22. An order term sorts by its type's order and direction, and missing sorts lowest unless nulls places it
VALUE-23. Rows equal on every order term keep their construction order

Inputs¶

Select input profiles, keep stored record order, and type source fields without inferring values.

Profiles¶

INPUT-1. The input path's extension selects the profile that reads it, and nothing is sniffed

Records¶

INPUT-2. A reader returns records in stored order, and nothing reorders them
INPUT-3. An input may declare a generated field holding each record's one-based stored position
INPUT-4. A generated ordinal must not collide with a stored field or join an ODM input

Field types¶

INPUT-5. Every input field has one type, from its container or its declaration, never from its values
INPUT-6. A types entry may only type an existing field of a source that supplies no type
INPUT-7. Stored text parses into its field's declared type, or fails ingestion before any derivation

Missing values¶

INPUT-8. A null or an empty field is missing, and no other text spells missing
INPUT-9. Each input declares whether a stored empty string is missing or present

Producing specifications¶

INPUT-10. A schema link types a stored artifact from its producing specification without running it
INPUT-11. A linked artifact's fields are exactly its producer's output columns, with their types

Row construction¶

Construct output rows from declared input records or groups, or from the distinct keys of the base records.

Row templates¶

ROW-1. Each row template has a unique id and builds its rows from one declared driver
ROW-2. An intermediate drives rows only when it reads an input and exposes typed fields
ROW-3. A row template without group_by is record-driven and builds one candidate row per driver record
ROW-4. A row template with group_by builds one candidate row per group of driver records, then filters it
ROW-5. A grouped row derivation reads a driver field only as a group key or through an aggregate
ROW-6. A row filter names only columns its row template derives, and only values complete at its phase

Rows and their order¶

ROW-7. Row templates are ordered sections, and each retained driver record or group yields one row

Rows without row templates¶

ROW-8. Without row templates, the output rows are the distinct key combinations of the base records
ROW-9. A root filter selects base records before the distinct-keys step and reads only base fields

Column derivation¶

Sequence a run, order column dependencies, complete each value's lifecycle, handle conditions at their site, and name the phase and rule each condition reports.

Run order¶

COLUMN-1. A run validates, builds rows, derives columns, verifies, orders and publishes, in that order
COLUMN-2. A column derivation yields exactly one value per row and never changes the row count

Dependencies¶

COLUMN-3. Dependencies come from every reference an expression makes, closed grammars included
COLUMN-4. Row derivations follow their template's dependency graph and never read a value only column derivation produces
COLUMN-5. Columns follow declaration order except for causal backward row reads
COLUMN-6. A column dependency cycle fails unless every cycle crosses a causal backward row read
COLUMN-7. Without row templates, a key is computed from one base record and never from a non-key output column

Value lifecycle¶

COLUMN-8. Every derived value is evaluated, converted and handled before anything reads it, then verified
COLUMN-9. A row filter reads candidate values after stages 1 to 3 and never hides a failed derivation

Local handlers¶

COLUMN-10. A local handler answers its condition with a literal, and an omitted handler gives missing for a missing single input or absent record and leaves every other condition fatal
COLUMN-11. absent answers a lookup that finds no record, and missing answers a missing input value
COLUMN-12. A structured source carries its own handlers, and order_by with keep selects one of several records
COLUMN-13. A result wrapper's unconvertible replaces only a failed conversion to the column type
COLUMN-14. Every handler path reports how many records it answered as a report row, default substitutions and zero included

Diagnostics¶

COLUMN-15. A language diagnostic cites the rule that owns its failure; only listed implementation conditions cite none
COLUMN-16. Every condition reports one phase from a closed list, which names the part of the run that decides it

Expressions¶

Register and dispatch expressions, give each one result type, restrict nesting, and copy, select and classify values.

Registry and evaluation¶

EXPR-1. A derivation holds exactly one registered expression keyword, which owns its payload
EXPR-2. An expression yields one value per row, and only windows and aggregates read other rows
EXPR-3. Expressions nest only in case results, key match values and a nearest selection's to
EXPR-14. Every expression has one declared result type function, and every context reads its type

Values and shorthand¶

EXPR-4. A reference or a structured source copies a named variable, and a literal returns its value unchanged
EXPR-5. An expression field reads a bare name as a reference and quoted text as a literal
EXPR-6. An expression that selects a value returns it unchanged, with its collected precision

Conditional selection¶

EXPR-7. case returns the value of its first TRUE branch, then otherwise, else missing
EXPR-8. flag returns a declared value for each of TRUE, FALSE and UNKNOWN
EXPR-9. first_available returns the first source that is not missing
EXPR-10. greatest and least return the largest or smallest non-missing source

Classification¶

EXPR-11. mapping replaces a string source with its dictionary value
EXPR-12. A case-insensitive mapping folds ASCII letters, and its folded keys must be unique
EXPR-15. decode replaces a coded source with its item's decode in a codelist the project file declares
EXPR-13. cut assigns a numeric source to one of its labeled break intervals

Formulas¶

Read the one closed expression language in its predicate, aggregate and expression fields, and evaluate it exactly as written.

The language¶

FORMULA-1. An expression is one text in the closed expression grammar
FORMULA-2. Keywords and the names of functions and reducers are case-insensitive, and a sign binds tightest and OR loosest
FORMULA-3. A literal has a fixed runtime type, an integer literal fits int, and a temporal literal needs its keyword
FORMULA-27. The field an expression is written in fixes its root and the constructs it admits
FORMULA-9. An identifier names a value visible at its site, and any other name fails

Predicates¶

FORMULA-4. A predicate is TRUE, FALSE or UNKNOWN, and only TRUE retains a row or record
FORMULA-5. Two non-missing operands compare only when their runtime types are mutually comparable
FORMULA-6. IN and BETWEEN expand into comparisons, and their NOT forms negate the result
FORMULA-7. LIKE matches text scalar by scalar, case-sensitively, with an optional one-scalar ESCAPE
FORMULA-8. str_contains is the one truth-valued function and searches its source for a portable regex

Arithmetic¶

FORMULA-12. Every operand of arithmetic or of a numeric function must be numeric
FORMULA-13. The numeric functions are a closed list, each with a fixed argument count
FORMULA-14. A missing operand makes arithmetic and numeric functions missing, except NULLIF's second argument
FORMULA-15. Division by zero, an invalid root, logarithm, power or probability, and int overflow fail the run
FORMULA-16. There is one rounding: half away from zero, with near ties treated as ties

Aggregate expressions¶

FORMULA-17. aggregate reduces one relation to one value with a closed reducer expression
FORMULA-18. The reducers are exactly SUM, COUNT, MIN, MAX, MEAN, ONLY, STR_JOIN, MEDIAN, Q1, Q3, QUANTILE, STDDEV, VARIANCE and GEOMEAN
FORMULA-19. SUM adds in relation record order, MEAN divides that SUM by COUNT, and VARIANCE, STDDEV and GEOMEAN add in the same order
FORMULA-20. ONLY returns the value of a group's one record and fails on more than one
FORMULA-21. Reductions do not nest
FORMULA-22. An identifier outside a reduction must name a grouping column
FORMULA-23. A reduction keeps the type it reduces, and arithmetic over reductions is numeric
FORMULA-24. Value reducers skip missing values, and a group with no record reduces to missing

Evaluation¶

FORMULA-26. Expressions evaluate deterministically and exactly as written

Lookup and joins¶

Match declared keys, narrow and select records, answer absence, and scope aggregates to one relation.

The lookup model¶

LOOKUP-25. Every lookup runs the same four steps, match, narrow, select and answer absence, whatever its kind
LOOKUP-9. Match keeps the records whose keys equal the current row's match values and whose bounds hold its range value
LOOKUP-8. Narrow keeps the matched records for which the lookup's filter is TRUE
LOOKUP-10. Select chooses the first or last candidate by order_by, and an aggregate reduces every candidate
LOOKUP-26. A named intermediate's nearest selection chooses the candidate whose date or datetime lies nearest the current row's reference
LOOKUP-28. A named intermediate's pivot exposes each listed value of one donor field as a field of its selection
LOOKUP-24. A lookup with no candidate answers absence, missing by default, unless require_match fails it
LOOKUP-14. Lookup handler counts, failures and reviews report the match in one vocabulary

Keys¶

LOOKUP-5. A key pairs each column of the lookup's relation with a current-row match value
LOOKUP-6. A key match value may be an expression whose static result type is checked before matching

The implicit join¶

LOOKUP-1. A dataset-qualified source reads through the implicit join on the input's declared key
LOOKUP-2. Row construction reads another dataset through the same join, using only values available before column derivation

Named intermediates¶

LOOKUP-3. A named intermediate declares an id, a dataset and a key
LOOKUP-4. A named intermediate must narrow, derive or reshape its dataset
LOOKUP-7. A named intermediate may read SELF, the completed rows of earlier row templates
LOOKUP-11. A lookup-qualified variable reads a column of the selected record
LOOKUP-12. A lookup matches once per row when first read and never changes the row count

Intermediate derivations¶

LOOKUP-15. Intermediate derivations add per-record fields to the donor records
LOOKUP-16. An intermediate derivation may read another intermediate's selection for its donor record
LOOKUP-17. An intermediate's unique verification asserts that no donor column combination repeats, warning instead only where a selection chooses

Aggregates over a relation¶

LOOKUP-18. An aggregate reduces one declared dataset, the constructed output rows, or the current input group
LOOKUP-19. A qualified aggregate groups by output keys and an unqualified one by output columns
LOOKUP-22. A qualified aggregate's derivations compute typed per-record values before reduction

The pair relation¶

LOOKUP-27. A named intermediate that writes inner_join is a pair relation, one record for each matching pair of its two sides

Windows¶

Declare complete named windows and compute partitions, ranks, neighbors, carried values, and baseline flags.

Named windows¶

WINDOW-1. A window reference names one complete definition declared in root windows
WINDOW-2. A named reference expands to an independent copy of its definition before analysis

Partitions¶

WINDOW-3. A window partitions the rows of its evaluation context by group_by and preserves row count
WINDOW-11. A column-level window reads a driver field only when every row template has that driver
WINDOW-4. A window filter excludes rows before partitioning, and an excluded row receives missing
WINDOW-5. Positional window operations require window.order_by, and baseline_flag defaults it to its date

Window operations¶

WINDOW-6. row_number numbers positions from 1, and rank gives rows tied on every term one number
WINDOW-7. row_value reads the source from the row a nonzero offset away along the declared order
WINDOW-12. Strictly backward row_value reads can carry calculated history in one independently ordered column group
WINDOW-8. locf carries the closest strictly earlier non-missing source forward unless include_current lets a present current source answer
WINDOW-9. baseline_flag flags the last eligible row of each partition in date order, and a tie for last fails
WINDOW-10. Window operations convert nothing: numbering returns int and selections keep their type

Text¶

Pad, take pieces by position, count scalars, write numbers with fixed decimals, change case, interpolate templates, and extract and replace with portable regular expressions.

String verbs¶

TEXT-1. Text verbs take string sources and fail rather than convert another type
TEXT-2. str_pad left-pads a value's canonical text with ASCII spaces to a minimum width
TEXT-13. str_substr takes the scalars of a text from a position counted from 1, and gives missing past its end
TEXT-16. str_substr with piece breaks a text between words into pieces of at most length scalars and takes one
TEXT-14. str_length counts the scalar values of a text as an int
TEXT-15. str_decimal writes a number with exactly decimals digits by the one rounding, left-padded with spaces to a minimum width

String templates¶

TEXT-3. A string template is literal text, braced placeholders and doubled braces, nothing else
TEXT-4. str_template replaces each placeholder with its string value, or answers a missing one

Case¶

TEXT-5. str_case changes the case of ASCII letters only, one scalar for one

Regular expressions¶

TEXT-6. Every pattern is a portable ECMA-262 subset without lookaround or backreferences
TEXT-7. A pattern carries the Unicode flag only: case-sensitive, anchored to the whole subject, stateless
TEXT-8. Character classes are fixed ASCII and ECMA-262 sets, and a non-ASCII scalar is an escape
TEXT-9. A schema pattern is a full match, while matches and str_extract search the subject
TEXT-10. Capture groups are numbered by opening parenthesis, and str_extract returns one group
TEXT-11. str_replace replaces every non-overlapping match, left to right, with literal text
TEXT-12. A pattern too deep or too large to compile is refused at its path; matching is never refused

Dates and times¶

Compute differences, study days and times of day, read declared date forms, assemble and complete collected dates and times, report collected precision and imputation flags, and apply civil arithmetic.

Operations¶

TIME-1. Each temporal operation has fixed input types, and a type-dispatched one declares its type
TIME-2. to_date returns the calendar date of a datetime or of text carrying a complete date, and partial text gives missing
TIME-3. to_epoch_day converts a date to its signed day count from 1970-01-01
TIME-20. time_of_day returns a datetime's time of day as whole seconds since midnight
TIME-23. to_time returns a datetime's time of day as a time

Differences¶

TIME-4. diff returns end minus start in whole calendar units for dates, or in an elapsed unit for datetimes
TIME-5. closed sets which endpoints a day count between two dates includes, and only that count takes it
TIME-6. study_day counts from the reference date as day 1, with no day zero

Collected dates and times¶

TIME-7. Collected date text is ISO 8601 read up to its first unknown field, and precision reports how much of the declared type a source carries
TIME-22. date_text reads text in one of its declared date forms into the collected date text it denotes
TIME-19. dtc writes collected date parts as collected date text that stops before the first missing part
TIME-8. impute completes collected text into a value of its declared type from declared month, day and time rules
TIME-9. Imputation bounds move only supplied fields, a bound outside the collected interval is ignored, and crossed bounds give not_after
TIME-10. impute's month and day rules must fit each other and the calendar
TIME-13. A missing source gives missing unless impute declares a fallback, and invalid text fails unless invalid answers it

Civil arithmetic¶

TIME-14. shift moves a date by whole days, or a datetime by a signed elapsed amount rounded to a whole second
TIME-17. duration_value reads ISO 8601 fixed-length duration text as a number in a declared unit
TIME-21. duration_text writes a number in a declared unit as canonical ISO 8601 fixed-length duration text
TIME-24. calendar_duration writes the whole months from one date to another, then the remaining days, as ISO 8601 duration text

Imputation flags¶

TIME-18. imputation_flag returns the ADaM date or time imputation flag from an imputed value's collected precision

Verification¶

Apply assertions, severity, and grouped counts to completed values, and record what each run read, counted and checked in one run report.

Running verifications¶

CHECK-1. Column checks run after their column converts; dataset checks run after all column work
CHECK-2. A failed error verification fails the run; a violated warning fails nothing and changes no row
CHECK-3. A verification may declare a nonempty id, unique within its verification list
CHECK-4. A verification declares known fields, ordered bounds and existing columns of a compatible type

Column verifications¶

CHECK-5. Only not_missing rejects a missing value, on the rows its when selects and beyond any share its max_fraction allows, and declaring it on a key column is redundant
CHECK-6. allowed_values requires every present value to equal one listed value
CHECK-7. range requires every present numeric value to lie within its inclusive bounds
CHECK-8. max_length bounds a string column's values by their count of Unicode scalar values
CHECK-9. matches requires every present string to match its regular expression

Dataset verifications¶

CHECK-10. unique requires the combined values of a column set other than the keys to be unique, missing values included
CHECK-11. all_or_none requires the listed values of each row to be all missing or all present
CHECK-12. assert requires its predicate to be TRUE on every output row its when does not exempt
CHECK-13. row_count bounds how many output rows its filter admits, as a count and as a share
CHECK-14. Row count bounds apply to every group of the artifact's rows that its when does not exempt

Assertions a verification cannot state¶

CHECK-15. An ordered-series assertion is a derivation followed by a row-wise assertion
CHECK-16. A cross-dataset link is asserted by the derivation that produces it, not by a verification

The run report¶

CHECK-22. Every run builds one report, which a warning requires a file for and a failed run still writes
CHECK-23. The report has one row per run, input dataset, row template, handler path, output column and evaluated check, in that order
CHECK-24. The report has fourteen fixed columns, and a kind leaves missing each column it has no value for
CHECK-25. KEYS and DETAILS are compact ASCII JSON with names in text order
CHECK-26. The report is verified before it is written or returned, and a malformed report is an execution defect

Output¶

Select artifact columns, keys, profiles, and row order, and publish complete files atomically.

Columns and keys¶

OUTPUT-1. The artifact's columns are exactly output.columns, in that order
OUTPUT-2. keys lists one or more output columns, each once
OUTPUT-3. Every artifact row's combined key values are non-missing and unique

Profiles¶

OUTPUT-4. output.path is required and names the primary file
OUTPUT-5. Each output path's extension selects the csv or parquet profile from a closed mapping
OUTPUT-6. A profile carries every value the language admits without changing it

Publication¶

OUTPUT-7. A run publishes once, after the whole artifact is complete
OUTPUT-8. Each file replaces its target atomically, and a failed run leaves the target unchanged

The report file¶

OUTPUT-9. The primary file and the report file have distinct paths, each selecting its own profile
OUTPUT-10. The report file publishes before the primary file, and a failed run publishes only its report

Row order¶

OUTPUT-11. output.order_by orders the artifact's rows once, after every verification
OUTPUT-12. Each output.order_by term names a different declared column

Resource limits¶

Bound each file, table, declaration, pattern and built text by one fixed policy, refuse past it with one identity, and bound each partition of a partitioned run.

Refusals¶

LIMIT-1. A size past a fixed limit refuses its operation as engine_rejected, naming the boundary, its limit and the observed size
LIMIT-2. One fixed policy bounds each file, table, declaration, pattern and built text

Partitioned runs¶

LIMIT-3. A build run over key-bounded partitions gives the whole-table run's result, and its row and cell limits bound each partition

File resolution¶

Resolve declared local paths from their base, reach one regular file, and read one immutable snapshot per physical file.

Resolution base¶

FILE-1. A document's relative paths resolve from the file that declares them, never from a discovered root
FILE-2. relative_to selects the declaring file's or the entry file's directory as a relative path's base
FILE-3. A schema-linked artifact is an ordinary input at its own path, and its producer is never run

Written form¶

FILE-4. A path is relative or rooted, decided from its text before the filesystem is consulted
FILE-5. A path has no URI scheme, backslash, empty segment or trailing separator
FILE-6. . and .. segments resolve textually from the base and may leave it

Reaching a file¶

FILE-7. Resolution walks each component from a directory anchor and must reach one regular file
FILE-8. No component below the anchor is a symbolic link

One snapshot per file¶

FILE-9. A run reads one immutable byte snapshot of each physical file, whatever spelling reaches it
FILE-10. Content that changes between validation and ingestion fails the run

Path failures¶

FILE-11. A path failure names the written path and declaring field and reports exactly one condition

CSV profile¶

Read admitted CSV spellings and write canonical CSV bytes and display precision.

Writing¶

CSV-1. A csv artifact is UTF-8 without a byte-order mark, and every record ends in a line feed
CSV-2. The header names output.columns in order, and each row follows as one record
CSV-3. A field is quoted exactly when it is empty or contains a quote, comma, U+000D or U+000A
CSV-4. Missing is written as an empty bare field and the empty string as two quote characters
CSV-5. Each value is written as the canonical text of its column type

Display precision¶

CSV-6. output.decimals is an integer from 0 to 18 for every float column of a csv artifact
CSV-7. With output.decimals of n, every float is written in fixed point with exactly n decimal digits
CSV-8. Display rounding happens once, when a field is written, and no other stage sees it
CSV-9. Display rounding applies the one rounding to the exact binary64 value

Reading¶

CSV-10. A csv source is UTF-8, with a byte-order mark only where its input declares one
CSV-11. A record ends in U+000A, optionally preceded by U+000D, the last may end the file, and blank lines that end the file are no record
CSV-12. Fields are separated by commas and are either bare or quoted, with nothing trimmed
CSV-13. The header names each field once, and every later record has exactly that many fields
CSV-14. A field reaches ingestion as its text, or as missing when it has no characters
CSV-15. No reader option changes what the csv profile fixes, and a reader repairs nothing
CSV-16. A csv reading failure names the dataset, the written path, and the record and field

Parquet profile¶

Read and write the closed Parquet field and value mapping.

Writing¶

PARQUET-1. Each column type maps to exactly one Parquet physical and logical type
PARQUET-2. A date counts days from 1970-01-01, and a datetime and a time whole-second microseconds from 1970-01-01T00:00:00 and from midnight, with no zone
PARQUET-3. A float is stored as the binary64 value its derivation produced, never rounded
PARQUET-4. Parquet artifacts of one completed dataset read back identically, without identical bytes

Reading¶

PARQUET-5. A parquet source is one Parquet file, read in its stored field and record order
PARQUET-6. Every Parquet field has a non-empty unique name and a scalar type
PARQUET-7. The embedded Parquet type supplies the field type through a closed mapping that inverts the write mapping and admits signed integers, single-precision floats and other timestamp and time units
PARQUET-8. A Parquet null is the missing value, and present values arrive unchanged
PARQUET-9. A stored date, timestamp or time names an in-range civil value on a whole second
PARQUET-10. File metadata and layout never change the fields, records, types or values delivered

SAS source profiles¶

Read SAS datasets and Transport v5/v8 through declared encoding and explicit numeric narrowing.

Containers and text¶

SAS-1. .sas7bdat and .xpt select the read-only SAS dataset and Transport profiles
SAS-2. A SAS input uses its stored encoding, and a Transport input declares its encoding

Field types¶

SAS-3. Character storage supplies str and numeric storage float, and types may only narrow a numeric field
SAS-4. Numeric narrowing accepts only an exact integral value, and SAS dates count from 1960

Values and order¶

SAS-5. SAS missing values become missing, and character values lose only trailing padding
SAS-6. A SAS reader decodes only the captured snapshot, in stored field and record order
SAS-7. A SAS or Transport file that cannot be decoded fails ingestion with one closed cause
SAS-8. A SAS dataset yields only its live records, in stored order

ODM source profile¶

Read an ODM 1.3 or 2.0 snapshot into the item table, refusing transactional content and documents without clinical data, and skip audit records.

Reading¶

ODM-1. An .xml input is read as an ODM snapshot, refusing transactional content and skipping audit records

Dataset-JSON source profiles¶

Read a Dataset-JSON 1.1 file in either representation, typing each field by its column's dataType and narrowing only by declaration.

Reading¶

DSJSON-1. .json and .ndjson select the read-only Dataset-JSON 1.1 profiles, and the file alone is the source

Field types¶

DSJSON-2. Each column's dataType supplies its field's type through a closed table, and types may only narrow it

Values and order¶

DSJSON-3. Each value is spelled as its column's dataType writes it and parses into its field's type, or fails ingestion
DSJSON-4. Fields are the columns in order and records the rows in stored order, and records counts them
DSJSON-5. A Dataset-JSON file that cannot be read fails ingestion with one closed cause

Schema language¶

Define schema notation, registries, constraints, canonical shorthand expansion, and the canonical spelling of a written value.

Bundles and documents¶

SCHEMA-1. A schema bundle is one entry point and every document it includes, loaded whole
SCHEMA-2. A yamaa file declares its type and the oldest language version it needs, and an older engine refuses it
SCHEMA-3. Documents reject duplicate keys, aliases, merge keys, explicit tags and unknown constructs
SCHEMA-4. Scalars resolve under the YAML 1.2 core schema, where only true and false spellings are Boolean

Named types and registries¶

SCHEMA-5. A named type is a class, a value descriptor or a registry-backed type
SCHEMA-6. Class fields are closed, and fields_from copies another class's fields unchanged
SCHEMA-7. A registry maps keywords to payload shapes, and a registry value names exactly one keyword
SCHEMA-8. Two things in one naming list never share a name

Type expressions¶

SCHEMA-9. A type expression is a type name, a list or dict of type expressions, or a union
SCHEMA-10. One union shape is shorthand, expanded to its canonical class while validating
SCHEMA-15. A written long form that holds only what its shorthand fills means the shorthand, and the lint reports it

Descriptors¶

SCHEMA-11. A descriptor uses only the closed descriptor keywords, each where it applies
SCHEMA-12. A value satisfies its type and every constraint its descriptor declares
SCHEMA-13. A class value carries every required field and no undeclared field
SCHEMA-14. A descriptor written in flow form and in block form is the same descriptor

Submission metadata¶

Govern dataset and column metadata, origin, methods, and document references.

Declaring submission metadata¶

SUB-1. Submission metadata is optional and is declared only for columns the artifact carries
SUB-2. The free-form metadata map is never read and never carries a governed field name

Models¶

SUB-3. A specification declares its model, and the implementation guide a document binds it to must belong to that model

Dataset metadata¶

SUB-4. Dataset metadata states its label, observation class, record structure and whether it repeats
SUB-5. A dataset's domain defaults to the specification's domain and is declared only for SDTM and SEND

Column metadata¶

SUB-6. A column's submission data type defaults from its column type and model, within the closed set that type admits
SUB-7. length and significant_digits are declared exactly where the submission data type needs them
SUB-8. A str column's length is a bound its completed values must meet, stated once
SUB-26. A str column's date or time data type is an ISO 8601 form its completed values must have
SUB-9. display_format and role describe a column for a reader and never change a value
SUB-10. core takes its model's designations, and mandatory comes from the keys, the declaration, or the class and core
SUB-11. A mandatory column is checked for missing values, and a not_missing beside it is redundant

Value-level metadata¶

SUB-12. A column's value_level entries give the rows their where clause selects their own metadata
SUB-13. Each row matches at most one entry of a column, an entry that matches no row warns, and an entry's origin is refuted on the rows it selects

Origin¶

SUB-14. Every column with submission metadata declares an origin from closed type and source vocabularies
SUB-15. Each model admits its own origin type and source pairs
SUB-16. A Predecessor, Other or Not Available origin carries a description
SUB-17. A value collected from a person references the annotated case report form, with a page under Submission
SUB-18. A Derived origin declares its method

What the derivation refutes¶

SUB-19. A declared origin the column's derivation contradicts is rejected, and an origin is never inferred
SUB-20. A Predecessor column is a bare copy of the column its description names

Methods, comments and documents¶

SUB-21. A method states its algorithm in prose and may add a name, a kind, code and documents
SUB-22. Comments and document references are declared where they are used and name declared documents

When the profile is checked¶

SUB-23. Rules needing no project file are checked on the specification alone, and the rest when a document composes it

Transport measurements and dates¶

SUB-24. With a transport, a character column is as long as the longest value the section's datasets carry
SUB-28. With a transport, a column or value-level definition that holds no value in its dataset has no data
SUB-25. Under ADaM a date, datetime or time is submitted as a SAS numeric date with a fixed display format and length
SUB-27. An SDTM or SEND dataset carries its dates and times as text, never in a datetime or time column

Controlled terminology¶

Declare codelists, compose them by id across their sources, and enforce the items each binding lists.

Declaring codelists¶

CT-1. A project's codelists compose by id across its sources, and a column binds one by id
CT-2. A codelist source's standard names its published terminology, and a source without one is sponsor-defined
CT-12. A codelist's format_name begins with $ exactly when the codelist is text

Values¶

CT-3. A codelist lists its items or defers to an external dictionary, never both and never neither
CT-4. Coded values are unique within a codelist, of its data type, and keep their declared order
CT-5. decode and rank are declared for every item of a codelist or for none
CT-6. Only an extensible published codelist may carry sponsor-extended items, and an extended item declares no alias

Bindings¶

CT-13. A project binding needs a project file, and a specification checked or run without one fails validation
CT-7. A bound codelist that lists items is the column's value set, and an external dictionary enforces nothing
CT-8. A bound column's declared type admits its codelist's data type
CT-9. An allowed_values verification beside a column's codelist of items is redundant

Documents and inventories¶

CT-10. A generated document carries exactly the codelists its output columns bind
CT-11. An inventory-vocabulary column holds only names of datasets any submission section of the project declares

Define-XML¶

Compose study metadata into deterministic Define-XML 2.1 documents.

The study document¶

DEFINE-1. A Define-XML document is composed from one selected submission section of the project file
DEFINE-2. The section's define target names the document's file, ends in .xml, and is define.xml beside a transport
DEFINE-3. Two generations with the same captured inputs and creation time produce the same bytes
DEFINE-4. A Submission context adds the requirements a regulator imposes, and Other relaxes nothing else

Composition¶

DEFINE-5. Each specs entry contributes one dataset, identified by its domain and placed in entry order
DEFINE-6. The section's implementation guides belong to its model, and each dataset is bound to the one it selects
DEFINE-7. Every composed specification declares dataset and column submission metadata and column labels
DEFINE-8. Composition resolves the codelist, standard and document names a specification leaves unresolved

Identifiers¶

DEFINE-9. Every generated identifier is a fixed prefix and the declared identifiers of the object's position
DEFINE-10. Declared identifiers are unique in their space, so generated identifiers cannot collide

Document structure¶

DEFINE-11. The document has one fixed element structure in a fixed order, and omits an element without content

Element mapping¶

DEFINE-12. ODM, Study and MetaDataVersion carry fixed attributes from the study and the selected section
DEFINE-13. Each standard and supporting document produces its definition and its leaf
DEFINE-14. At most one annotated case report form is declared, and a value collected from a person requires one
DEFINE-15. Each dataset produces one ItemGroupDef, whose Purpose follows from its model
DEFINE-16. A dataset's leaf names its transport file or its artifact, relative to the document and inside its directory
DEFINE-17. A dataset declared to have no data carries a comment that says why
DEFINE-18. Each output column produces one numbered ItemRef, keyed by its position in keys
DEFINE-19. Each column produces one base ItemDef carrying one def:Origin per declared origin
DEFINE-20. Each codelist produces one CodeList, whose item elements follow from whether its items declare decodes
DEFINE-21. Methods and comments produce MethodDef and def:CommentDef elements, and all translated text carries one language
DEFINE-22. Transport names are written only when they match the SAS transport name patterns

Serialization¶

DEFINE-23. The document's bytes are fixed: UTF-8, one element per line, two-space indentation and fixed escapes
DEFINE-24. Generated text never carries a carriage return or tab, nor a line feed in an attribute value
DEFINE-25. A complete document, valid against the selected Define-XML schema, is published atomically or not at all

Scope¶

DEFINE-26. Constructs outside the profile cannot be declared and are never generated
DEFINE-27. A column's value-level entries generate one value list, and a where clause and a definition per entry
DEFINE-28. Define-XML requires the project's study and the section's standards and define, which a run does not
DEFINE-29. A section selects its Define-XML release explicitly, retaining 2.1.11 when omitted
DEFINE-30. Historical metadata is a literal record and makes no assertion about current derivations or data

Dataset-JSON¶

Write a dataset and its submission metadata as one deterministic Dataset-JSON 1.1 file.

Requesting a transport file¶

JSON-1. A section's transport requests one transport file per composed dataset, named for the dataset beside the document
JSON-2. Every transport file has exactly one writer
JSON-3. A dataset declared to have no data has no transport file, and a dataset without rows declares it
JSON-4. The file is Dataset-JSON 1.1 in the representation transport selects and valid against that standard's schema

Content¶

JSON-5. The rows are the published artifact's rows, read the way a consumer reads a producer's artifact, and no display precision reaches them
JSON-6. Every metadata member comes from the declarations the Define-XML document is generated from
JSON-7. The file carries the standard's members in a fixed order, omitting absent ones: one object, or a first line and one row per line
JSON-8. metaDataRef names the Define-XML document beside the file
JSON-9. Each output column produces one column object carrying that column's Define-XML metadata
JSON-10. dataType maps the resolved submission data type through a closed table, and an ADaM date, datetime or time targets an integer
JSON-11. Each row is an array of its values written by column type, with missing as null

Serialization and publication¶

JSON-12. When Dataset-JSON is requested, each dataset's leaf names its Dataset-JSON file
JSON-13. The file's bytes are fixed: UTF-8, one layout per representation, one spelling per string and number
JSON-14. Transport files are published before the document, and only after every file renders and validates

Reading¶

JSON-15. A file this profile writes reads back through the Dataset-JSON source profiles as its published table

SAS Transport¶

Write a dataset as one deterministic SAS Transport version 5 file in the public record layout.

Requesting a transport file¶

XPT-1. A section's transport: xpt writes each composed dataset as one SAS Transport version 5 file

Records¶

XPT-2. The file is a fixed sequence of 80-byte records in the transport layout
XPT-3. The header records carry fixed text, the dataset's name and label, and the creation time
XPT-4. Each output column produces one NAMESTR, in output.columns order

Names, types and values¶

XPT-5. Names are names DEFINE-22 admits, and labels are at most 40 printable ASCII characters
XPT-6. A str column is a character variable, a number a numeric one, and a date travels as its model submits it
XPT-7. A character value is printable ASCII of at most 200 bytes without a trailing blank, and missing is blanks
XPT-8. A numeric value is an IBM System/360 double converted exactly from binary64, and missing is a period
XPT-9. A numeric variable carries a display format only when its declared one parses as a SAS format
XPT-10. A file above 5,000,000,000 bytes, or one ending in an all-blank observation, is refused before writing

Project functions¶

Define each project function once in a function file, resolve immutable runtimes, and validate function inputs, results, and activation conformance.

Function files¶

FUNC-1. Project functions come only from the function files the caller's project lists, through the running host's implementation
FUNC-2. A function file is admitted on its own against the function schema before any activation or study read
FUNC-3. One host language applies to every function, and a declared lock pins the called packages

Definitions and calls¶

FUNC-4. Each logical function has one closed definition, and a call names only the function and its arguments
FUNC-5. A signature is closed and named, and every argument and default matches its declared type exactly
FUNC-6. A call argument is one leaf of the expression grammar, never a composed expression
FUNC-7. An omitted optional argument takes its default, and a missing argument short-circuits unless its parameter accepts missing

Binding and invocation¶

FUNC-8. A binding is a statically named installed callable, invoked once for each logical row
FUNC-9. A binding returns one scalar of its declared type, and missing only when its definition permits it
FUNC-10. An invocation failure identifies the logical function, its callable and its authored source

Activation conformance¶

FUNC-11. Each definition's inline tests cover every obligation its contract implies
FUNC-12. Every build runs every case of every called function before any study data is read
FUNC-13. A numeric result is compared at comparison_decimals with the one rounding, and is never changed

Project file¶

List a study's function files and host locks, declare its terminology and submission sections once, compose a project file on one parent, and check, run and package the specifications its sections list.

The project file¶

PROJECT-1. A project file is passed explicitly and names one study's functions, host locks, terminology and submission sections
PROJECT-16. A project file names at most one parent, and composes on it section by section
PROJECT-2. environments holds each host language's lock, and each host reads only its own
PROJECT-3. A section needs only specs, and the study metadata a run does not read is required only by Define-XML
PROJECT-4. A project file's relative paths resolve from the project file that writes them

Specification names¶

PROJECT-5. A listed specification's name is its domain, unique across the project
PROJECT-6. Selecting a project specification by name accepts only a name the project defines

Selecting a section¶

PROJECT-14. Selecting a project section accepts only a section the project declares

Checking a project¶

PROJECT-7. Checking a project checks every listed specification in section and listing order, with the project findings

Project runs¶

PROJECT-8. A project run builds each specification of its run scope once, a linked producer before its consumers
PROJECT-9. A producer outside the run scope is not built, and its consumer reads the saved file
PROJECT-10. An input that reads a run-scope artifact without a schema link fails
PROJECT-11. A failed run-scope specification stops only the specifications that depend on it

Saving a project run¶

PROJECT-12. Saving a project run validates every target first, then publishes specification by specification in run order

Project submissions¶

PROJECT-13. A project submission composes each selected section's Define-XML document in section order
PROJECT-15. A project submission composes a specification that declares a run report only with the artifact and inputs that report records