Text Operations

Review ask

As a clinical trial statistician or programmer, challenge anything in this benchmark: describe a problem in Comments, or propose a fix with the Edit button on any source file.

Task: Would a real study need this derivation, specified this way? Does it follow CDISC ADaM: variable names, controlled terms, record structure? Flag any rule that is ambiguous or that a sponsor would handle differently.
Inputs: Do they look like data collected in a real trial, and do they include the situations you would expect to meet in one?
Expected output: Would independent QC programming reproduce every value from these inputs and rules?

The other tabs beside Comments are background for assessment, not under review.

Summary

EditUnder reviewWhat this benchmark means

README.md

Goal: derive UPPERTXT, LOWERTXT, EQUALFL, LEASTTXT, GREATESTTXT, MAPCAT, TEXTSEQ, MINTXT, and MAXTXT for each subject from the collected text and a comparison text.

Input: demographics records carrying RAWTXT (collected text) and PEERTXT (comparison text).

Variables:

UPPERTXT is RAWTXT with American Standard Code for Information Interchange (ASCII) lowercase letters changed to uppercase; every other character is unchanged, and missing text gives the text MISSING.
LOWERTXT is RAWTXT with ASCII uppercase letters changed to lowercase; every other character is unchanged, and missing text gives the text MISSING.
EQUALFL is Y when RAWTXT and PEERTXT hold identical character sequences (same code points in the same order), including when both are missing; otherwise it is N.
LEASTTXT is the earlier non-missing value of RAWTXT and PEERTXT by Unicode code-point order (called scalar order below), left blank when both are missing.
GREATESTTXT is the later non-missing value of RAWTXT and PEERTXT by scalar order, left blank when both are missing.
MAPCAT groups spellings that differ only in the case of ASCII letters: any casing of abc gives ASCII and i or I gives ASCII_I. Missing text gives the text MISSING, and any other text gives OTHER, so text holding a character outside ASCII never joins a group.
TEXTSEQ numbers rows in scalar order of the collected text, breaking ties by subject, with missing text last.
MINTXT is the earliest collected text in the study by scalar order, shown on every row.
MAXTXT is the latest collected text in the study by scalar order, shown on every row.

Note: text is never normalized, so a composed and a decomposed spelling of the same word (an accented letter stored as one character, or as the base letter plus a combining accent) are different values. The flag, the earlier and later values, the sequence, and the minimum and maximum all follow scalar order.

YAML specification

1 spec file

spec.yaml

Edit145 lines

1schema_version: "1.0"2domain: ADSL3keys: [STUDYID, USUBJID]4input:5  DM: input/dm.csv67output:8  path: adsl.csv9  columns:10    - STUDYID11    - USUBJID12    - RAWTXT13    - UPPERTXT14    - LOWERTXT15    - EQUALFL16    - LEASTTXT17    - GREATESTTXT18    - MAPCAT19    - TEXTSEQ20    - MINTXT21    - MAXTXT22  order_by: [RAWTXT, USUBJID]2324columns:25  - name: STUDYID26    type: str27    label: Study Identifier28    derivation: DM.STUDYID2930  - name: USUBJID31    type: str32    label: Unique Subject Identifier33    derivation: DM.USUBJID3435  - name: RAWTXT36    type: str37    label: Raw Text38    derivation: DM.RAWTXT39    verifications:40      max_length:41        max: 64243  - name: PEERTXT44    type: str45    label: Comparison Text46    derivation: DM.PEERTXT4748  - name: UPPERTXT49    type: str50    label: Uppercase ASCII Text51    derivation:52      str_case:53        source: RAWTXT54        to: upper55        missing: MISSING5657  - name: LOWERTXT58    type: str59    label: Lowercase ASCII Text60    derivation:61      str_case:62        source: RAWTXT63        to: lower64        missing: MISSING6566  # Two missing texts count as equal. With only one side missing the67  # comparison is unknown, not true, so the flag falls through to N.68  - name: EQUALFL69    type: str70    label: Exact Text Equality Flag71    derivation:72      case:73        - when: "RAWTXT IS NULL AND PEERTXT IS NULL"74          then: {literal: "Y"}75        - when: "RAWTXT = PEERTXT"76          then: {literal: "Y"}77        - otherwise: {literal: "N"}7879  - name: LEASTTXT80    type: str81    label: Earlier Text by Scalar Order82    derivation:83      least:84        sources: [RAWTXT, PEERTXT]8586  - name: GREATESTTXT87    type: str88    label: Later Text by Scalar Order89    derivation:90      greatest:91        sources: [RAWTXT, PEERTXT]9293  # Missing text is answered by the first branch, so the mapping needs only94  # an unmapped value, which answers text that has no dictionary entry95  # after ASCII case folding.96  - name: MAPCAT97    type: str98    label: ASCII Folded Mapping Category99    derivation:100      case:101        - when: "RAWTXT IS NULL"102          then: {literal: MISSING}103        - otherwise:104            mapping:105              source: RAWTXT106              case_sensitive: false107              dict:108                ABC: ASCII109                I: ASCII_I110              unmapped: OTHER111112  # One sequence over the whole study: text order, then subject; missing113  # text sorts last.114  - name: TEXTSEQ115    type: int116    label: Text Scalar Order Sequence117    derivation:118      row_number:119        window:120          order_by: [RAWTXT, USUBJID]121122  - name: MINTXT123    type: str124    label: Minimum Study Text125    derivation:126      aggregate:127        expr: "MIN(DM.RAWTXT)"128        group_by: [DM.STUDYID]129        key: [STUDYID]130131  - name: MAXTXT132    type: str133    label: Maximum Study Text134    derivation:135      aggregate:136        expr: "MAX(DM.RAWTXT)"137        group_by: [DM.STUDYID]138        key: [STUDYID]139140verifications:141  - unique:142      columns: [STUDYID, USUBJID]143  - row_count:144      min: 8145      max: 8

Inputs

Under review1 source file

input/dm.csv

Edit8 rows

input/dm.csv
STUDYID	RAWTXT	PEERTXT
TEXT	abc	abc
TEXT	AbC	abc
TEXT	café	café
TEXT	café	café
TEXT	straße	straße
TEXT	İIıi	İIıi
TEXT	😀ABCDE	😀ABCDE
TEXT	—	—

Expected output

Under review8 expected rows

expected/adsl.csv

Edit8 rows

expected/adsl.csv
STUDYID	RAWTXT	UPPERTXT	LOWERTXT	EQUALFL	LEASTTXT	GREATESTTXT	MAPCAT	TEXTSEQ	MINTXT	MAXTXT
TEXT	AbC	ABC	abc	N	AbC	abc	ASCII	1	AbC	😀ABCDE
TEXT	abc	ABC	abc	Y	abc	abc	ASCII	2	AbC	😀ABCDE
TEXT	café	CAFÉ	café	N	café	café	OTHER	3	AbC	😀ABCDE
TEXT	café	CAFé	café	N	café	café	OTHER	4	AbC	😀ABCDE
TEXT	straße	STRAßE	straße	Y	straße	straße	OTHER	5	AbC	😀ABCDE
TEXT	İIıi	İIıI	İiıi	Y	İIıi	İIıi	OTHER	6	AbC	😀ABCDE
TEXT	😀ABCDE	😀ABCDE	😀abcde	Y	😀ABCDE	😀ABCDE	OTHER	7	AbC	😀ABCDE
TEXT	—	MISSING	MISSING	Y	—	—	MISSING	8	AbC	😀ABCDE

Shaded columns are derived— = missing value

Comments and assessment materials

Dataset Name	Variable Name	Variable Label	Type	Computational Method	Origin
ADSL	STUDYID	Study Identifier	character	Copy value from DM.STUDYID.	Predecessor
ADSL	USUBJID	Unique Subject Identifier	character	Copy value from DM.USUBJID.	Predecessor
ADSL	RAWTXT	Raw Text	character	Copy value from DM.RAWTXT.	Predecessor
ADSL	UPPERTXT	Uppercase ASCII Text	character	Convert RAWTXT to upper case; a missing source → "MISSING".	Derived
ADSL	LOWERTXT	Lowercase ASCII Text	character	Convert RAWTXT to lower case; a missing source → "MISSING".	Derived
ADSL	EQUALFL	Exact Text Equality Flag	character	If RAWTXT IS NULL AND PEERTXT IS NULL then "Y"; If RAWTXT = PEERTXT then "Y"; otherwise "N".	Derived
ADSL	LEASTTXT	Earlier Text by Scalar Order	character	Least of RAWTXT and PEERTXT.	Derived
ADSL	GREATESTTXT	Later Text by Scalar Order	character	Greatest of RAWTXT and PEERTXT.	Derived
ADSL	MAPCAT	ASCII Folded Mapping Category	character	If RAWTXT IS NULL then "MISSING"; otherwise recode RAWTXT: "ABC" → "ASCII", "I" → "ASCII_I"; missing values are errors; unlisted values → "OTHER"; matching ignores letter case.	Derived
ADSL	TEXTSEQ	Text Scalar Order Sequence	numeric	Row number within each group of , ordered by RAWTXT then USUBJID.	Derived
ADSL	MINTXT	Minimum Study Text	character	Aggregate MIN(DM.RAWTXT) over records grouped by DM.STUDYID, matching on STUDYID.	Derived
ADSL	MAXTXT	Maximum Study Text	character	Aggregate MAX(DM.RAWTXT) over records grouped by DM.STUDYID, matching on STUDYID.	Derived

Version	Date	Author	Description
1.3	2026-09-29	yamaa docs build	Show study codelist names, external dictionaries, and ISO 8601 formats in variable rows; list the document's codelist values and metadata
1.2	2026-09-29	yamaa docs build	Classify ADaM copies from a declared input as Predecessor and recodes as Derived, and SDTM recodes of locally derived columns as Derived
1.1	2026-09-29	yamaa docs build	Resolve lookups into the donor record they select, and give every derivation shape a plain-language rule instead of its raw YAML
1.0		yamaa docs build	Initial generation from spec.yaml

run.R

Edit4 lines

1library(yamaa)23adsl <- yamaa_domain("spec.yaml")$output4adsl

run.py

Edit4 lines

1import yamaa23adsl = yamaa.yamaa_domain("spec.yaml").output4adsl

input/dm.csv

expected/adsl.csv

Comments

Mapping spec

yamaa code

run.R

run.py