irw_validate — one format validator, with an exit code

irw-validate out/*.csv                       # exit 1 if anything blocks
irw-validate out/                            # every table directly in out/
irw-validate -r .                            # ...and in every subdirectory
irw-validate out/x.csv --profile core        # the validate_irw.R subset
irw-validate out/x.csv --strict              # warnings block too
irw-validate out/x.csv --json                # for CI
irw-validate out/x.csv --verbose             # every affected item, not the first few

Exit codes: 0 ok · 1 something blocks · 2 bad input. Same contract as red_up.

A directory argument picks up .csv, .tsv, .rdata, .rda and .rds files and skips everything else silently, including hidden files and __MACOSX/. .txt is read when you name the file but not picked up from a directory, because there it is usually a README. A directory with no tables in it exits 2.

Installing

pip install irw-validate                    # from PyPI
pip install -e /path/to/irw/src/irw_validate   # from a checkout

pandas and the standard library, nothing else — no Redivis, no credentials, no network. That is deliberate: this is the thing an outside contributor runs against their own file before depositing it, and they should not have to clone the pipeline or hold a token to do it.

Two optional extras exist for cases that are not that:

Until 2026-09-09 this package shipped inside irw-red-up, the uploader distribution, so the only way to get the validator was to install the writer. That was an accident of packaging: ../pyproject.toml keeps red_up out of Python-pkg because a write-scoped uploader would change what that package is, and none of that reasoning applies to a checker that opens a CSV.

irw-validate is a console script. In a checkout without the install, python3 -m irw_validate.cli works from src/.

Why this exists

The checks were forked, and neither half could gate anything:

So "is this table valid?" had two answers and no way to act on either. This is roadmap item 1 (ben-domingue/irw#1703), sub-items 1.3 and 1.4 — the other half of the work red_up started.

It also makes datastandard.md executable. The standard is 348 lines of rules stated in prose — a sample floor of 100 unique ids, table names ≤40 characters, [id, item, resp] first — and until now not one was enforced by anything. ARCHITECTURE.md's Rule 2 asks for exactly this: where a rule can be made executable, make it executable instead of writing it down.

Two verdicts: the standard, and the gate

Every report under the upload and legacy profiles opens with two lines:

IRW Data Standard 1.0: does not conform (C4, C5)
IRW upload gate: blocked by 3 error(s)

They answer different questions. Conformance asks whether the file is a valid IRW table, judged against the numbered clauses of the IRW Data Standard — only the checks in model.CLAUSES bear on it, and each finding names its clause ([C4]). The gate asks whether IRW would accept the table, which adds intake policy that is not part of the standard: the 100-id sample floor, table-name rules, and the heuristics. A researcher's table can conform and still fall under the floor.

Three rules keep the verdict honest:

• A waiver (--override) lets a table through the gate and leaves it nonconforming. nonconforming counts overridden errors.
• There is no verdict under core or triage. Those profiles keep inherited readings of C4 and C5 for their callers, so a rater design would be called nonconforming there and conforming at the gate.
• There is no verdict for item text (__items), which Standard 1.0 does not cover.

STANDARD_VERSION moves only when the standard's changelog does. Adding a check is not a new standard; changing what a clause requires is.

The browser validator at https://itemresponsewarehouse.org/validate.html runs this package unchanged, in Pyodide, pinned to a released version.

Profiles, and why severity is not a property of a check

Severity depends on the (check, profile) pair. This is the central design decision, and it exists because the checks were written for triage — is a machine's guess at a conversion worth a human's time — and are now also asked to gate publication, which has a different cost of being wrong.

Response-scale checks illustrate the distinction. An unused top category can make items on one scale have different observed ranges. Upload already capped that heuristic at warn, but callers of run_qc() still received a raw fail. The shared checks now emit warn for observed range differences in either direction; a response-scale fail requires explicit documentation, as below.

GATE_ERRORS contains resp_variation*, resp_outside_permitted and resp_scale_constructs. The latter two are documentation-backed failures. The width-only resp_scale_mixed warning has a separate name and remains nonblocking unless strict mode is selected. A waiver for that old width-check name cannot waive a documented resp_scale_constructs failure.

Response-scale evidence

run_qc() accepts optional permitted_values and item_constructs arguments. validate_frame() and validate_file() accept the same information through context. These are Python API inputs; there are no corresponding CLI flags or automatic codebook extraction.

permitted_values is either one collection of allowed numeric responses shared by all items, or a mapping from item identifiers to their allowed collections. Use the source's documented categories, never categories inferred from the observed responses. item_constructs maps each item identifier to an explicit, documented construct name. Prefixes and disjoint respondent sets are not construct evidence.

Item keys must match the original identifiers exactly. Permitted categories must be finite numeric values (numeric strings are accepted). Integer codes retain exact precision; fractional categories are compared at the response's floating-point storage precision, without an approximate matching tolerance.

from irw_validate import validate_frame, validate_file

# Illustrative codebook for one construct assessed in two item formats.
context = {
    "permitted_values": {
        "mc_1": {0, 1}, "mc_2": {0, 1}, "cr_1": {0, 1, 2, 3},
    },
    "item_constructs": {
        "mc_1": "reasoning", "mc_2": "reasoning", "cr_1": "reasoning",
    },
}
report = validate_frame(df, label="reasoning", context=context)
file_report = validate_file("out/reasoning.csv", context=context)
# With run_qc imported by a conversion script: run_qc(df, **context)

The two documented failures become error under triage, upload and legacy; core excludes these response-scale checks. Complete permitted values suppress only width warnings, not a documented construct finding. A warning is not permission to drop an item or split a scale: multi_scale* remains a prefix-based warning requiring source verification before a split. Strict mode can make warnings block; default upload does not. An HPQ-shaped isolated count column is named by item_scale_outlier even when its range contains the modal range. A rare legitimate item format also only warns. SDV-shaped ranges of 1–5, 1–7, 1–8 and 1–9 remain nested-support warnings without construct evidence; this is an explicit limit of inference from response widths.

Literal missing-value tokens (#2029)

For upload and legacy, every non-missing resp must parse as a number. There is no 1% allowance for invalid values. File validation preserves literal text such as NA, N/A, NULL and whitespace so it can report an error with the count and up to five examples. A genuinely empty CSV field (including "") remains missing: partial missingness is a resp_na warning; an entirely missing response column still blocks. No source file or input frame is edited.

Because of that, id_na / item_na / resp_na word themselves by profile. On upload and legacy the loader keeps NA-style text as a response, so a missing value counted here is an empty cell. On triage and for the data/ callers the reader parses those tokens as missing, so the count may mix the two and the message says so instead of implying a count of empty cells (#2314).

CSV/TSV/TXT validation makes a second pass reading only resp with pandas' default NA-token conversion disabled. Other columns keep their existing parsing behavior, and clean numeric response columns still infer numeric types. Item-text tables use their separate schema and retain their existing reader. The core and triage profiles retain their earlier parsing and 99% numeric threshold, now measured over non-missing responses (#2314 item 2). Raw run_qc() callers use that same denominator. Missing responses are reported by resp_na; with no present responses, resp_numeric records that there is nothing to check, while resp_na still fails. Literal text supplied in an in-memory frame remains present and is tested for numeric parsing. The reader's NA-token handling is unchanged. Callers using run_qc should use validate_file(..., profile="upload") on the written file when they need the strict publication check.

For an in-memory frame, genuine nulls remain missing and literal text is checked, but a token already erased by an upstream reader cannot be recovered. The existing 512 MiB file-size cap still applies; files over it receive only name checks. This change does not repair historical tables, resolve the meaning of their missingness, or alter published response counts. Review source coding before changing rows; the finding deliberately does not prescribe deletion.

The rights register (#2154)

On the upload profile, a table is matched against itemtext/instrument_rights_register.csv before it ships: match_item_text against an __items table's item_text, and match_item_code against a response table's item codes. Rows ruled block, hold or escalate produce a rights_register warning that quotes the register row and asks for a hold; ship_with_note rows produce one asking that the note exists.

It never errors and a clean result says nothing. A hit is a lead (the sweep's three searches each produced a false positive), and a miss is not a clearance (complete PSS reproductions scored 1 of 4 on substrings). Installed without a checkout there is no register, and checks_run records rights_register:unavailable. IRW_RIGHTS_REGISTER points at a copy.

The override

irw-validate x.csv --override-check resp_scale_mixed \
    --override "two response formats, one construct; author confirmed 2026-09-02"

The reason is the flag's argument, so overriding without saying why is structurally impossible. It is not called --force or --no-verify: those names invite reflex use. A reason under 20 characters is rejected. Overridden findings are reprinted under OVERRIDDEN rather than suppressed, and appended to processing_notes/validator_overrides.csv.

Without --override-check the reason waives every error; with it, only the named errors, so unrelated failures keep blocking. Warnings are never waived and continue to block under --strict.

In particular, composite_items* is now a warning even when every item label matches (#2369): label names alone cannot establish computed scores. An otherwise valid all-match table passes default triage validation, but exits 1 with --profile triage --strict, including when --override-check 'composite_items*' --override <reason> is supplied. Before the demotion, that scoped override could waive the triage error and exit 0 under strict mode. The errors-only override contract is unchanged: the warning is retained and no waiver is recorded for it. Upload/legacy already treated this check as a warning; core omits it.

Existing conversion callers

data/*.py scripts do from irw_triage_updated import run_qc and read .name / .status / .detail. That interface and the existing positional arguments remain compatible. The implementation lives in _checks.py and is re-exported by triage; profiles are layered on top by core.py. The optional documentation inputs above refine response-scale results without requiring existing callers to provide them.

The denominator correction intentionally changes raw/core/triage findings and failure counts: numeric responses with missing values no longer fail resp_numeric solely for missingness. In triage_dataset(), such a candidate can move from human_assistance to good if no other routing condition blocks it; the resp_na warning remains. Scripts gating on raw fail statuses see the same correction. All-missing responses still fail, and upload/legacy validation and CSV parsing are unchanged.

tests/test_validate.py pins the exact (name, status) emission order for eight fixtures, with explicit corrections for text-only response ranges (#1697) and the all-missing numeric check (#2314). Response-scale regression tests in irw_validate/tests/ additionally cover the shared checks and public API profile behavior, and run in the existing unittest CI suite.

Staying merged

A shared runtime between R and Python is not possible here — validate_irw.R's whole value is that it works for a stranger with an R session and a URL, with nothing else installed. So instead the R file carries # @check <name> markers, and a test parses them and asserts set-equality with model.CORE_CHECKS. Two languages, one list, enforced. Edit one copy and the suite fails.