rivaldefs

Which of two rival labeling rules sits inside the other, and is the gap between the rules larger than the gap between their implementations?

Two rules applied to the same units split them into four groups: positive under both, under the first only, under the second only, under neither. Agreement statistics such as κ reduce the four to one number and lose the direction of the disagreement. Two pairs of rules can share κ while one pair is nested (one rule is a stricter cut of the other) and the other crosses (each rule picks out units the other misses).

rivaldefs reports the direction as a quantity, with its uncertainty, and compares the disagreement between two rules with the disagreement among admissible implementations of each rule. It implements the indices of a frozen preregistration (criteria-direction, PREREG_v2, OSF https://osf.io/bpw6e/).

Terms

The package was written before the manuscript settled its vocabulary. The code that ran the analyses is released unchanged, so the identifiers keep the earlier words.

Version and scope

The analyses of the criteria-direction paper (simulation, MIMIC-IV, NHANES) ran on commit 76cd25a684614de93e6ae96b5b58527782825d64. The SHA-256 of src/rivaldefs, taken as the analysis code takes it (sha256 over name\0<sha256 of the file>\n for each *.py file in name order), is 76c690d8363f95f569f6c9349cfa2296e1d70cd1d9801aff0bfac9921809df7b, the value recorded in the result files. Version 0.1.0 has the same src/.

rivaldefs is the statistical core. The simulation generators and the MIMIC-IV and NHANES classification pipelines are in the criteria-direction repository.

Known limitations in 0.1.0

None of the three affects the recorded analyses; the two checks that establish this are in criteria-direction, papers/paper1_bmc_mrm_rival_definitions/experiments/PREREG_v2_rival_definitions/results/rivaldefs_release_preflight_2026-10-07.json.

• qualifying_cells reads orientation_error=None as no adverse evidence, and OCRecord does not require that None go with an undefined population orientation. A record with a defined orientation and no orientation error would not block its cell. Pass None only for pairs of equal population prevalence.
• ladder orders weighted prevalences by their floating values and does not apply TIE_RTOL. Two prevalences that tie under the pairwise rule but differ in the last bits can be ordered either way, which can change the on-ladder share.
• gram and grid_from_gram trust their input. grid_from_gram checks shape, symmetry and N, and does not check that the matrix is a feasible Gram matrix of binary columns; gram does not validate weights. grid validates before calling them.

The problem, concretely

N = 1,000, two pairs of rules:

κ is 13/34 for the first pair and 61/160 for the second, 0.38 both. In the first pair five units break the nesting where independence would put seventy; in the second each rule has 99 units the other misses.

Install

git clone https://github.com/elliottower/rivaldefs && cd rivaldefs && pip install -e ".[test]"

Python 3.10+, NumPy.

Use

import numpy as np
import rivaldefs as rd

t = rd.PairTable(n11=95, n10=5, n01=205, n00=695)
t.orientation   # 'a smaller'
t.h             # 0.928...  = 1 - m/E0 = 13/14, Loevinger's H (kappa / kappa_max)
t.c             # 0.95      share of the smaller set inside the larger
t.kappa         # 0.382...

# Many labelers at once: a units x labelers 0/1 matrix.
rng = np.random.default_rng()
theta = rng.normal(size=2000)
labels = np.column_stack([
    theta > 1.0,                                          # rule A, implementation 0
    theta > 0.9,                                          # rule A, implementation 1
    theta + rng.normal(scale=0.8, size=2000) > 0.6,       # rule B, implementation 0
    theta + rng.normal(scale=0.8, size=2000) > 0.7,       # rule B, implementation 1
]).astype(int)

g = rd.grid(labels)          # K x K grids of H, C, kappa, discordance, NODF, orientation
g.h[0, 2]

a = rd.Rule("A", columns=(0, 1), levels=((0,), (1,)))
b = rd.Rule("B", columns=(2, 3), levels=((0,), (1,)))
cmp = rd.compare_rules(g, a, b)      # factor-balanced implementation distribution
cmp.delta                    # D_between - max(D_within), percentage points
cmp.s                        # probability a random implementation pair keeps the reference orientation

res = rd.bootstrap(labels, pairs=[(0, 2)], rule_pairs=[rd.RulePair(a, b)], n_boot=2000)
res.pairs[0].h                         # Interval(estimate, lo, hi, fraction_undefined, reported)
res.pairs[0].orientation_probabilities
res.rule_pairs[0].delta

What it computes

• Pairs. The 2×2 table (counts or weights), orientation (the smaller set, or equal), the minority cell m and its independence value E0, H = 1 − m/E0, % inside C, κ, κmax, discordance d and the NODF pair score. Undefined values are NaN: H when a prevalence is 0 or 1, C when the smaller set is empty.
• Many labelers. K × K grids, Mokken's scale H, and the prevalence ladder (on-ladder share, its independence value, ladder H).
• Rules and implementations. A Rule is a family of labelers with factor levels. Its implementation distribution is factor-balanced by default (uniform on a full factorial), reference-weighted as an alternative, and the within-rule draws can be conditioned on being distinct. compare_rules returns D_between, D_within, Δ, R, S and the weighted quantiles of H over implementation pairs, computed exactly over every variant pair.
• Uncertainty. bootstrap resamples the counts of observed label patterns (the participant bootstrap, from pattern counts) and recomputes every labeler, orientation, H and rule comparison per replicate. replicate_weights does the same from survey replicate weights. A replicate where H or R is undefined is excluded; an interval is reported only when at least 90% of replicates are defined, and the fraction undefined is always returned. The result objects apply the registered gates: PairResult.h_delta_interval (the delta-method interval) is None unless the modal orientation has bootstrap probability ≥ 0.95 and 0 < m < E0, and RulePairResult.r is withheld (NaN, reported=False) unless the lower percentile of R's denominator exceeds 0.5 percentage points. The ungated values sit in h_delta_interval_ungated and r_ungated, for diagnostics.
• Ties and undefined terms. Two positive counts are equal when they differ by at most TIE_RTOL × N (2^20 machine epsilons), so floating survey weights that sum to the same prevalence tie; integer counts never merge. Under the conditional-distinct-pair law a rule with one implementation has no within term, and Δ and R are then NaN whichever side it is on.
• Orientation rule. qualifying_cells reads a simulation's operating characteristics by E0 band × prevalence-ratio band; orientation_status marks a real pair resolved when its cells qualify and its modal orientation has bootstrap probability ≥ 0.95. A pair with E0 = 0 (a prevalence of 0 or 1) is never resolved, and malformed records or probabilities are refused rather than read.
• Missing inputs. delta_bounds gives conservative outer bounds on Δ from each unit's possible completions.

Tests

make test

The tests check exact values on hand-worked tables, H against an independent count of Guttman errors, H = κ/κmax on random tables, the invariance of Δ, R and S to duplicated implementations, the equivalence of the pattern-count and unit bootstrap (for H, Δ, R and S), survey replicate intervals against a hand computation, the registered gates on both sides of their thresholds, and that every completion of a small table with missing inputs falls inside the bounds. Randomized tests draw a new seed each run and print it when they fail; replay one with RIVALDEFS_TEST_SEED=<seed> make test.

License

MIT