iyipada

Drift detection where the threshold is calculated instead of quoted.

Production drift monitoring runs on numbers that came from a blog post. The most common rule in the industry is "PSI above 0.10 means watch it, above 0.25 means act". Those numbers are stated as though they were properties of the statistic. They are not. On data where nothing has changed at all, that first rule fires 85% of the time at 100 samples per batch, and it is mathematically incapable of firing at 1,000.

This library computes the threshold from the null distribution, so a 1% false-alarm rate is a 1% false-alarm rate.

iyipada is Yoruba for change.

The three ways a drift monitor lies to you

Each of these is measured in this repository, not asserted.

One. A fixed threshold means different things at different sample sizes. PSI under the null is distributed as (1/n + 1/m) * chi2_{B-1}, so it depends on both sample sizes and the bin count. The industry rule depends on neither. False-alarm rate of the "watch" rule on identically distributed data:

samples per batch	5 bins	10 bins	20 bins
50	64.3%	98.2%	100.0%
100	29.5%	85.1%	100.0%
200	4.1%	37.0%	96.0%
500	0.0%	0.3%	17.6%
1,000 and up	0.0%	0.0%	0.0%

The calibrated 1% threshold across that same grid runs from 1.4476 down to 0.0004, a span of about 5,400. One number cannot serve all of it.

Two. Twenty features is twenty chances to be wrong. Twenty clean features tested independently at alpha = 0.05 produce at least one alarm on 73% of clean batches. With Benjamini-Hochberg, 5%. Monitoring is a multiple-testing problem before it is a drift problem.

Three. Fifty-two weekly checks is not one check. A test calibrated to 5%, run weekly, is wrong at least once during a year with probability 93%. Getting the single check right and then scheduling it re-creates the problem it solved.

Using it

from iyipada import Detector, Monitor

d = Detector(reference_values, statistic="psi", alpha=0.01)
r = d.check(this_weeks_values)

r.drifted     # bool, decided against a calibrated threshold
r.pvalue      # what you actually want to look at
r.threshold   # computed from your sample sizes, not quoted from anywhere
r.warning     # set when the batch is too small for the test to mean much

Nothing in the API accepts a raw divergence threshold, because a raw divergence threshold cannot be interpreted without knowing the sample sizes and the bin count.

For a table of features, with the multiple testing handled:

m = Monitor(reference_dataframe_as_dict, alpha=0.05, correction="fdr")
result = m.check(live_dataframe_as_dict)
result.drifted        # the features that moved, after FDR control

For a monitor that runs on a schedule, stop re-testing and accumulate:

from iyipada import sequential as sq

h, arl = sq.calibrate_run_length("cusum", target_arl=260)   # weekly checks
# h = 3.75, one false alarm every 4.8 years of weekly monitoring
c = sq.CUSUM(h=h)
c.update(this_weeks_statistic)   # True when the evidence has accumulated

Which detector should you actually use

Wasserstein, in most cases. Every statistic below was first calibrated to the same 1% false-alarm rate, so this compares power and nothing else. Detection rate at 1,000 samples a side:

	mean shift	variance	skew	discretisation	mixture	tail only
Wasserstein	100%	100%	100%	100%	99.5%	15.7%
PSI	99.5%	100%	100%	100%	86.2%	3.5%
Jensen-Shannon	99.5%	100%	100%	100%	85.8%	3.5%
chi-square	99.5%	100%	100%	100%	85.7%	3.5%
Kolmogorov-Smirnov	100%	99.8%	88.7%	100%	83.0%	2.3%
total variation	98.3%	100%	99.8%	100%	59.7%	2.8%

Wasserstein wins or ties everywhere, and it reports in the units of the variable, so a result reads as "the distribution moved by 0.3 mg/dL" rather than as an abstract divergence. PSI, the industry default, is beaten on the two hardest cases.

The blind spot

A small fraction of values jumping to the far tail is close to invisible. At 5% contamination Wasserstein catches it every time, PSI 29% of the time and Kolmogorov-Smirnov 18%. At 1.5% nothing is reliable.

The binned statistics fail because quantile binning puts the reference's entire top decile into one bin, so moving mass from +2 to +6 does not change the histogram at all. KS fails because a few per cent in the far tail cannot move the largest vertical gap between two cumulative curves. Wasserstein survives because it measures horizontal distance.

If rare extreme values are the failure you fear, monitor the tail directly with a quantile or an exceedance count. A distributional distance is the wrong instrument, however well calibrated.

What is verified, and how

The asymptotic null is not taken on faith. Simulated 95th percentile divided by theory, across ten sample sizes and three bin counts, sits within 2% of 1.0 from 200 samples upward. It is 6 to 9% optimistic around n = 100 and it breaks down at n = 50 with 20 bins, where each bin holds two or three points and the chi-square approximation has nothing to stand on.

So the library uses the closed form above 300 samples and simulates the null below it, rather than trusting one answer everywhere. With that switch, the calibrated threshold holds between 0.0% and 2.2% actual false-alarm rate across the entire grid, against 0% to 100% for the fixed rule.

The ringed cell is 112 samples at 10 bins. That is a real case from loom, an earlier project of mine, where a fixed 0.10 rule fired a spurious retrain on identically distributed data. It fires there 78.6% of the time. That bug is why this library exists.

The score, animated

The null distribution of PSI collapsing as the sample grows, while the fixed rule stays exactly where it is. Every number in the animation comes from benchmarks/false_alarms.py. (Source: assets/manim_null.py, video in assets/null.mp4.)

What is in here

• iyipada/divergence.py the statistics: PSI, Jensen-Shannon, total variation, Kolmogorov-Smirnov, Wasserstein, chi-square. None of them decides anything.
• iyipada/null.py the null distributions: analytic for PSI, chi-square and KS, permutation for everything else, plus false_alarm_rate() for measuring what a threshold you already use is actually doing.
• iyipada/detect.py Detector and Monitor, the latter with FDR or Bonferroni control.
• iyipada/sequential.py CUSUM and Page-Hinkley calibrated to an average run length, alpha spending across a known horizon, and detection_delay() so a monitor that never false-alarms can be checked for being merely deaf.
• benchmarks/ the three studies behind every number above.

Honest limits

• Univariate only. Correlation between features can shift while every marginal stays put, and nothing here would see it. Multivariate drift needs a different tool.
• The permutation path is slow. Wasserstein and total variation have no closed-form null here, so their thresholds are simulated. That is a few seconds per feature, cached per sample size.
• Independence is assumed across checks. The Sidak correction and the run length calibration both assume successive batches are independent. Weekly batches of the same seasonal process are not, and the true false-alarm rate will be somewhat higher than advertised.
• Detecting drift is not detecting harm. A feature can move a long way without the model caring, and can stay still while the relationship between it and the target rots. This library measures the first thing. Watching the second needs labels.
• Below about 100 samples, nothing works. The thresholds are honest there, which mostly means they are honestly enormous. A detector that cannot fire is correctly calibrated and useless. Detector says so in r.warning.

Running it

pip install numpy scipy pandas

python tests/test_null.py         # 9 contracts on the null distributions
python tests/test_detect.py       # 10 on the detector and monitor APIs
python tests/test_sequential.py   # 8 on run lengths and detection delay

python benchmarks/false_alarms.py # the headline table, about 7 minutes
python benchmarks/power.py        # detector against drift type, about 4 minutes
python assets/make_visuals.py     # the six figures

27 tests. The interesting ones assert the findings themselves, so if the industry thresholds ever stop being miscalibrated, the suite will say so.

Papers

• B. Yurdakul (2018), Statistical properties of population stability index, Western Michigan University dissertation. Derives the asymptotic distribution, and states plainly that the 0.10 and 0.25 traffic-light values have no support in the literature.
• Bracher et al. and the wider proper-scoring literature for why a decision rule needs a stated error rate rather than a threshold.
• Page (1954), Continuous inspection schemes, Biometrika. The original CUSUM, and the reason average run length is the right quantity for a monitor that runs forever.
• Benjamini and Hochberg (1995), Controlling the false discovery rate, JRSS-B. What Monitor uses when you hand it fifty features.