idaniloju

Conformal prediction from scratch: the exact finite-sample guarantee, proof by experiment, the deployment that breaks it, and the online repair. The tenth and last project of this series, and it closes the loop: the drift that kills the guarantee here is detected by iyipada, my drift library, installed from PyPI like any other user would.

pip install idaniloju

Idaniloju is Yoruba for assurance. Conformal prediction is the cleanest promise in machine learning: wrap ANY point predictor, and its intervals will contain the truth 90% of the time, guaranteed at finite sample size with no distributional assumptions at all. The entire method is one counting argument, and this repository walks it from the theorem to the hurricane that broke it.

The guarantee is a law, not a hope

600 fresh train/calibrate/test splits of the same data. Conditional coverage is not just ">= 90% on average": it has a known exact distribution, Beta(n+1-l, l) over the calibration draw (Vovk 2012). The 600-run empirical mean lands on the law's mean to the fourth digit (0.9001 vs 0.9001), and the histogram sits inside the density like it was poured there. Also visible: 47% of individual runs are below 0.90, which is exactly what the law says should happen, and exactly the fine print people miss.

The whole proof: an exchangeable point's score has uniform rank among the n+1 scores, so it falls at or below the ceil((n+1)(1-alpha))-th calibration score with probability at least 1-alpha. Counting, nothing else. (Source: assets/manim_ranks.py, video in assets/ranks.mp4.)

Then you deploy it, and the fine print detonates

Train on early 2011 of the Washington DC bike-sharing panel, calibrate on late 2011, then serve every hour of 2012 in order:

2012 ridership grew about 65% over 2011. Exchangeability with the calibration set is simply false, and the guarantee follows it down: 78% delivered against 90% promised for the year, and when Hurricane Sandy closes the city on October 29 (22 rides all day against a normal 7,500), the rolling window bottoms out at 42%.

Three things about that figure that I care about:

• iyipada alarms in the first week of 2012 (PSI 0.15 on the model's conformity scores, p = 0.005), eleven months before Sandy. The drift was structural, not meteorological.
• The shuffled control behaves: run the same machinery on a shuffled (exchangeable) version of the stream and coverage holds at 90.3% with zero alarm weeks out of 51. The alarm is not trigger-happy; the guarantee did not break by accident.
• Adaptive conformal repairs it online. ACI (Gibbs & Candes 2021) moves the miscoverage dial with the observed errors, alpha_t+1 = alpha_t + gamma(alpha - err_t), and ends the year at 89.97% against the 90% target, hurricane included, without ever refitting the model.

What the repair costs

Nothing is free: the adaptive intervals are about 5% wider in the median hour, and for 36 hours around Sandy the dial demands more assurance than 4,395 calibration points can certify, so the honest interval is unbounded. I consider that a feature: a system that says "this week I can promise nothing" is telling the truth, and the fixed system that kept quoting tidy intervals through a hurricane was not.

Using it

from idaniloju import Ridge, SplitConformal, AdaptiveConformal, coverage

sc = SplitConformal(Ridge(lam=5.0), alpha=0.1)   # any model with fit/predict
sc.fit(X_train, y_train, X_cal, y_cal)
lo, hi = sc.predict(X_new)

aci = AdaptiveConformal(sc, gamma=0.01)           # for streams
lo, hi, err = aci.step(x_t, y_t)

SplitConformal takes any object with fit/predict (a scale model makes the intervals difficulty-aware without touching the guarantee), and conformal_quantile is the exact order statistic the theorem is proved for, not an interpolated quantile: at small n and small alpha it returns an infinite interval instead of a fake finite one.

Honest limits

• The guarantee is marginal: 90% over everything, not 90% for every hour of the day or every kind of input. Group-conditional coverage needs group-wise calibration this library does not implement.
• ACI trades a broken assumption for a feedback loop: it needs y_t observed reasonably soon after predicting. With delayed labels the dial lags exactly that delay.
• The base model here is a ridge regression on purpose (any model works). Sharper models give narrower intervals; nothing about the coverage story changes.
• The 65% growth and the hurricane make this dataset a vivid drift story, which is why I chose it; datasets with gentler drift break more slowly and less photogenically.

Reproduce it

python -m venv .venv && .venv/bin/pip install numpy pytest iyipada
bash data/get_data.sh                        # or let the loader fetch it

.venv/bin/python -m pytest tests/ -q         # 6 contracts
.venv/bin/python benchmarks/bench_guarantee.py   # the Beta law, 600 refits
.venv/bin/python benchmarks/bench_stream.py      # the 2012 deployment
.venv/bin/python assets/make_visuals.py          # the four figures

Layout

• idaniloju/conformal.py the method: exact quantile, split conformal, difficulty scaling, ACI
• idaniloju/models.py a closed-form ridge, the deliberately boring predictor
• idaniloju/bike.py the panel, with the year column deliberately withheld from the features (deployment does not get told the world changed)
• benchmarks/ the law and the deployment, everything into JSON the figures and tests read

Papers

• Vovk, Gammerman, Shafer (2005), Algorithmic Learning in a Random World. The theory.
• Lei et al. (2018), Distribution-Free Predictive Inference for Regression, JASA. The split conformal recipe used here.
• Vovk (2012), Conditional validity of inductive conformal predictors. The Beta law the first figure draws.
• Gibbs & Candes (2021), Adaptive Conformal Inference Under Distribution Shift. The repair.