A point model tells you its best answer. Conformity helps you ask how uncertain that answer is, using independent held-out observations to calibrate prediction intervals or sets of plausible class labels. Wrap a scikit-learn estimator or pipeline, keep ordinary point predictions for model development, and request conformal output when your decision needs a range or several candidates.

Installation

uv add conformity-calib
# Or:
pip install conformity-calib

Requires Python 3.9+, NumPy >=2.0.2 and scikit-learn >=1.6.1. Install matplotlib separately for plotting.

Choose a component

Quick start

Wrap the complete pipeline so automatic calibration splits before any learned preprocessing fits. Automatic calibration reserves 20% of rows passed to fit by default; set calibration_splitter=ShuffleSplit(n_splits=1, test_size=0.25) to change that. Here alpha=0.1 requests 90% marginal coverage:

from sklearn.datasets import make_regression
from sklearn.linear_model import Ridge
from sklearn.model_selection import train_test_split, ShuffleSplit
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

from conformity import ConformalRegressor

X, y = make_regression(n_samples=1000, n_features=5, noise=15, random_state=7)
X_dev, X_test, y_dev, y_test = train_test_split(X, y, test_size=0.25, random_state=8)
model = ConformalRegressor(
    Pipeline([("scaler", StandardScaler()), ("ridge", Ridge())]),
    calibration_splitter=ShuffleSplit(n_splits=1, test_size=0.2, random_state=9),
).fit(X_dev, y_dev)

points, intervals = model.predict(X_test, alpha=0.1, return_interval=True)
assert points.shape == (250,)
assert intervals.shape == (250, 2)
print(model.evaluate(X_test, y_test, alpha=0.1))

The report pairs point error with coverage, width and interval score. The default bounds are symmetric and constant-width at a fixed alpha; choose the asymmetric regression method when lower and upper residual corrections should be calibrated separately. Test coverage fluctuates around the population target. The complete walkthrough explains the configuration, results and next steps.

For classification, predict_set returns a boolean membership matrix in fitted classes_ order. predict(..., return_set=True) returns (membership, probabilities). Run the classification example to interpret label sets, empty sets and review decisions.

Guarantees and limitations

Coverage is marginal under exchangeability, with fitting and model selection independent of final calibration. It does not guarantee coverage for each individual or subgroup. Distribution shift, time dependence and calibration leakage can invalidate the guarantee. Very small calibration sets can require infinite intervals or all-class sets.

Regression supports one numeric target with symmetric absolute-residual or asymmetric signed-residual intervals. Classification supports LAC and deterministic APS scores and requires valid predict_proba output. Calibration is unweighted; weights affect fitting or evaluation only.

Read calibration and assumptions and the theory guide before making decisions from uncertainty output.

Documentation

The documentation site covers installation, plotting, guides, estimator classes, diagnostics, runnable examples, generated API reference and development. It becomes available after the first successful Pages deployment.

Start with getting started, then browse the examples, user guides, and theory.

Development

Use the locked checkout environment and run bash scripts/validate.sh. See development for setup, documentation preview, reference regeneration, example validation, tests, linting and local distribution builds.