A point model tells you its best answer. Conformity helps you ask how uncertain that answer is, using independent held-out observations to calibrate prediction intervals or sets of plausible class labels. Wrap a scikit-learn estimator or pipeline, keep ordinary point predictions for model development, and request conformal output when your decision needs a range or several candidates.

Installation

uv add conformity-calib
# Or:
pip install conformity-calib

Requires Python 3.9+, NumPy >=2.0.2 and scikit-learn >=1.6.1. Install matplotlib separately for plotting.

Choose a component

Quick start

Reserve a test set for evaluation. Put learned preprocessing inside the wrapped estimator so it never fits on calibration observations. Here alpha=0.1 requests 90% marginal coverage:

from sklearn.datasets import make_regression
from sklearn.linear_model import Ridge
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

from conformity import ConformalRegressor

X, y = make_regression(n_samples=1000, n_features=5, noise=15, random_state=7)
X_dev, X_test, y_dev, y_test = train_test_split(X, y, test_size=0.25, random_state=8)
model = ConformalRegressor(
    make_pipeline(StandardScaler(), Ridge()),
    calibration_size=0.25,
    random_state=9,
).fit(X_dev, y_dev)

points, intervals = model.predict(X_test, alpha=0.1, return_interval=True)
assert points.shape == (250,)
assert intervals.shape == (250, 2)
print(model.evaluate(X_test, y_test, alpha=0.1))

The report pairs point error with coverage, width and interval score. Bounds are symmetric and constant-width at a fixed alpha; lower miscoverage usually makes them wider. Test coverage fluctuates around the population target. The complete walkthrough explains the configuration, results and next steps.

For classification, predict_set returns a boolean membership matrix in fitted classes_ order. predict(..., return_set=True) returns (membership, probabilities). Run the classification example to interpret label sets, empty sets and review decisions.

Guarantees and limitations

Coverage is marginal under exchangeability, with fitting and model selection independent of final calibration. It does not guarantee coverage for each individual or subgroup. Distribution shift, time dependence and calibration leakage can invalidate the guarantee. Very small calibration sets can require infinite intervals or all-class sets.

Regression supports one numeric target with symmetric absolute-residual intervals. Classification supports LAC and deterministic APS scores and requires valid predict_proba output. Calibration is unweighted; weights affect fitting or evaluation only.

Read calibration and assumptions and the theory guide before making decisions from uncertainty output.

Documentation

The documentation site covers installation, plotting, guides, estimator classes, diagnostics, runnable examples, generated API reference and development. It becomes available after the first successful Pages deployment.

Start with getting started, then examples. The original README workflows, existing API guide, and theory are preserved in full.

Development and releases

uv python install 3.12
uv sync --locked --group docs --group release --python 3.12
bash scripts/validate.sh

CI tests Python 3.9, 3.10, 3.11 and 3.12. Documentation and release tooling use locked dependencies on Python 3.12. See development and deployment for local preview, reference generation, distribution builds, GitHub Pages settings and PyPI trusted publishing.

For a release, update project.version and conformity.__version__ together, refresh uv.lock, validate and push the revision, then run Create GitHub Release. It rejects existing tags and publishes checked artifacts through the separate PyPI workflow. No release is created by the local build commands above.