conformalpy

A Python library for Conformal Prediction with a focus on operational deployment, visualization, and explainability.

Features

• Classification: Binary and multiclass with ICP, CV+, and Mondrian methods
• Regression: Split conformal, CQR, Normalized, and Jackknife+ methods
• Runtime Alpha: Change confidence level at prediction time without recalibration
• Operational Framework: Outcome categorization (SC/SI/TS0/TS1 for binary; SC/SI/MC/MU for multiclass), decision rules, cost-benefit analysis
• Rich Visualization: Coverage analysis, prediction intervals, FCODs (Feature-Conditioned Outcome Distributions)
• sklearn Compatible: Follows scikit-learn API conventions

Installation

pip install conformalpy

For development:

pip install conformalpy[dev]

Quick Start

Classification

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier

from conformalpy.classifier import ConformalClassifier
from conformalpy.nonconformity.classification import lac_nonconformity

# Load data and split
X, y = load_iris(return_X_y=True)
X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.4, random_state=42)
X_calib, X_test, y_calib, y_test = train_test_split(X_temp, y_temp, test_size=0.5, random_state=42)

# Train your model
model = RandomForestClassifier(random_state=42)
model.fit(X_train, y_train)

# Create conformal classifier
conformal = ConformalClassifier(
    model=model,
    alpha=0.1,  # 90% coverage target
    nonconformity_function=lac_nonconformity
)

# Calibrate on held-out data
conformal.calibrate(X_calib, y_calib)

# Get prediction sets
prediction_sets = conformal.predict(X_test)
# Example output: [[0], [1, 2], [2], ...]

# Runtime alpha: get sets at different confidence levels without recalibration
sets_95 = conformal.predict(X_test, alpha=0.05)  # 95% coverage
sets_80 = conformal.predict(X_test, alpha=0.20)  # 80% coverage

Regression

from sklearn.datasets import fetch_california_housing
from sklearn.ensemble import GradientBoostingRegressor

from conformalpy.regressor import ConformalRegressor

# Load data
X, y = fetch_california_housing(return_X_y=True)
X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.4, random_state=42)
X_calib, X_test, y_calib, y_test = train_test_split(X_temp, y_temp, test_size=0.5, random_state=42)

# Train model
model = GradientBoostingRegressor(random_state=42)
model.fit(X_train, y_train)

# Create conformal regressor
conformal = ConformalRegressor(model=model, alpha=0.1)
conformal.calibrate(X_calib, y_calib)

# Get prediction intervals
intervals = conformal.predict_interval(X_test)
# intervals.shape = (n_samples, 2) -> [lower, upper]

# Runtime alpha: multiple confidence levels at once
intervals_multi = conformal.predict_interval(X_test, alpha=[0.05, 0.10, 0.20])
# intervals_multi.shape = (n_samples, 2, 3)

Main Components

Classification

from conformalpy.classifier import ConformalClassifier
from conformalpy.nonconformity.classification import (
    lac_nonconformity,    # LAC (Least Ambiguous Classifier): smallest sets
    aps_nonconformity,    # APS (Adaptive Prediction Sets): no empty sets
    raps_nonconformity,   # RAPS (Regularized APS): balanced
    saps_nonconformity,   # SAPS (Sorted Adaptive Prediction Sets): sorted variant
    hinge_nonconformity,  # Hinge: margin-based scores
)

# Standard ICP
icp = ConformalClassifier(model, alpha=0.1, nonconformity_function=lac_nonconformity)

# Mondrian (class-conditional coverage)
mondrian = ConformalClassifier(model, alpha=0.1, nonconformity_function=lac_nonconformity, mondrian=True)

# Flexible Mondrian (arbitrary groups)
conformal.calibrate(X_calib, y_calib, groups=group_labels)
conformal.predict(X_test, groups=test_groups)

# CV+ (cross-validation based)
from conformalpy.classifier import CVPlusClassifier
cv_plus = CVPlusClassifier(model, alpha=0.1, n_splits=5)
cv_plus.fit(X_train, y_train)  # No separate calibration needed

Regression

from conformalpy.regressor import ConformalRegressor, CQR, NormalizedConformalRegressor

# Split conformal (symmetric intervals)
split = ConformalRegressor(model, alpha=0.1)

# CQR (adaptive intervals via quantile regression)
cqr = CQR(model_lower=quantile_lower, model_upper=quantile_upper, alpha=0.1)

# Normalized (scaled by predicted uncertainty)
normalized = NormalizedConformalRegressor(model, sigma_model, alpha=0.1)

Evaluation

from conformalpy.evaluation import (
    coverage_score,
    average_set_size,
    class_conditional_coverage,
    interval_coverage_score,
    average_interval_width,
    winkler_score,
)

# Classification metrics
coverage = coverage_score(prediction_sets, y_test)
avg_size = average_set_size(prediction_sets)
class_cov = class_conditional_coverage(prediction_sets, y_test)

# Regression metrics
cov = interval_coverage_score(y_test, intervals)
width = average_interval_width(intervals)
winkler = winkler_score(y_test, intervals, alpha=0.1)

Operational Framework

from conformalpy.outcomes import categorize_outcomes, outcome_summary
from conformalpy.framework import classify_zones, compute_expected_cost

# Binary: SC (correct), SI (incorrect), TS0/TS1 (two-set)
# Multiclass: SC (correct), SI (incorrect), MC (multi-covered), MU (multi-uncovered)
outcomes = categorize_outcomes(prediction_sets, y_test)

# Operational zones: Safe, Uncertain, Problematic
zones = classify_zones(prediction_sets, y_test)

# Cost-benefit analysis
from conformalpy.framework import CostMatrix
costs = CostMatrix(false_negative_cost=1000, false_positive_cost=100, manual_review_cost=50)
y_pred = model.predict(X_test)
cost_report = compute_expected_cost(prediction_sets, y_pred, y_test, costs)  # dict with cost breakdown

Score Function Comparison

Method	Empty Sets	Set Size	Best For
LAC	Possible	Smallest	When empty sets are acceptable
APS	Never	Larger	When adaptiveness is important
RAPS	Never	Small	General use (recommended)
SAPS	Never	Small	Sorted variant of APS
Hinge	Possible	Smallest	Margin-based scoring

Additional Features

Time Series

Adaptive Conformal Inference (ACI) for sequential data under distribution shift.

from conformalpy.timeseries import AdaptiveConformalRegressor

aci = AdaptiveConformalRegressor(model=model, target_coverage=0.9, gamma=0.01)

Risk Control

Conformal Risk Control (CRC), RCPS, and Learn-then-Test (LTT) for multi-label and custom loss functions.

from conformalpy.risk_control import ConformalRiskController

FCODs (Feature-Conditioned Outcome Distributions)

Visualize how prediction outcomes (SC, SI, MC, MU, etc.) vary as a function of input features — an original contribution of conformalpy.

from conformalpy.fcod import compute_fcod, plot_fcod

Multi-Output Regression

Marginal, Bonferroni-corrected, and max-residual strategies for multi-target regression.

from conformalpy.regressor import MultiOutputConformalRegressor

Jackknife+

Leave-one-out conformal regressor with finite-sample coverage guarantees.

from conformalpy.regressor import JackknifePlusRegressor

Conformal Predictive Distributions

Full predictive distributions (CDFs) via conformal prediction.

from conformalpy.regressor import ConformalPredictiveDistribution

Deep Learning Integration

Adapters for PyTorch and TensorFlow models.

from conformalpy.deep_learning import TorchClassifierWrapper

Diagnostics

Exchangeability testing via martingales and plugin methods.

from conformalpy.diagnostics import ExchangeabilityTest

Also available and documented on the docs site: weighted / clustered / group-balanced classifiers, fairness metrics, SHAP dependence plots, model persistence, MLflow integration, and optional GPU backends.

Citation

The conformalpy library paper is in preparation. If you use the FCOD or p-value-margin SHAP functionality, please cite:

@inproceedings{caparrini2026explaining,
  author    = {Antonio Caparrini and Miller-Janny Ariza-Garz{\'o}n and Javier Arroyo},
  title     = {Explaining Conformal Prediction: Diagnosing Reliability through Feature-Conditioned Outcomes and p-Value Margins},
  booktitle = {Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications},
  series    = {Proceedings of Machine Learning Research},
  volume    = {329},
  year      = {2026},
  publisher = {PMLR},
  note      = {Conditionally accepted}
}

Benchmarks

conformalpy achieves equivalent coverage to established libraries while being faster:

Library	Avg Time (Classification)	Coverage
conformalpy	0.04s	Valid
Crepes	0.11s	Valid

See benchmarks/ for full comparison.

Documentation

Full documentation: caparrini.github.io/conformalpy

To build and preview the documentation locally:

# Install documentation dependencies
uv sync --extra dev --extra docs --extra explainability

# Build the documentation
cd docs && uv run quarto render

# Preview the documentation (in another terminal)
cd docs && uv run quarto preview

The documentation uses Quarto with quartodoc for API reference generation. See docs/_quarto.yml for the full configuration.

References

Conformal Prediction:

• Vovk, V., Gammerman, A., & Shafer, G. (2005). Algorithmic Learning in a Random World. Springer.
• Angelopoulos, A. N., & Bates, S. (2023). Conformal Prediction: A Gentle Introduction. Foundations and Trends in ML.

FCOD and SHAP Margins:

• Caparrini, A., Ariza-Garzón, M.-J., & Arroyo, J. (2026). Explaining Conformal Prediction: Diagnosing Reliability through Feature-Conditioned Outcomes and p-Value Margins. COPA, PMLR 329. (Conditionally accepted)

Score Functions:

• Sadinle, M., Lei, J., & Wasserman, L. (2019). Least Ambiguous Set-Valued Classifiers With Bounded Error Levels. JASA. (LAC)
• Romano, Y., Sesia, M., & Candès, E. J. (2020). Classification with Valid and Adaptive Coverage. NeurIPS. (APS)
• Angelopoulos, A. N., et al. (2021). Uncertainty Sets for Image Classifiers using Conformal Prediction. ICLR. (RAPS)
• Huang, J., et al. (2024). Conformal Prediction for Deep Classifier via Label Ranking. ICML. (SAPS)
• Romano, Y., Patterson, E., & Candès, E. J. (2019). Conformalized Quantile Regression. NeurIPS. (CQR)

Additional Methods:

• Gibbs, I. & Candès, E. (2021). Adaptive Conformal Inference Under Distribution Shift. NeurIPS. (ACI)
• Barber, R. F., et al. (2021). Predictive Inference with the Jackknife+. Annals of Statistics.
• Bates, S., et al. (2021). Distribution-Free, Risk-Controlling Prediction Sets. Journal of the ACM.

Contributing

Contributions are welcome! See CONTRIBUTING.md for the full workflow.

# Clone the repository
git clone https://github.com/Caparrini/conformalpy.git
cd conformalpy

# Install with uv
uv sync --extra dev --extra docs --extra explainability

# Run tests
uv run pytest tests/

# Build API reference from Python docstrings
cd docs && uv run quartodoc build

# Render docs website
cd docs && uv run quarto render

# Preview docs (in another terminal)
cd docs && uv run quarto preview

License

This project is licensed under the MIT License.

See CHANGELOG.md for release history.