MAPIE 1.0.1

A scikit-learn-compatible module for estimating prediction intervals.

MAPIE - Model Agnostic Prediction Interval Estimator

🎉 MAPIE v1.0.0 is live! 🎉 You’re seeing the documentation of this new version, which introduces major changes to the API. Extensive release notes are available here. You can switch to the documentation of previous versions using the button on the bottom right of ReadTheDoc pages.

MAPIE is an open-source Python library for quantifying uncertainties and controlling the risks of machine learning models.

MAPIE allows you to:

• Compute prediction intervals or prediction sets for regression [3,4,8], classification [5-7], and time series [9], by estimating your model uncertainty on a conformalization dataset.

• Control risks of more complex tasks such as multi-label classification, semantic segmentation in computer vision, providing probabilistic guarantees on metrics like recall and precision [10-12].

• Easily use any model (scikit-learn, TensorFlow, PyTorch) thanks to scikit-learn-compatible wrapper if needed. MAPIE is part of the scikit-learn-contrib ecosystem.

MAPIE relies notably on the fields of Conformal Prediction and Distribution-Free Inference. It implements peer-reviewed algorithms that are model and use case agnostic and possesses theoretical guarantees under minimal assumptions on the data and the model.

🛠 Requirements & installation

MAPIE runs on:

• Python >=3.9, <3.12

• NumPy >=1.23

• scikit-learn >=1.4

MAPIE can be installed in different ways:

$ pip install mapie  # installation via `pip`
$ conda install -c conda-forge mapie  # or via `conda`
$ pip install git+https://github.com/scikit-learn-contrib/MAPIE  # or directly from the github repository

⚡ Quickstart and documentation

Below are two simple examples from our documentation that show how MAPIE is used in a regression setting and a classification setting:

• Uncertainty quantification for a regression problem

• Uncertainty quantification for a classification problem

📝 Contributing

You are welcome to propose and contribute new ideas. We encourage you to open an issue so that we can align on the work to be done. It is generally a good idea to have a quick discussion before opening a pull request that is potentially out-of-scope. For more information on the contribution process, read our contribution guidelines.

🤝 Affiliations

MAPIE has been developed through a collaboration between Capgemini Invent, Quantmetry, Michelin, ENS Paris-Saclay, and with the financial support from Région Ile de France and Confiance.ai.

🔍 References

[1] Vovk, Vladimir, Alexander Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World. Springer Nature, 2022.

[2] Angelopoulos, Anastasios N., and Stephen Bates. “Conformal prediction: A gentle introduction.” Foundations and Trends® in Machine Learning 16.4 (2023): 494-591.

[3] Rina Foygel Barber, Emmanuel J. Candès, Aaditya Ramdas, and Ryan J. Tibshirani. “Predictive inference with the jackknife+.” Ann. Statist., 49(1):486–507, (2021).

[4] Kim, Byol, Chen Xu, and Rina Barber. “Predictive inference is free with the jackknife+-after-bootstrap.” Advances in Neural Information Processing Systems 33 (2020): 4138-4149.

[5] Sadinle, Mauricio, Jing Lei, and Larry Wasserman. “Least ambiguous set-valued classifiers with bounded error levels.” Journal of the American Statistical Association 114.525 (2019): 223-234.

[6] Romano, Yaniv, Matteo Sesia, and Emmanuel Candes. “Classification with valid and adaptive coverage.” Advances in Neural Information Processing Systems 33 (2020): 3581-3591.

[7] Angelopoulos, Anastasios, et al. “Uncertainty sets for image classifiers using conformal prediction.” International Conference on Learning Representations (2021).

[8] Romano, Yaniv, Evan Patterson, and Emmanuel Candes. “Conformalized quantile regression.” Advances in neural information processing systems 32 (2019).

[9] Xu, Chen, and Yao Xie. “Conformal prediction interval for dynamic time-series.” International Conference on Machine Learning. PMLR, (2021).

[10] Bates, Stephen, et al. “Distribution-free, risk-controlling prediction sets.” Journal of the ACM (JACM) 68.6 (2021): 1-34.

[11] Angelopoulos, Anastasios N., Stephen, Bates, Adam, Fisch, Lihua, Lei, and Tal, Schuster. “Conformal Risk Control.” (2022).

[12] Angelopoulos, Anastasios N., Stephen, Bates, Emmanuel J. Candès, et al. “Learn Then Test: Calibrating Predictive Algorithms to Achieve Risk Control.” (2022).

Image credits: Portrait: Cemrecan Yurtman / Zebra Horse Hybrid: hogrmahmood

📚 License & citation

MAPIE is free and open-source software licensed under the BSD-3-Clause license.

If you use MAPIE in your research, please cite the main paper:

Cordier, Thibault, et al. “Flexible and systematic uncertainty estimation with conformal prediction via the mapie library.” Conformal and Probabilistic Prediction with Applications. PMLR, 2023.

Or equivalently the BibTex entry:

@inproceedings{Cordier_Flexible_and_Systematic_2023,
author = {Cordier, Thibault and Blot, Vincent and Lacombe, Louis and Morzadec, Thomas and Capitaine, Arnaud and Brunel, Nicolas},
booktitle = {Conformal and Probabilistic Prediction with Applications},
title = {{Flexible and Systematic Uncertainty Estimation with Conformal Prediction via the MAPIE library}},
year = {2023}
}

You can also have a look at the ICML workshop manuscript:

Taquet, Vianney, et al. “MAPIE: an open-source library for distribution-free uncertainty quantification.” arXiv preprint arXiv:2207.12274 (2022).

@article{taquet2022mapie,
title={MAPIE: an open-source library for distribution-free uncertainty quantification},
author={Taquet, Vianney and Blot, Vincent and Morzadec, Thomas and Lacombe, Louis and Brunel, Nicolas},
journal={arXiv preprint arXiv:2207.12274},
year={2022}
}