torchcp 1.1.0

A Python toolbox for conformal prediction on deep learning models.

TorchCP: A Python toolbox for Conformal Prediction in Deep Learning.
Technical Report · Documentation

TorchCP is a Python toolbox for conformal prediction research on deep learning models, built on the PyTorch Library with strong GPU acceleration. In the toolbox, we implement representative methods (including posthoc and training methods) for many tasks of conformal prediction, including: Classification, Regression, Graph Neural Networks, and LLM. We for many tasks of conformal prediction, including: Classification, Regression, Graph Neural Networks, and LLM. We build the basic framework of TorchCP based on AdverTorch. This codebase is still under construction and maintained by Hongxin Wei's research group at SUSTech. Comments, issues, contributions, and collaborations are all welcomed!

Updates of New Version (1.1.0)

This release features a comprehensive refactoring of predictor modules, along with the addition of RC3P, EntmaxScore, and the SCPO trainer. This release features a comprehensive refactoring of predictor modules, along with the addition of RC3P, EntmaxScore, and the SCPO trainer. Detailed changelog can be found in the Documentation.

Overview

TorchCP has implemented the following methods:

Classification

Year	Title	Venue	Code Link	Implementation
2025	Sparse Activations as Conformal Predictors	AISTATS'25	Link	classification.score.entmax
2025	C-Adapter: Adapting Deep Classifiers for Efficient Conformal Prediction Sets	ECAI'25		classification.loss.cd
2024	Conformal Prediction for Class-wise Coverage via Augmented Label Rank Calibration	NeurIPS'24	Link	classification.predictor.rc3p
2024	Does confidence calibration improve conformal prediction?	Arxiv		classification.loss.confts
2024	Conformal Prediction for Deep Classifier via Label Ranking	ICML'24	Link	classification.score.saps
2023	Class-Conditional Conformal Prediction with Many Classes	NeurIPS'23	Link	classification.predictor.cluster
2023	Conformal Prediction Sets for Ordinal Classification	NeurIPS'23		classification.trainer.ordinal
2022	Training Uncertainty-Aware Classifiers with Conformalized Deep Learning	NeurIPS'22	Link	classification.loss.uncertainty_aware
2022	Learning Optimal Conformal Classifiers	ICLR'22	Link	classification.loss.conftr
2021	Optimized conformal classification using gradient descent approximation	Arxiv		classification.loss.scpo
2021	Uncertainty Sets for Image Classifiers using Conformal Prediction	ICLR'21	Link	classification.score.raps classification.score.topk
2020	Classification with Valid and Adaptive Coverage	NeurIPS'20	Link	classification.score.aps
2019	Conformal Prediction Under Covariate Shift	NeurIPS'19	Link	classification.predictor.weight
2016	Least Ambiguous Set-Valued Classifiers with Bounded Error Levels	JASA		classification.score.lac
2016	Hedging Predictions in Machine Learning	The Computer Journal		classification.score.knn
2015	Bias reduction through conditional conformal prediction	Intell. Data Anal.		classification.score.margin
2012	Conditional Validity of Inductive Conformal Predictors	ACML'12		classification.predictor.class_conditional
2007	Hedging Predictions in Machine Learning	The Computer Journal		classification.score.knn

Regression

Year	Title	Venue	Code Link	Implementation	Remark
2023	Conformal Prediction via Regression-as-Classification	RegML @ NeurIPS 2023	link	regression.score.r2ccp
2022	Adaptive Conformal Predictions for Time Series	ICML'22	link	regression.predictor.agaci	support time series
2021	Adaptive Conformal Inference Under Distribution Shift	NeurIPS'21	Link	regression.predictor.aci	support time series
2020	A comparison of some conformal quantile regression methods	Stat	Link	regression.score.cqm regression.score.cqrr
2020	Conformal Prediction Interval for Dynamic Time-Series	ICML'21	Link	regression.predictor.ensemble	support time series
2019	Adaptive, Distribution-Free Prediction Intervals for Deep Networks	AISTATS'19	Link	regression.score.cqrfm
2019	Conformalized Quantile Regression	NeurIPS'19	Link	regression.score.cqr
2017	Distribution-Free Predictive Inference For Regression	JASA	Link	regression.predictor.split
2005	Inductive Confidence Machines for Regression	Springer		regression.score.abs regression.score.norabs

Graph

Year	Title	Venue	Code Link	Implementation
2024	Similarity-Navigated Conformal Prediction for Graph Neural Networks	NeuIPS'24	Link	graph.score.snaps
2023	Distribution Free Prediction Sets for Node Classification	ICML'23	Link	graph.predictor.naps
2023	Conformal Prediction Sets for Graph Neural Networks	ICML'23	Link	graph.score.daps
2023	Uncertainty Quantification over Graph with Conformalized Graph Neural Networks	NeurIPS'23	Link	graph.trainer.cfgnn

Language Models

Year	Title	Venue	Code Link	Implementation
2023	Conformal Language Modeling	ICLR'24	Link	llm.predictor.conformal_llm

TODO

TorchCP is still under active development. We will add the following features/items down the road:

Year	Title	Venue	Code
2023	Conformal Prediction for Time Series with Modern Hopfield Networks	NeuIPS'23	Link
2022	Conformal Prediction Sets with Limited False Positives	ICML'22	Link

Installation

TorchCP is developed with Python 3.9 and PyTorch 2.0.1. To install TorchCP, simply run

pip install torchcp

To install from TestPyPI server, run

pip install --index-url https://test.pypi.org/simple/ --no-deps torchcp

If you encounter errors while installing Torchsort, you can try the following steps to resolve them:

git clone https://github.com/teddykoker/torchsort.git
cd torchsort
pip install .

If you encounter errors while installing Torchsort, you can try the following steps to resolve them:

git clone https://github.com/teddykoker/torchsort.git
cd torchsort
pip install .

Unit Test

TorchCP achieves 100% unit test coverage. You can use the following command to test the code implementation:

pytest --cov=torchcp tests

Examples

Here, we provide a simple example for a classification task, with LAC score and SplitPredictor.

from torchcp.classification.score import LAC
from torchcp.classification.predictor import SplitPredictor

# Preparing a calibration data and a test data.
cal_dataloader = ...
test_dataloader = ...
# Preparing a pytorch model
model = ...

model.eval()

# Options of score function: LAC, APS, SAPS, RAPS
# Define a conformal prediction algorithm. Optional: SplitPredictor, ClusteredPredictor, ClassConditionalPredictor
# We recommend setting both alpha and device during initialization
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
predictor = SplitPredictor(score_function=LAC(), model=model, alpha=0.1, device = device)

# Calibrating the predictor 
# You can also call `calibrate()` again to update the alpha value if needed
#predictor.calibrate(cal_dataloader, alpha=0.1)
predictor.calibrate(cal_dataloader)

#########################################
# Predicting for test instances
########################################
test_instances = ...
predict_sets = predictor.predict(test_instances)
print(predict_sets)

#########################################
# Evaluating the coverage rate and average set size on a given dataset.
########################################
result_dict = predictor.evaluate(test_dataloader)
print(f"Coverage Rate: {result_dict['coverage_rate']:.4f}")
print(f"Average Set Size: {result_dict['average_size']:.4f}")

You may find more tutorials in examples folder.

License

This project is licensed under the LGPL. The terms and conditions can be found in the LICENSE and LICENSE.GPL files.

Citation

If you find our repository useful for your research, please consider citing the following technical report:

@misc{huang2024torchcp,
      title={TorchCP: A Python Library for Conformal Prediction}, 
      author={Jianguo Huang and Jianqing Song and Xuanning Zhou and Bingyi Jing and Hongxin Wei},
      year={2024},
      eprint={2402.12683},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
}

We welcome you to cite the following works:

@inproceedings{huangconformal,
  title={Conformal Prediction for Deep Classifier via Label Ranking},
  author={Huang, Jianguo and Xi, HuaJun and Zhang, Linjun and Yao, Huaxiu and Qiu, Yue and Wei, Hongxin},
  booktitle={Forty-first International Conference on Machine Learning}
}

@article{xi2024does,
  title={Does Confidence Calibration Help Conformal Prediction?},
  author={Xi, Huajun and Huang, Jianguo and Feng, Lei and Wei, Hongxin},
  journal={TMLR},
  year={2024}
}

@inproceedings{
  liu2025cadapter,
  title={C-Adapter: Adapting Deep Classifiers for Efficient Conformal Prediction Sets},
  author={Kangdao Liu and Hao Zeng and Jianguo Huang and Huiping Zhuang and Chi Man VONG and Hongxin Wei},
  booktitle={The 28th European Conference on Artificial Intelligence},
  year={2025},
}

Contributors

• Hongxin Wei
• Jianguo Huang
• Xuanning Zhou
• Jianqing Song