Python implementation of Approximate full Conformal Prediction (ACP)
Project description
Approximate full Conformal Prediction
This repository contains the Python implementation of Approximating Full Conformal Prediction at Scale via Influence Functions.
Overview
Approximate full Conformal Prediction (ACP) outputs a prediction set that contains the true label with at least a probability specified by the practicioner. In large datasets, ACP inherits the statistical power of the highly efficient full Conformal Prediction. The method works as a wrapper for any differentiable ML model.
Contents
This repository is organized as follows. In the folder src/acp
you can find the following modules:
methods.py
Python implementation of the ACP algorithms.others.py
Python implementation of the comparing methods (SCP, APS, RAPS, CV+, JK+).wrapper.py
Python implementation of ACP as a wrapper for any differentiable PyTorch model. Seemodels.py
for examples.models.py
Examples of models compatible withwrapper.py
(e.g., logistic regression, neural network, convolutional neural network).experiments.py
Python file to run the experiments from the command line.models/
Saved models.
The folder src/third_party/
contains additional third-party software.
Third-party software
We include the following third-party packages for comparison with ACP:
Usage
Installation
ACP can be utilized as a fully-independent pip
package. You can download the framework by running the following command in the terminal:
pip install approx-cp
In order to use ACP in your own models, just include the following imports in your file:
from acp.wrapper import ACP_D, ACP_O #Deleted scheme (ACP_D) and ordinary scheme (ACP_O)
Alternatively, you can clone this repo by running:
git clone https://github.com/cambridge-mlg/acp
cd acp
And install the ACP Python package in a customizable conda environment:
conda create -n myenv python=3.9
conda activate myenv
pip install --upgrade pip
pip install -e .
Now, just include the import:
from acp.wrapper import ACP_D, ACP_O
Constructing prediction sets with ACP
ACP works as a wrapper for any PyTorch model with .fit()
and .predict()
methods. Once you instantiate your model, you can generate tight prediction sets that contain the true label with a specified probability. Here is an example with synthetic data:
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from acp.models import NeuralNetwork
from acp.wrapper import ACP_D
X, Y = make_classification(n_samples = 1100, n_features = 10, n_classes = 2, n_clusters_per_class = 1, n_informative = 3, random_state = 42)
Xtrain, Xtest, Ytrain, Ytest = train_test_split(X, Y, test_size = 100, random_state = 42)
model = NeuralNetwork(input_size = 10, num_neurons = [20, 10], out_size = 2, seed = 42, l2_reg = 0.01)
ACP = ACP_D(Xtrain, Ytrain, model, seed = 42, verbose = True)
sets = ACP.predict(Xtest, epsilon = 0.1, out_file = "results/test")
Reference
J. Abad Martinez, U. Bhatt, A. Weller and G. Cherubin. Approximating Full Conformal Prediction at Scale via Influence Functions. Association for the Advancement of Artificial Intelligence Conference on Artificial Intelligence (AAAI), 2023.
BiBTeX:
@inproceedings{Abad2023ApproximatingFC,
title={Approximating Full Conformal Prediction at Scale via Influence Functions},
author={Javier Abad and Umang Bhatt and Adrian Weller and Giovanni Cherubin},
booktitle={AAAI Conference on Artificial Intelligence},
year={2023}
}
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for approx_cp-0.0.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d43940f641b9dea51f79ffc258c3d3a9084f7125b867705620bf4eb391648c88 |
|
MD5 | c6cb88f47e437878a18719329cefb3fc |
|
BLAKE2b-256 | 24f69a0136066fac52af1f6034af9481180d61596addfc4cde61ac52b6fda9a5 |