Weighted version of crepes: Conformal classifiers, regressors, and predictive systems (crepes)
Project description
crepes-weighted
crepes-weighted
is an extension of crepes
, a Python package that implements conformal classifiers, regressors, and predictive systems on top of any standard classifier and regressor. crepes-weighted
extends crepes
by adding support for weighted conformal prediction and predictive systems, in the future this could potentially be merged into the main crepes
package.
🛠️ Installation
command | |
---|---|
pip | pip install crepes-weighted |
:rocket: Quick Start
First we create a synthetic dataset using the data-generating process described in Kang and Schafer (2007). This function generates a dataset with 4 features and a target variable. The target variable is generated using the following formula:
import numpy as np
def synthetic_kang_schafer_2007(n=2000, weights=None):
if weights is None:
weights = np.ones(n)/n
x1 = np.random.normal(size=n)
x2 = np.random.normal(size=n)
x3 = np.random.normal(size=n)
x4 = np.random.normal(size=n)
y = 210 + 27.4*x1 + 13.7*x2 + 13.7*x3 + 13.7*x4 + np.random.normal(size=n)
return np.stack([x1, x2, x3, x4], axis=1), y
Next, we first split it into a training and a test set using train_test_split from sklearn, and then further split the training set into a proper training set and a calibration set.
from sklearn.model_selection import train_test_split
X, y = synthetic_kang_schafer_2007()
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5)
X_train, X_cal, y_train, y_cal = train_test_split(X_train, y_train, test_size=0.5)
We now emulate a covariate shift by creating a new dataset with a different distribution, and we calculate the likelihood of each observation under this new distribution.
shift_weights = np.array([-1, 0.5, -0.25, -0.1])
shifted_likelihood_test = np.exp(X_test @ shift_weights)
shifted_weights_test = shifted_likelihood_test / np.sum(shifted_likelihood_test)
shifted_likelihood_cal = np.exp(X_cal @ shift_weights)
def weighted_sample(weights, frac=0.5):
return np.random.choice(range(len(weights)), size=int(len(weights) * frac), p=weights)
idx_no_shift = weighted_sample(np.ones(len(shifted_weights_test))/len(shifted_weights_test), frac=0.25)
idx_shift = weighted_sample(shifted_weights_test, frac=0.25)
We now "wrap" a random forest regressor, fit it to the proper training set, and fit a weighted conformal classifier through the calibrate
method, using the calibration set and a set of weights to account for the covariate shift.
from sklearn.ensemble import RandomForestRegressor
from crepes_weighted import WrapRegressor
rf_wcps = WrapRegressor(RandomForestRegressor(n_estimators=100, random_state=17))
rf_wcps.fit(X_train_prop, y_train_prop)
rf_wcps.calibrate(X_cal, y_cal, likelihood_ratios=shifted_likelihood_cal, cps=True)
Finally, we can make predictions (intervals, p_values, and distributions) on the test set and calculate the coverage of the conformal predictive system.
int_wcps = rf_wcps.predict_int(X_test[idx_shift], y=y_test[idx_shift], likelihood_ratios=shifted_likelihood_test[idx_shift])
dist_wcps = rf_wcps.predict_cps(X_test[idx_shift], y=y_test[idx_shift], likelihood_ratios=shifted_likelihood_test[idx_shift], return_cpds=True)
p_values_wcps = rf_wcps.predict_cps(X_test[idx_shift], y=y_test[idx_shift], likelihood_ratios=shifted_likelihood_test[idx_shift])
Citing crepes-weighted
If you use crepes-weighted
for a scientific publication, you are kindly requested to cite the following paper:
@misc{jonkers2024conformal,
title={Conformal Predictive Systems Under Covariate Shift},
author={Jef Jonkers and Glenn Van Wallendael and Luc Duchateau and Sofie Van Hoecke},
year={2024},
eprint={2404.15018},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
The preprint version of the paper can be found at https://arxiv.org/abs/2404.15018.
We also recommend citing the original crepes
package:
@InProceedings{crepes,
title = {crepes: a Python Package for Generating Conformal Regressors and Predictive Systems},
author = {Bostr\"om, Henrik},
booktitle = {Proceedings of the Eleventh Symposium on Conformal and Probabilistic Prediction and Applications},
year = {2022},
editor = {Johansson, Ulf and Boström, Henrik and An Nguyen, Khuong and Luo, Zhiyuan and Carlsson, Lars},
volume = {179},
series = {Proceedings of Machine Learning Research},
publisher = {PMLR}
}
License
This project is licensed under the BSD 3-Clause License - see the LICENSE file for details.
👤 Jef Jonkers
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for crepes_weighted-0.1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 903312477d14b5fc11f1c01aae35e82b3df9d421ad34de0c5e81a2505c46b00b |
|
MD5 | d1d58340f5439ca0b88d36561e960876 |
|
BLAKE2b-256 | cbf581ef2850c6d30fc1c790ada1d2e1cea545eb61dbab1a4a6446f3984d83e7 |