Calibration
a python package for calibrating the deep learning models
Install
pip install calibrator
Use case
from calibrator import LocalCalibrator
import torch
val_logits = torch.randn(1000, 10)
val_labels = torch.randint(0, 10, (1000,))
test_logits = torch.randn(1000, 10)
calibrator = LocalCalibrator()
eps_opt = calibrator.fit(val_logits, val_labels)
calibrated_probability = calibrator.calibrate(test_logits)
Benchmarking
| Datasets |
Method |
Method Description |
ECE |
ACE |
MCE |
CECE |
PIECE |
| CIFAR-10 |
|
|
|
|
|
|
|
| CIFAR-100 |
|
|
|
|
|
|
|
| ImageNet-1K |
|
|
|
|
|
|
|
Package Information
Post-Hoc Calibration methods
| Calibration methods |
Description |
Paper |
Source Code |
Status |
| Temperature Scaling (TS) |
ICML 2017 |
paper |
code |
✅ Implemented |
| Parameterized Temperature Scaling (PTS) |
ECCV 2022 |
paper |
code |
🔜 Pending |
| Ensemble Temperature Scaling (ETS) |
ICML 2020 |
paper |
code |
🔜 Pending |
| Class-based Temperature Scaling (CTS) |
EUSIPCO 2021 |
paper |
unavailable |
🔜 Pending |
| Group Calibration with Temperature Scaling (GCTS) |
NeurIPS 2023 |
paper |
code |
🔜 Pending |
| Proximity-informed Calibration (PROCAL) |
NeurIPS 2023 |
paper |
code |
🔜 Pending |
| Isotonic Regression |
|
|
|
|
| Histogram Binning |
|
|
|
|
| Platt Scaling |
|
|
|
|
| Bayesian Binning into Quantiles (BBQ) |
AAAI 2015 |
paper |
|
🔜 Pending |
| BetaCal |
|
|
|
|
| Scaling-Binning Calibrator |
NeuIPS 2019 |
paper |
code |
🔜 Pending |
| Dirichlet calibration |
NeurIPS 2019 |
paper |
code |
🔜 Pending |
Train-time Calibration Methods
| Calibration Losses |
Description |
Paper |
Source Code |
Status |
| Focal Loss |
|
|
|
|
| Dual Focal Loss |
|
|
|
|
| Adaptive Focal Loss |
|
|
|
|
Metrics
| Metrics |
Description |
Paper |
Source Code |
Status |
| Expected Calibration Error (ECE) |
AAAI 2015 |
paper |
code |
✅ Implemented |
| Maximum Calibration Error (MCE) |
AAAI 2015 |
paper |
code |
🔜 Pending |
| Adaptive Calibration Error (ACE) |
CVPRW 2019 |
paper |
code |
✅ Implemented |
| Classwise Expected Calibration Error (CECE) |
NeurIPS 2019 |
paper |
code |
✅ Implemented |
| Negative Log Likelihood (NLL) |
|
|
|
|
| Accuracy |
|
|
|
|
| Proximity-informed Expected Calibration Error (PIECE) |
NeurIPS 2023 |
paper |
code |
🔜 Pending |
Pre-trained Model Weights
| Datasets |
Description |
Paper |
Source Code |
Status |
| CIFAR-10 |
|
|
|
|
| CIFAR-100 |
|
|
|
|
| ImageNet |
|
|
|
|
| ImageNet-100 |
|
|
|
|
| ImageNet-1000 |
|
|
|
|
Pre-calculated Logits
| Datasets |
Description |
Paper |
Source Code |
Status |
| CIFAR-10 |
|
|
|
|
| CIFAR-100 |
|
|
|
|
| ImageNet |
|
|
|
|
| ImageNet-100 |
|
|
|
|
| ImageNet-1000 |
|
|
|
|
