lightning-uq-box 0.2.0

Lightning-UQ-Box: A toolbox for uncertainty quantification in deep learning

 

lightning-uq-box

The lightning-uq-box is a PyTorch library that provides various Uncertainty Quantification (UQ) techniques for modern neural network architectures.

We hope to provide the starting point for a collaborative open source effort to make it easier for practitioners to include UQ in their workflows and remove possible barriers of entry. Additionally, we hope this can be a pathway to more easily compare methods across UQ frameworks and potentially enhance the development of new UQ methods for neural networks.

The project is currently under active development, but we nevertheless hope for early feedback, feature requests, or contributions. Please check the Contribution Guide for further information.

The goal of this library is threefold:

1. Provide implementations for a variety of Uncertainty Quantification methods for Modern Deep Neural Networks that work with a range of neural network architectures and have different theoretical underpinnings
2. Make it easy to compare UQ methods on a given dataset
3. Focus on reproducibility of experiments with minimum boiler plate code and standardized evaluation protocols

To this end, each UQ-Method is essentially just a Lightning Module which can be used with a Lightning Data Module and a Trainer to execute training, evaluation and inference for your desired task. The library also utilizes the Lightning Command Line Interface (CLI) for better reproducibility of experiments and setting up experiments at scale.

Theory Guide

For a comprehensive document that provides more mathematical details for each method and generally forms the basis of our implementations, please see the Theory Guide. As a living document, we plan to update it as the library encompasses more methods. If you have any questions, or find typos or errors, feel free to open an issue.

Installation

The recommended way to install the latest released version is via pip,

pip install lightning-uq-box

For the latest development version you can run,

pip install git+https://github.com/lightning-uq-box/lightning-uq-box.git

The package is also available for installation via conda or spack. You can find instructions in the documention

UQ-Methods

In the tables that follow below, you can see what UQ-Method/Task combination is currently supported by the Lightning-UQ-Box via these indicators:

• ✅ supported
• ❌ not designed for this task
• ⏳ in progress

The implemented methods are of course not exhaustive, as the number of new methods keeps increasing. For an overview of methods that we are tracking or are planning to support, take a look at this issue.

Classification of UQ-Methods

The following sections aims to give an overview of different UQ-Methods by grouping them according to some commonalities. We agree that there could be other groupings as well and welcome suggestions to improve this overview. We also follow this grouping for the API documentation in the hopes to make navigation easier.

Single Forward Pass Methods

UQ-Method	Regression	Classification	Segmentation	Pixel Wise Regression
Quantile Regression (QR)	✅	❌	❌	✅
Deep Evidential (DE)	✅	⏳	⏳	✅
Mean Variance Estimation (MVE)	✅	❌	❌	✅
ZigZag	✅	✅	❌	❌
Mixture Density Networks	✅	❌	❌	⏳

Approximate Bayesian Methods

UQ-Method	Regression	Classification	Segmentation	Pixel Wise Regression
Bayesian Neural Network VI ELBO (BNN_VI_ELBO)	✅	✅	✅	⏳
Bayesian Neural Network VI (BNN_VI)	✅	⏳	⏳	⏳
Deep Kernel Learning (DKL)	✅	✅	❌	❌
Deterministic Uncertainty Estimation (DUE)	✅	✅	❌	❌
Laplace Approximation (Laplace)	✅	✅	❌	❌
Monte Carlo Dropout (MC-Dropout)	✅	✅	✅	✅
Stochastic Gradient Langevin Dynamics (SGLD)	✅	✅	⏳	⏳
Spectral Normalized Gaussian Process (SNGP)	✅	✅	❌	❌
Stochastic Weight Averaging Gaussian (SWAG)	✅	✅	✅	✅
Variational Bayesian Last Layer (VBLL)	✅	✅	❌	❌
Deep Ensemble	✅	✅	✅	✅
Masked Ensemble	✅	✅	⏳	⏳
Density Uncertainty Layer	✅	✅	❌	❌

Generative Models

UQ-Method	Regression	Classification	Segmentation	Pixel Wise Regression
Classification And Regression Diffusion (CARD)	✅	✅	❌	❌
Probabilistic UNet	❌	❌	✅	❌
Hierarchical Probabilistic UNet	❌	❌	✅	❌
Variational Auto-Encoder (VAE)	❌	❌	❌	✅

Post-Hoc methods

UQ-Method	Regression	Classification	Segmentation	Pixel Wise Regression
Test Time Augmentation (TTA)	✅	✅	⏳	⏳
Temperature Scaling	❌	✅	⏳	❌
Conformal Quantile Regression (Conformal QR)	✅	❌	❌	⏳
Regularized Adaptive Prediction Sets (RAPS)	❌	✅	❌	❌
Image to Image Conformal	❌	❌	❌	✅

Tutorials

We try to provide many different tutorials so that users can get a better understanding of implemented methods and get a feel for how they apply to different problems. Head over to the tutorials page to get started. These tutorials can also be launched in google colab if you navigate to the rocket icon at the top of a tutorial page.

Documentation

We aim to provide an extensive documentation on all included UQ-methods that provide some theoretical background, as well as tutorials that illustrate these methods on toy datasets.

Citation

If you use this software in your work, please cite our paper:

@article{Lehmann_Lightning_UQ_Box_2024,
  author = {Lehmann, Nils and Gawlikowski, Jakob and Stewart, Adam J. and Jancauskas, Vytautas and Depeweg, Stefan and Nalisnick, Eric and Gottschling, Nina M.},
  journal = {arXiv preprint arXiv:2410.03390},
  title = {{Lightning UQ Box}: A Comprehensive Framework for Uncertainty Quantification in Deep Learning},
  year = {2024}
}