lightning-uq-box

The lightning-uq-box is a PyTorch library that provides various Uncertainty Quantification (UQ) techniques for modern neural network architectures.

We hope to provide the starting point for a collaborative open source effort to make it easier for practitioners to include UQ in their workflows and remove possible barriers of entry. Additionally, we hope this can be a pathway to more easily compare methods across UQ frameworks and potentially enhance the development of new UQ methods for neural networks.

The project is currently under active development, but we nevertheless hope for early feedback, feature requests, or contributions. Please check the Contribution Guide for further information.

The goal of this library is threefold:

1. Provide implementations for a variety of Uncertainty Quantification methods for Modern Deep Neural Networks that work with a range of neural network architectures and have different theoretical underpinnings
2. Make it easy to compare UQ methods on a given dataset
3. Focus on reproducibility of experiments with minimum boiler plate code and standardized evaluation protocols

To this end, each UQ-Method is essentially just a Lightning Module which can be used with a Lightning Data Module and a Trainer to execute training, evaluation and inference for your desired task. The library also utilizes the Lightning Command Line Interface (CLI) for better reproducibility of experiments and setting up experiments at scale.

Theory Guide

For a comprehensive document that provides more mathematical details for each method and generally forms the basis of our implementations, please see the Theory Guide. As a living document, we plan to update it as the library encompasses more methods. If you have any questions, or find typos or errors, feel free to open an issue.

Installation

The recommended way to install the latest released version is via pip,

pip install lightning-uq-box

For the latest development version you can run,

pip install git+https://github.com/lightning-uq-box/lightning-uq-box.git

The package is also available for installation via conda or spack. You can find instructions in the documention

UQ-Methods

In the tables that follow below, you can see what UQ-Method/Task combination is currently supported by the Lightning-UQ-Box via these indicators:

• ✅ supported
• ❌ not designed for this task
• ⏳ in progress

The implemented methods are of course not exhaustive, as the number of new methods keeps increasing. For an overview of methods that we are tracking or are planning to support, take a look at this issue.

Classification of UQ-Methods

The following sections aims to give an overview of different UQ-Methods by grouping them according to some commonalities. We agree that there could be other groupings as well and welcome suggestions to improve this overview. We also follow this grouping for the API documentation in the hopes to make navigation easier.

Single Forward Pass Methods

Approximate Bayesian Methods

Generative Models

Post-Hoc methods

Tutorials

We try to provide many different tutorials so that users can get a better understanding of implemented methods and get a feel for how they apply to different problems. Head over to the tutorials page to get started. These tutorials can also be launched in google colab if you navigate to the rocket icon at the top of a tutorial page.

Documentation

We aim to provide an extensive documentation on all included UQ-methods that provide some theoretical background, as well as tutorials that illustrate these methods on toy datasets.

Citation

If you use this software in your work, please cite our paper:

@article{JMLR:v26:24-2110,
  author  = {Nils Lehmann and Nina Maria Gottschling and Jakob Gawlikowski and Adam J. Stewart and Stefan Depeweg and Eric Nalisnick},
  title   = {Lightning UQ Box: Uncertainty Quantification for Neural Networks},
  journal = {Journal of Machine Learning Research},
  year    = {2025},
  volume  = {26},
  number  = {54},
  pages   = {1--7},
  url     = {http://jmlr.org/papers/v26/24-2110.html}
}