ml-uncertainty 0.1.0a1

Uncertainty quantification and model inference for machine learning models

ML Uncertainty

Compute prediction intervals and parameter uncertainties for ML models in 4 lines of code

ML Uncertainty is a Python module for machine learning inference build on top of scikit-learn and autograd packages, and is distributed under the MIT license.

This package has been built by Archit Datar ([e-mail]).

Intended audience

This package is intended to benefit data scientists and ML enthusiasts.

Motivation

• Too often in machine learning, we fit complex models, but cannot quantity their precision via prediction intervals or feature significance.

• This is especially true of the scikit-learn environment which is extremely easy to use but does not offer these functionalities.

• However, in many use cases, especially where we have small and fat datasets, these are insights are critical to produce reliable models and insights.

• Enter ML Uncertainty! This provides an easy API to get all these insights from models.

• It takes scikit-learn fitted models as inputs and uses appropriate statistics to quantify the uncertainties in ML models.

Computing stats as easy as:

# Set up the model inference.
inf = ParametricModelInference()
inf.set_up_model_inference(X_train=X, y_train=y, estimator=regr)

# Get parameter importance estimates.
df_imp = inf.get_parameter_errors()

# Get prediction intervals.
df_int = inf.get_intervals(X)

Features

1. Model parameter significance testing: Tests whether the given model parameters are truly significant or not.

   For ensemble models, it can inform if given features are truly important or if they just seem so due to the instability of the model.

2. Prediction intervals: Can produce prediction and confidence intervals for parametric and non-parametric ML models.

3. Error propagation: Propagates error from input / model parameters to the outputs.

4. Non-Linear regression: Scikit-learn-style API to fit non-linear models.

Installation

Dependencies

Python versions: See badges above. Packages: See requirements.txt.

User installation

See ./docs/installation.md.

Examples

To run the examples, some additional plots need to be made which require matplotlib and seaborn packages. These can be installed using:

pip install matplotlib seaborn

Check out some of the these examples to try out the package. These examples are best run in VS code.

• Non-linear regression example with a quadratic model
• Non-linear regression example with an Arrhenius model
• Parametric model inference with Arrhenius model
• Error Propagation with Arrhenius model
• Model inference for a random forest regressor model

Theoretical foundations

Discussion about the theory used can be found here:

• Parametric models
• Ensemble models

Benchmarking

NonLinearRegression, ParametricModelInference, and ErrorPropagation classes have been benchmarked against the Python statsmodels package. The codes for this can be found here.

To run these benchmarking codes, please install statsmodels using:

pip install statsmodels==0.14.0

The EnsembleModelInference does not have a code to benchmark it against to the best of my knowledge. However, the code follows the ideas developed in the work by Zhang et al. (2020). The test is that a $(1-\alpha)\times100$ % prediction interval must contain $(1-\alpha)$ proportion of the training data. See benchmarking codes here.

Credits

1. This package was created with Cookiecutter_ and the audreyr/cookiecutter-pypackage_ project template.

   Cookiecutter

   audreyr/cookiecutter-pypackage

2. Some functions in ParametricModelInference are adopted from a Github repo by sriki18.