multiplicity 2.0.0

Evaluate predictive multiplicity

multiplicity

Library for evaluating predictive multiplicity of deep leearning models.

Setup

pip install multiplicity

Quickstart

The library provides a method to estimate viable prediction intervals: prediction intervals that are robust to a small change in model's loss at training or evaluation time.

Import the library:

from multiplicity import torch as multiplicity

Suppose we have a trained torch binary classifier which outputs softmax probabilities:

model(x)  # 0.75

Specify to the deviation of which metric we want to be robust to, and on which dataset:

robustness_criterion = multiplicity.ZeroOneLossCriterion(train_loader)

Then, we can compute the viable prediction range for a given example x like so:

lb, pred, ub = multiplicity.viable_prediction_range(
    model=model,
    target_example=x,
    robustness_criterion=robustness_criterion,
    criterion_thresholds=epsilon,
)
# lb=0.71, pred=0.75, ub=0.88