confidence_interval_estimator_ML

This repo contains code to perform estimation of Confidence Intervals both parametric and non-parametric (a.k.a. Monte Carlo Confidence Interval or Empirical Confidence Interval estimation) for Machine Learing models.

Colab Demo

You can easily try this tool in Google Colab by clicking on the following badge:

Quickstart

>>> import confidence_interval_estimator_ML.utils as cie
>>> # Parametric Estimation:
>>> cie.parametric_confidence_interval(confidence_level=.95, mean=model_accuracy, sample_size=sample_size, verbose=True)
Computing C.I. assuming the data follow a standard normal distribution
With 95.0% confidence the accuracy of the model is in 0.977 +/-0.023.
(0.9773132867658845, 0.02268671323411553)

>>> # Non-Parametric Estimation:
>>> test_accuracies = cie.get_accuracy_on_samples(get_classifier_instance, X, y, n_iterations=100, sample_ratio=1,
                             train_ratio=0.8, random_seed=None, sample_with_replacement=True, verbose=False) 

>>> cie.nonparametric_confidence_interval(accs, .9)
From the given data, with 90.0% confidence the accuracy of the model is in 0.959 +/-4.05.
(0.9595, 0.04050000000000009)

Further Details

You can find further details on the usage of the tool in the code documentation or in the examples provided in the Notebook.ipynb file. If you need help, please don't hesitate to open a github issue or getting in touch.