Skip to main content

Compute uncertainties on classification model metrics.

Project description

A library for the evaluation of model performance and estimation of the uncertainty on these metrics.

Uncertainty on the precision-recall curve
MacOS build Linux build Windows build Documentation License PyPi

Documentation

Functionality

On a high level MMU provides two types of functionality:

  • Metrics - functions to compute confusion matrix(ces) and binary classification metrics over classifier scores or predictions.

  • Uncertainty estimators - functionality to compute the joint uncertainty over classification metrics.

We currently focus on binary classification models but aim to include support for other types of models and their metrics in the future.

Confusion Matrix & Metrics

Metrics consist mainly of high-performance functions to compute the confusion matrix and metrics over a single test set, multiple classification thresholds and or multiple runs.

The binary_metrics functions compute the 10 most commonly used metrics:

  • Negative precision aka Negative Predictive Value (NPV)

  • Positive recision aka Positive Predictive Value (PPV)

  • Negative recall aka True Negative Rate (TNR) aka Specificity

  • Positive recall aka True Positive Rate (TPR) aka Sensitivity

  • Negative f1 score

  • Positive f1 score

  • False Positive Rate (FPR)

  • False Negative Rate (FNR)

  • Accuracy

  • Mathew’s Correlation Coefficient (MCC)

Uncertainty estimators

MMU provides two methods for modelling the joint uncertainty on precision and recall: Multinomial uncertainty and Bivariate-Normal.

The Multinomial approach estimates the uncertainty by computing the profile log-likelihoods scores for a grid around the precision and recall. The scores are chi2 distributed with 2 degrees of freedom which can be used to determine the confidence interval.

The Bivariate-Normal approach models the statistical uncertainty over the linearly propagated errors of the confusion matrix and the analytical covariance matrix. The resulting joint uncertainty is elliptical in nature.

Installation

mmu can be installed from PyPi.

pip install mmu

We provide wheels for:

  • MacOS [x86, ARM]

  • Linux

  • Windows

Installing the package from source requires a C++ compiler with support for C++14. If you have a compiler available it is advised to install without the wheel as this enables architecture specific optimisations.

pip install mmu --no-binary mmu

Other build options exist, see the Installation section of the docs.

Usage

import mmu

# Create some example data
scores, yhat, y = mmu.generate_data(n_samples=1000)

# Compute the joint uncertainty on precision-recall curve
pr_err = mmu.PrecisionRecallCurveUncertainty.from_scores(y, scores)

# Plot the uncertainty
pr_err.plot()

See Basics section of the docs or the tutorial notebooks for more examples.

Contributing

We very much welcome contributions, please see the contributing section for details.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mmu-0.1.2.tar.gz (2.1 MB view details)

Uploaded Source

Built Distributions

mmu-0.1.2-cp311-cp311-win_amd64.whl (184.3 kB view details)

Uploaded CPython 3.11 Windows x86-64

mmu-0.1.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (313.9 kB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

mmu-0.1.2-cp311-cp311-macosx_10_13_x86_64.whl (194.9 kB view details)

Uploaded CPython 3.11 macOS 10.13+ x86-64

mmu-0.1.2-cp310-cp310-win_amd64.whl (184.3 kB view details)

Uploaded CPython 3.10 Windows x86-64

mmu-0.1.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (314.0 kB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

mmu-0.1.2-cp310-cp310-macosx_10_13_x86_64.whl (194.8 kB view details)

Uploaded CPython 3.10 macOS 10.13+ x86-64

mmu-0.1.2-cp39-cp39-win_amd64.whl (184.3 kB view details)

Uploaded CPython 3.9 Windows x86-64

mmu-0.1.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (314.1 kB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

mmu-0.1.2-cp39-cp39-macosx_10_13_x86_64.whl (195.0 kB view details)

Uploaded CPython 3.9 macOS 10.13+ x86-64

mmu-0.1.2-cp38-cp38-win_amd64.whl (184.2 kB view details)

Uploaded CPython 3.8 Windows x86-64

mmu-0.1.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (313.4 kB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

mmu-0.1.2-cp38-cp38-macosx_10_16_x86_64.whl (195.0 kB view details)

Uploaded CPython 3.8 macOS 10.16+ x86-64

mmu-0.1.2-cp38-cp38-macosx_10_13_x86_64.whl (194.9 kB view details)

Uploaded CPython 3.8 macOS 10.13+ x86-64

mmu-0.1.2-cp37-cp37m-win_amd64.whl (183.0 kB view details)

Uploaded CPython 3.7m Windows x86-64

mmu-0.1.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (320.0 kB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

mmu-0.1.2-cp37-cp37m-macosx_10_13_x86_64.whl (191.1 kB view details)

Uploaded CPython 3.7m macOS 10.13+ x86-64

File details

Details for the file mmu-0.1.2.tar.gz.

File metadata

  • Download URL: mmu-0.1.2.tar.gz
  • Upload date:
  • Size: 2.1 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.1

File hashes

Hashes for mmu-0.1.2.tar.gz
Algorithm Hash digest
SHA256 e1b4ff24d98b8cbe8a4148a0855962d8d01c0929ecb152c29e3097a9cdb4c150
MD5 3ab513f297c966165ef43fcb8b5715b3
BLAKE2b-256 5040d71f3773b3bd76db632e3f8693a101a8f4a4c0786a9240969ec73cdda6e3

See more details on using hashes here.

Provenance

File details

Details for the file mmu-0.1.2-cp311-cp311-win_amd64.whl.

File metadata

  • Download URL: mmu-0.1.2-cp311-cp311-win_amd64.whl
  • Upload date:
  • Size: 184.3 kB
  • Tags: CPython 3.11, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.1

File hashes

Hashes for mmu-0.1.2-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 7045e122b0bc78b8ee512f7be780539f20aea2ac9c7c22061bc6075a4ccc9db2
MD5 9b5764d47a4a02fe396432714d65434a
BLAKE2b-256 0d2aa94c1c0b2dc6c443a207667f4d848e45e011c51badf4e3ee3d5360b8beaa

See more details on using hashes here.

Provenance

File details

Details for the file mmu-0.1.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for mmu-0.1.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 6cd0167530d2c0990089728ceee88c635d33dbbcc68ad56feb1ee35e43ede077
MD5 f448307cba5e3e5983daa6c21ae5606f
BLAKE2b-256 249a8bfd57aaf17cf89d2008a56e8826f3c759c6466611d3d05321302a924627

See more details on using hashes here.

Provenance

File details

Details for the file mmu-0.1.2-cp311-cp311-macosx_10_13_x86_64.whl.

File metadata

File hashes

Hashes for mmu-0.1.2-cp311-cp311-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 887afd31ed7dde1f610770104a3765d8e14d87e9e77e0bf586523eb0de90b2f0
MD5 24fbdab9ceb87db7b906a0548d6c80f4
BLAKE2b-256 eb1af84cc3f152971b7a1548565f6a970df5347d7039d50728144e283dd0f9b3

See more details on using hashes here.

Provenance

File details

Details for the file mmu-0.1.2-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: mmu-0.1.2-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 184.3 kB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.1

File hashes

Hashes for mmu-0.1.2-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 ab95725d1370aea991d6587d0ca924f0631c72ceaf2ca9e7ffbe0ec4af7c3ff2
MD5 4b90a7c6100c3fb5982a7895e2e9d903
BLAKE2b-256 a10ac567da305e8bfdfb7dac4271d14d08044e150dfba04230edde669b894989

See more details on using hashes here.

Provenance

File details

Details for the file mmu-0.1.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for mmu-0.1.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 af3816287c2b664fa346568baf457f5815334fc9156a3183a8b1c94e7d440749
MD5 46f92cec6e048cb1aaff1fccdb71b3cd
BLAKE2b-256 d1f2087630df468c04869b38bbac4d3525e503ba02ad1d027141543ef9853ab6

See more details on using hashes here.

Provenance

File details

Details for the file mmu-0.1.2-cp310-cp310-macosx_10_13_x86_64.whl.

File metadata

File hashes

Hashes for mmu-0.1.2-cp310-cp310-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 e14fd704892b576af6de52acff25d9ab5587a249f7092035a6b8c1f047ba64aa
MD5 9a1e7dfbe826374809a94aa59e9cadc9
BLAKE2b-256 7e9522be570bbac6e20d18fd282d6bcd14550b71b47499fdd4eac0d041b5d366

See more details on using hashes here.

Provenance

File details

Details for the file mmu-0.1.2-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: mmu-0.1.2-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 184.3 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.1

File hashes

Hashes for mmu-0.1.2-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 c8387d9e0e7bb41fa7f3aa4eba2b788b8cc49a25fb1ba5f117548f82f159afaa
MD5 7b37cbc0fe6e8ee0894389408e5d30ae
BLAKE2b-256 3e9cf0a7a875a0e525ab359dc9bc8f5e950077781df619b0331835f879bd3eb4

See more details on using hashes here.

Provenance

File details

Details for the file mmu-0.1.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for mmu-0.1.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 8ee5b99c9c0aa28d35d6a2cbacff19f1fae9fb9fe65d0cced03ee823b621c0aa
MD5 76410216a2c8a9929cc307b1b5ec3e1b
BLAKE2b-256 12d9175c772be3053374761446855ece8740dbcfd719d24f0cc6b0891e9691bf

See more details on using hashes here.

Provenance

File details

Details for the file mmu-0.1.2-cp39-cp39-macosx_10_13_x86_64.whl.

File metadata

File hashes

Hashes for mmu-0.1.2-cp39-cp39-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 c04db2ad4b5b2bb3fdd7db90728753e9ffdf5b2cccec7021cfde23c7cb24d169
MD5 a0770df9dbb2487d2aed9071afc4002a
BLAKE2b-256 e8b3d7fdfac93b9824b4507bf7339b0b647246cdef50df4a69c10f6460f203b6

See more details on using hashes here.

Provenance

File details

Details for the file mmu-0.1.2-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: mmu-0.1.2-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 184.2 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.1

File hashes

Hashes for mmu-0.1.2-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 ac176ddc1816c5f881692e65ed5fc5f46803a9fc5d2222283a027e3b0c36a9c8
MD5 51916ef5c4de6e0c00f238b6cf32fb51
BLAKE2b-256 5dd9c1df5d2be4e34c82bf9ba58c1ef21f623b9bc8d70b415588ef22fc8ee795

See more details on using hashes here.

Provenance

File details

Details for the file mmu-0.1.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for mmu-0.1.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 9d23b1efc092c7e35cb81f19626bd1bcbe9e9515ec830feef36ef9143d48358a
MD5 5ec6a073b7d0661895bbbc7edde2abbb
BLAKE2b-256 59d104d52143e6cbf1955368cdd111bfc1392fcf00b2b87cf3f5e88adc90c4a8

See more details on using hashes here.

Provenance

File details

Details for the file mmu-0.1.2-cp38-cp38-macosx_10_16_x86_64.whl.

File metadata

File hashes

Hashes for mmu-0.1.2-cp38-cp38-macosx_10_16_x86_64.whl
Algorithm Hash digest
SHA256 0ebbe9ee15f8f94c98847279d5a419b0c8323c65feed665b23948ba3a8043c92
MD5 403de3e19e0a20dca63d1129737f0b37
BLAKE2b-256 4c04c0dd04481dd51fc000d2a8f19d615f95d554211d5927153284c6e6d89944

See more details on using hashes here.

Provenance

File details

Details for the file mmu-0.1.2-cp38-cp38-macosx_10_13_x86_64.whl.

File metadata

File hashes

Hashes for mmu-0.1.2-cp38-cp38-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 d29e425ff1ca2a04f53cf8f08d9edab0a2056efab9f85ede5f8e9bc469c57473
MD5 f000382b52fe482f2fede6c7ca2b8270
BLAKE2b-256 d166e61af7f43a2367d631889dbd6b27a5d73b9feb7ccbd3d2c7f71038d0e92c

See more details on using hashes here.

Provenance

File details

Details for the file mmu-0.1.2-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: mmu-0.1.2-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 183.0 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.1

File hashes

Hashes for mmu-0.1.2-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 0056d12b775869d762a11ed389a7429a6c82e6f6d37cb2306a200a2e12d7ff8b
MD5 203c2f1e397bd89ec5b50caa67332427
BLAKE2b-256 acfa4fc93af190210be2d1a2f092922f960fcfb90d4c3cd35337d5cf1929be15

See more details on using hashes here.

Provenance

File details

Details for the file mmu-0.1.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for mmu-0.1.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 19c974db3b6fd02c85209e18351c06bec6eb989720e1cb15149c23ffcec7b74b
MD5 ad00eba837add6b6ee2dc16d1f5677e5
BLAKE2b-256 9d5a1e61035db7c2846f5f25eeedcd023314245edb0c8bcc3e813127a12c69b7

See more details on using hashes here.

Provenance

File details

Details for the file mmu-0.1.2-cp37-cp37m-macosx_10_13_x86_64.whl.

File metadata

File hashes

Hashes for mmu-0.1.2-cp37-cp37m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 735127c6a7e63c34760904ae4f969475651fd5cda40d9321eac32b353234d7de
MD5 3c651adc6b5951dc0b1ff42ce6eb732c
BLAKE2b-256 132d9e88491046f892e25e40bdfc0ee2eb31f6af7a85ef5f2c441efced18276c

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page