A library for the evaluation of model performance and estimation of the uncertainty on these metrics.
Documentation
Functionality
On a high level MMU provides two types of functionality:
Metrics - functions to compute confusion matrix(ces) and binary classification metrics over classifier scores or predictions.
Uncertainty estimators - functionality to compute the joint uncertainty over classification metrics.
We currently focus on binary classification models but aim to include support for other types of models and their metrics in the future.
Confusion Matrix & Metrics
Metrics consist mainly of high-performance functions to compute the confusion matrix and metrics over a single test set, multiple classification thresholds and or multiple runs.
The binary_metrics functions compute the 10 most commonly used metrics:
Negative precision aka Negative Predictive Value (NPV)
Positive recision aka Positive Predictive Value (PPV)
Negative recall aka True Negative Rate (TNR) aka Specificity
Positive recall aka True Positive Rate (TPR) aka Sensitivity
Negative f1 score
Positive f1 score
False Positive Rate (FPR)
False Negative Rate (FNR)
Accuracy
Mathew’s Correlation Coefficient (MCC)
Uncertainty estimators
MMU provides two methods for modelling the joint uncertainty on precision and recall: Multinomial uncertainty and Bivariate-Normal.
The Multinomial approach estimates the uncertainty by computing the profile log-likelihoods scores for a grid around the precision and recall. The scores are chi2 distributed with 2 degrees of freedom which can be used to determine the confidence interval.
The Bivariate-Normal approach models the statistical uncertainty over the linearly propagated errors of the confusion matrix and the analytical covariance matrix. The resulting joint uncertainty is elliptical in nature.
Installation
mmu can be installed from PyPi.
pip install mmuWe provide wheels for:
MacOS [x86, ARM]
Linux
Windows
Installing the package from source requires a C++ compiler with support for C++14. If you have a compiler available it is advised to install without the wheel as this enables architecture specific optimisations.
pip install mmu --no-binary mmuOther build options exist, see the Installation section of the docs.
Usage
import mmu
# Create some example data
scores, yhat, y = mmu.generate_data(n_samples=1000)
# Compute the joint uncertainty on precision-recall curve
pr_err = mmu.PrecisionRecallCurveUncertainty.from_scores(y, scores)
# Plot the uncertainty
pr_err.plot()See Basics section of the docs or the tutorial notebooks for more examples.
Contributing
We very much welcome contributions, please see the contributing section for details.
1 maintainer
Metadata
1 maintainer
Release files for mmu 0.1.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| mmu-0.1.2.tar.gz | 2.1 MB | Without provenance | Details for mmu-0.1.2.tar.gz |
Built distributions (wheels)
Total release size: 5.8 MB
Release files / mmu-0.1.2.tar.gz
| Download URL | mmu-0.1.2.tar.gz |
|---|---|
| Size | 2.1 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
e1b4ff24d98b8cbe8a4148a0855962d8d01c0929ecb152c29e3097a9cdb4c150
|
|
BLAKE2b-256 checksum How to use checksums |
5040d71f3773b3bd76db632e3f8693a101a8f4a4c0786a9240969ec73cdda6e3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.1
|
Release files / mmu-0.1.2-cp311-cp311-win_amd64.whl
| Download URL | mmu-0.1.2-cp311-cp311-win_amd64.whl |
|---|---|
| Size | 184.3 kB |
| Tags | CPython 3.11 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
7045e122b0bc78b8ee512f7be780539f20aea2ac9c7c22061bc6075a4ccc9db2
|
|
BLAKE2b-256 checksum How to use checksums |
0d2aa94c1c0b2dc6c443a207667f4d848e45e011c51badf4e3ee3d5360b8beaa
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.1
|
Release files / mmu-0.1.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | mmu-0.1.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 313.9 kB |
| Tags | CPython 3.11 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
6cd0167530d2c0990089728ceee88c635d33dbbcc68ad56feb1ee35e43ede077
|
|
BLAKE2b-256 checksum How to use checksums |
249a8bfd57aaf17cf89d2008a56e8826f3c759c6466611d3d05321302a924627
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.1
|
Release files / mmu-0.1.2-cp311-cp311-macosx_10_13_x86_64.whl
| Download URL | mmu-0.1.2-cp311-cp311-macosx_10_13_x86_64.whl |
|---|---|
| Size | 194.9 kB |
| Tags | CPython 3.11 macOS 10.13+ x86-64 |
|
SHA-256 checksum How to use checksums |
887afd31ed7dde1f610770104a3765d8e14d87e9e77e0bf586523eb0de90b2f0
|
|
BLAKE2b-256 checksum How to use checksums |
eb1af84cc3f152971b7a1548565f6a970df5347d7039d50728144e283dd0f9b3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.1
|
Release files / mmu-0.1.2-cp310-cp310-win_amd64.whl
| Download URL | mmu-0.1.2-cp310-cp310-win_amd64.whl |
|---|---|
| Size | 184.3 kB |
| Tags | CPython 3.10 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
ab95725d1370aea991d6587d0ca924f0631c72ceaf2ca9e7ffbe0ec4af7c3ff2
|
|
BLAKE2b-256 checksum How to use checksums |
a10ac567da305e8bfdfb7dac4271d14d08044e150dfba04230edde669b894989
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.1
|
Release files / mmu-0.1.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | mmu-0.1.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 314.0 kB |
| Tags | CPython 3.10 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
af3816287c2b664fa346568baf457f5815334fc9156a3183a8b1c94e7d440749
|
|
BLAKE2b-256 checksum How to use checksums |
d1f2087630df468c04869b38bbac4d3525e503ba02ad1d027141543ef9853ab6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.1
|
Release files / mmu-0.1.2-cp310-cp310-macosx_10_13_x86_64.whl
| Download URL | mmu-0.1.2-cp310-cp310-macosx_10_13_x86_64.whl |
|---|---|
| Size | 194.8 kB |
| Tags | CPython 3.10 macOS 10.13+ x86-64 |
|
SHA-256 checksum How to use checksums |
e14fd704892b576af6de52acff25d9ab5587a249f7092035a6b8c1f047ba64aa
|
|
BLAKE2b-256 checksum How to use checksums |
7e9522be570bbac6e20d18fd282d6bcd14550b71b47499fdd4eac0d041b5d366
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.1
|
Release files / mmu-0.1.2-cp39-cp39-win_amd64.whl
| Download URL | mmu-0.1.2-cp39-cp39-win_amd64.whl |
|---|---|
| Size | 184.3 kB |
| Tags | CPython 3.9 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
c8387d9e0e7bb41fa7f3aa4eba2b788b8cc49a25fb1ba5f117548f82f159afaa
|
|
BLAKE2b-256 checksum How to use checksums |
3e9cf0a7a875a0e525ab359dc9bc8f5e950077781df619b0331835f879bd3eb4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.1
|
Release files / mmu-0.1.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | mmu-0.1.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 314.1 kB |
| Tags | CPython 3.9 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
8ee5b99c9c0aa28d35d6a2cbacff19f1fae9fb9fe65d0cced03ee823b621c0aa
|
|
BLAKE2b-256 checksum How to use checksums |
12d9175c772be3053374761446855ece8740dbcfd719d24f0cc6b0891e9691bf
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.1
|
Release files / mmu-0.1.2-cp39-cp39-macosx_10_13_x86_64.whl
| Download URL | mmu-0.1.2-cp39-cp39-macosx_10_13_x86_64.whl |
|---|---|
| Size | 195.0 kB |
| Tags | CPython 3.9 macOS 10.13+ x86-64 |
|
SHA-256 checksum How to use checksums |
c04db2ad4b5b2bb3fdd7db90728753e9ffdf5b2cccec7021cfde23c7cb24d169
|
|
BLAKE2b-256 checksum How to use checksums |
e8b3d7fdfac93b9824b4507bf7339b0b647246cdef50df4a69c10f6460f203b6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.1
|
Release files / mmu-0.1.2-cp38-cp38-win_amd64.whl
| Download URL | mmu-0.1.2-cp38-cp38-win_amd64.whl |
|---|---|
| Size | 184.2 kB |
| Tags | CPython 3.8 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
ac176ddc1816c5f881692e65ed5fc5f46803a9fc5d2222283a027e3b0c36a9c8
|
|
BLAKE2b-256 checksum How to use checksums |
5dd9c1df5d2be4e34c82bf9ba58c1ef21f623b9bc8d70b415588ef22fc8ee795
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.1
|
Release files / mmu-0.1.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | mmu-0.1.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 313.4 kB |
| Tags | CPython 3.8 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
9d23b1efc092c7e35cb81f19626bd1bcbe9e9515ec830feef36ef9143d48358a
|
|
BLAKE2b-256 checksum How to use checksums |
59d104d52143e6cbf1955368cdd111bfc1392fcf00b2b87cf3f5e88adc90c4a8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.1
|
Release files / mmu-0.1.2-cp38-cp38-macosx_10_16_x86_64.whl
| Download URL | mmu-0.1.2-cp38-cp38-macosx_10_16_x86_64.whl |
|---|---|
| Size | 195.0 kB |
| Tags | CPython 3.8 macOS 10.16+ x86-64 |
|
SHA-256 checksum How to use checksums |
0ebbe9ee15f8f94c98847279d5a419b0c8323c65feed665b23948ba3a8043c92
|
|
BLAKE2b-256 checksum How to use checksums |
4c04c0dd04481dd51fc000d2a8f19d615f95d554211d5927153284c6e6d89944
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.1
|
Release files / mmu-0.1.2-cp38-cp38-macosx_10_13_x86_64.whl
| Download URL | mmu-0.1.2-cp38-cp38-macosx_10_13_x86_64.whl |
|---|---|
| Size | 194.9 kB |
| Tags | CPython 3.8 macOS 10.13+ x86-64 |
|
SHA-256 checksum How to use checksums |
d29e425ff1ca2a04f53cf8f08d9edab0a2056efab9f85ede5f8e9bc469c57473
|
|
BLAKE2b-256 checksum How to use checksums |
d166e61af7f43a2367d631889dbd6b27a5d73b9feb7ccbd3d2c7f71038d0e92c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.1
|
Release files / mmu-0.1.2-cp37-cp37m-win_amd64.whl
| Download URL | mmu-0.1.2-cp37-cp37m-win_amd64.whl |
|---|---|
| Size | 183.0 kB |
| Tags | CPython 3.7 CPython 3.7 pymalloc Windows x86-64 |
|
SHA-256 checksum How to use checksums |
0056d12b775869d762a11ed389a7429a6c82e6f6d37cb2306a200a2e12d7ff8b
|
|
BLAKE2b-256 checksum How to use checksums |
acfa4fc93af190210be2d1a2f092922f960fcfb90d4c3cd35337d5cf1929be15
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.1
|
Release files / mmu-0.1.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | mmu-0.1.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 320.0 kB |
| Tags | CPython 3.7 CPython 3.7 pymalloc Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
19c974db3b6fd02c85209e18351c06bec6eb989720e1cb15149c23ffcec7b74b
|
|
BLAKE2b-256 checksum How to use checksums |
9d5a1e61035db7c2846f5f25eeedcd023314245edb0c8bcc3e813127a12c69b7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.1
|
Release files / mmu-0.1.2-cp37-cp37m-macosx_10_13_x86_64.whl
| Download URL | mmu-0.1.2-cp37-cp37m-macosx_10_13_x86_64.whl |
|---|---|
| Size | 191.1 kB |
| Tags | CPython 3.7 CPython 3.7 pymalloc macOS 10.13+ x86-64 |
|
SHA-256 checksum How to use checksums |
735127c6a7e63c34760904ae4f969475651fd5cda40d9321eac32b353234d7de
|
|
BLAKE2b-256 checksum How to use checksums |
132d9e88491046f892e25e40bdfc0ee2eb31f6af7a85ef5f2c441efced18276c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.1
|
Security for mmu 0.1.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
