Calibration Belt
Assessment of the calibration belt and goodness of fit of binomial models.
Installation
pip install calibration-belt
import pandas as pd
from pathlib import Path
from calibration import CalibrationBelt
import matplotlib.pyplot as plt
%matplotlib inline
Loading data
df = pd.read_csv(Path('tests/data/example_data.csv'))
# Separate P (outcome) and E (probability outputed by the model)
# In this case we will evaluate two different models
P = df.target.to_numpy()
predictors = {
'SVM': df.SVM.to_numpy(),
'NN': df.NN.to_numpy()
}
CalibrationBelt object
# We create CalibrationBelt objects for
# each model we want to evaluate
belts = {}
for key, E in predictors.items():
belts[key] = CalibrationBelt(P, E)
Plotting the belt
# Calculate belt in the confidence intervals .8 and .95
for model, belt in belts.items():
fig, ax = belt.plot(confidences=[.8, .95])
ax.set_title(model, fontsize=30)
Calculating T-statistic and p_value
for model, belt in belts.items():
T, p_value = belt.test()
print(f"Model: {model:3}, T-statistic: {T:08.5f} , p-value: {p_value:07.5f}")
Model: SVM, T-statistic: 08.37500 , p-value: 0.01518
Model: NN , T-statistic: 31.05418 , p-value: 0.00001
Calculating boundaries of the belt
boundaries = belt.calculate_boundaries(.95)
lower, upper = boundaries[0, 1:]
print(f"Lower bound: {lower:.4f}, Upper bound: {upper:.4f}")
Lower bound: 0.0000, Upper bound: 0.0493
Cumulative distribution function
The plot below shows the computed distribution of the T-statistic in polynomials of degree m.
T = [i / 10 for i in range(0, 301)]
viridis = plt.cm.get_cmap("viridis", 4)
fig, ax = plt.subplots(1, figsize=[10, 7])
for m in [1, 2, 3, 4]:
cdf = []
for t in T:
cdf.append(CalibrationBelt.calculate_cdf(t, m, .8))
ax.plot(T, cdf, color=viridis(m-1))
References
Based on Nattino, Giovanni, Stefano Finazzi, and Guido Bertolini. "A new calibration test and a reappraisal of the calibration belt for the assessment of prediction models based on dichotomous outcomes." Statistics in medicine 33.14 (2014): 2390-2407.
Metadata
Release files for calibration-belt 0.1.41
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| calibration-belt-0.1.41.tar.gz | 6.0 kB | Without provenance | Details for calibration-belt-0.1.41.tar.gz |
Release files / calibration-belt-0.1.41.tar.gz
| Download URL | calibration-belt-0.1.41.tar.gz |
|---|---|
| Size | 6.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ae0615dd4e6d11aca58c04d221ee9891bbfc4edd65ad70249b546ffbc8e36304
|
|
BLAKE2b-256 checksum How to use checksums |
0c911a9afccbead21b5d1cbbd1ee6c41372a7690900156a3af57a8419d25b523
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.6.0.post20200814 requests-toolbelt/0.9.1 tqdm/4.48.2 CPython/3.8.5
|
Security for calibration-belt 0.1.41
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.