Skip to main content

Build Status

Calibration Belt

Assessment of the calibration belt and goodness of fit of binomial models.

Installation

pip install calibration-belt

import pandas as pd
from pathlib import Path
from calibration import CalibrationBelt

import matplotlib.pyplot as plt
%matplotlib inline  

Loading data

df = pd.read_csv(Path('tests/data/example_data.csv'))

# Separate P (outcome) and E (probability outputed by the model)
# In this case we will evaluate two different models
P = df.target.to_numpy()
predictors = {
    'SVM': df.SVM.to_numpy(),
    'NN': df.NN.to_numpy()
}

CalibrationBelt object

# We create CalibrationBelt objects for 
# each model we want to evaluate
belts = {}
for key, E in predictors.items():
    belts[key] = CalibrationBelt(P, E)

Plotting the belt

# Calculate belt in the confidence intervals .8 and .95
for model, belt in belts.items():
    fig, ax  = belt.plot(confidences=[.8, .95])
    ax.set_title(model, fontsize=30)

png

png

Calculating T-statistic and p_value

for model, belt in belts.items():
    T, p_value = belt.test()
    print(f"Model: {model:3}, T-statistic: {T:08.5f} , p-value: {p_value:07.5f}")
Model: SVM, T-statistic: 08.37500 , p-value: 0.01518
Model: NN , T-statistic: 31.05418 , p-value: 0.00001

Calculating boundaries of the belt

boundaries = belt.calculate_boundaries(.95)
lower, upper = boundaries[0, 1:]
print(f"Lower bound: {lower:.4f}, Upper bound: {upper:.4f}")
Lower bound: 0.0000, Upper bound: 0.0493

Cumulative distribution function

The plot below shows the computed distribution of the T-statistic in polynomials of degree m.

T = [i / 10 for i in range(0, 301)]
viridis = plt.cm.get_cmap("viridis", 4)
fig, ax = plt.subplots(1, figsize=[10, 7])
for m in [1, 2, 3, 4]:
    cdf = []
    for t in T:
        cdf.append(CalibrationBelt.calculate_cdf(t, m, .8))  
    ax.plot(T, cdf, color=viridis(m-1))

png

References

Based on Nattino, Giovanni, Stefano Finazzi, and Guido Bertolini. "A new calibration test and a reappraisal of the calibration belt for the assessment of prediction models based on dichotomous outcomes." Statistics in medicine 33.14 (2014): 2390-2407.

Metadata

Release files for calibration-belt 0.1.41

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for calibration-belt 0.1.41
File Size Uploaded Provenance Actions
calibration-belt-0.1.41.tar.gz 6.0 kB Without provenance Details for calibration-belt-0.1.41.tar.gz

Release files / calibration-belt-0.1.41.tar.gz

Download URL calibration-belt-0.1.41.tar.gz
Size 6.0 kB
Tags Source
SHA-256 checksum
How to use checksums
ae0615dd4e6d11aca58c04d221ee9891bbfc4edd65ad70249b546ffbc8e36304
BLAKE2b-256 checksum
How to use checksums
0c911a9afccbead21b5d1cbbd1ee6c41372a7690900156a3af57a8419d25b523
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.6.0.post20200814 requests-toolbelt/0.9.1 tqdm/4.48.2 CPython/3.8.5

Security for calibration-belt 0.1.41

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page