Python package to measure the calibration of probabilistic classifiers
Project description
classifier-calibration
Python package to measure the calibration of probabilistic classifers. Calculates the classwise expected calibration error which returns the weighted average deviation from the expected rate of occurrence for predictions across all classes in multi-class classification problems.
Example Usage
import sklearn.linear_model
import sklearn.ensemble
import pandas as pd
import numpy as np
from classifier-calibration import classwise_ece
Binary classification
X, Y = sklearn.datasets.make_classification(n_samples=100000,n_informative=6,n_classes=2)
clf = sklearn.linear_model.LogisticRegression().fit(X,Y)
predicted_probabilities = clf.predict_proba(X)
calibration_loss = classwise_ece(pred_probs=predicted_probabilities,labels=Y)
print(calibration_loss)
Multi-class classification
We implement and compare calibration and accuracy of random forest and logistic regression classifiers for the 3-class classification problem
# Make classification data
num_classes = 3
X, Y = sklearn.datasets.make_classification(n_samples=100000,n_informative=6,n_classes=num_classes)
lr = sklearn.linear_model.LogisticRegression().fit(X,Y)
rf = sklearn.ensemble.RandomForestClassifier().fit(X,Y)
lr_predicted_probabilities = lr.predict_proba(X)
rf_predicted_probabilities = rf.predict_proba(X)
lr_calibration_loss, lr_bin_weights = classwise_ece(pred_probs=lr_predicted_probabilities,labels=Y,return_weights=True)
rf_calibration_loss, rf_bin_weights = classwise_ece(pred_probs=rf_predicted_probabilities,labels=Y,return_weights=True)
print(round(100*lr_calibration_loss,2),'%',round(100*rf_calibration_loss,2),'%')
# print distribution of weights across bins for each class' set of predictions
for k in range(num_classes):
print('Logistic regression bin weights for class',k,'predictions:')
print(lr_bin_weights[k])
print('Random forest bin weights for class',k,'predictions:')
print(rf_bin_weights[k])
print()
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for classifier_calibration-0.1.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5c29e0d4fa9bccd6f6ab9eb7ee6ce3b9e18c4f3332d7ba7165ef44cb6a57ee73 |
|
MD5 | 01feff5d5b15eec187df76b0d89b1fa9 |
|
BLAKE2b-256 | 85ab2e4d0058a1ce9310fbd15ec57b41efb8b05dd7c99f427a1f9c04c8a3e6dd |
Close
Hashes for classifier_calibration-0.1.1-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6d25433cc5fe1dd47432fc5876fc3f2af193fb2e6dd3770aef1610e4f90e0006 |
|
MD5 | 0b94f6c729bc5bd23ac10b90d2154921 |
|
BLAKE2b-256 | 25f2a880e52639da0d419404fc4511b5421a84a3699adc9e8de01355de73507b |