independent-validation 0.1.3

IV is a Python package designed to assess the accuracy of machine learning classifiers by generating a full probability distribution of their performance. This method goes beyond traditional cross-validation by using an iterative process and Markov Chain Monte Carlo (MCMC) to estimate uncertainty. By only using samples for training after they have been tested alpha inflation is prevented.

IV: Independent Validation

IV is a Python package designed to assess the accuracy of machine learning classifiers by generating a full probability distribution of their performance. This method goes beyond traditional cross-validation by using an iterative process and Markov Chain Monte Carlo (MCMC) to estimate uncertainty. By only using samples for training after they have been tested alpha inflation is prevented.

Installation

pip install independent-validation

Usage

An example of running the independent validation process with a classifier:

from sklearn.datasets import load_wine
import independent_validation as iv
from sklearn.svm import SVC

wine = load_wine()
X, y = wine.data, wine.target

iv_svm = iv.IV(X, y, SVC(gamma='scale'))  # initiating
iv_svm.run_iv()  # classify samples
iv_svm.compute_posterior()  # use mcmc to compute posterior
bacc_svm_dist = iv_svm.get_bacc_dist()  # get desired output
print("Mode (MAP) value:", bacc_svm_dist.map())

License

MIT License