trialML package
Project description
trialML: Preparing a machine learning model for a statistical trial
trialML
is a python
package designed to help researchers and practioners prepare their machine learning models for a statistical trial to establish a lowerbound on model performance. Specifically, this package helps to calibrating the operating threshold of a binary classifier and carry out a power analysis. A more formal description of these techniques can be found in the corresponding arXiv paper.
Features
The main modules from trialML
can be called in with one line of code: from trialML import trial, power
. Their key methods are outlined below, and are described with more detail by the docstrings (e.g. help('trialML.trial.classification')
).
trial.classification(gamma, m, alpha)
: determine optimal threshold and calculate power of future trialstatistic(y, s, threshold, pval=..)
: return the performance measure for a given threshold (and possibly pvalue from null hypothesis)learn_threshold(y, s, method='..')
: calibrate the opearting threshold to obtain at leastgamma
1alpha
% of the time.calculate_power(spread, n_trial, threshold)
: estimate power for a given trial sample size and null hypothesis margin (spread). Threshold can be provided to estimate percent of samples that are classspecific.
power.twosided_classification(m1, m2, alpha)
: estimate performance measure and power range (confidence interval) for two performance measures:m1
andm2
.set_threshold(y, s, gamma1)
: Set the threshold to get a performance level ofgamma1
for the first performance measurem1
.statistic_CI(y, s, threshold)
: Get the (1alpha
) confidence interval for the empirical values ofm1
andm2
.statistic_pval(y, s, gamma0)
: Get the pvalue on trial data for a given null hypothesis.
How to use
The code block below shows how to calibrate a classifier for toy example of a classifier trained on random data. For more detailed examples wiht real data, please see the tutorials folder.
# Load modules
import numpy as np
from sklearn.linear_model import LogisticRegression
from trialML.trial import classification
## (1) Train a model and obtain scores on a test set
np.random.seed(1)
n, p = 150, 10
k1, k2 = 50, 100
X, y = np.random.randn(n, p), np.random.binomial(1, 0.5, n)
X_train, y_train, X_test, y_test = X[:k1], y[:k1], X[k1:k2], y[k1:k2]
mdl = LogisticRegression(penalty='none', solver='lbfgs')
mdl.fit(X=X_train, y=y_train)
# test set scores
s_test = mdl.predict_proba(X_test)[:,1]
s_test = np.log(s_test / (1s_test)) # logit transform
## (2) Calibrate operating threshold to achieve 50% sensitivity, 95% of the time
gamma = 0.5 # performance measure target
alpha = 0.05 # typeI error rate for threshold selection
m = 'sensitivity' # currently supports sensitivity/specificity/precision
# Set up statistical tool
calibration = classification(gamma=gamma, alpha=alpha, m=m)
# Learn threshold
calibration.learn_threshold(y=y_test, s=s_test, method='percentile', n_bs=1000, seed=1)
# Observe testset performance
gamma_hat_test = calibration.statistic(y=y_test, s=s_test, threshold=calibration.threshold_hat)
print('Empirical sensitivity on testset: %0.1f%%' % (100*gamma_hat_test))
## (3) Estimate power for trial data
X_trial, y_trial = X[k1:], y[k1:]
n_trial = len(X_trial)
gamma0 = 0.45
spread = gamma  gamma0
calibration.calculate_power(spread, n_trial, threshold=calibration.threshold_hat)
print('Expected trial power for a %0.1f%% margin is at least %0.1f%%' % (100*spread, 100*calibration.power_hat))
## (4) Run trial
s_trial = mdl.predict_proba(X_trial)[:,1]
s_trial = np.log(s_trial / (1s_trial)) # logit transform
gamma_trial, pval_trial = calibration.statistic(y=y_trial, s=s_trial, gamma0=gamma0, threshold=calibration.threshold_hat)
print('Trial sensitivity: %0.1f%%, trial nullhypothesis: %0.1f%%, trial pvalue: %0.5f' % (100*gamma_trial, 100*gamma0, pval_trial))
How to install
trialML
is available on PyPI can be installed in one line: pip install trialML
.
Project details
Release history Release notifications  RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for trialML0.1.0py2.py3noneany.whl
Algorithm  Hash digest  

SHA256  6590f6cb178cd1722bffb671c34ca31cace571516f346b67d3ee03f93c1aec86 

MD5  87dca0ec4b1e8d5fd0646a5647611ec0 

BLAKE2b256  4b4faad31c6e941f903bbea93e4428bf2ac9361e9cfca0e6b36c13dd19c0263a 