sharp-cv

Statistical tests for comparing predictive performance with re-designed cross-validation, built on scikit-learn.

Predictive performance estimates from cross-validation folds are not independent, so a paired t-test assuming independent estimates cannot control the false positive rate. sharp-cv implements the SHARP (Split-Half Analysis of Repeated Performance) test, which on every repetition splits the data into two disjoint halves and runs a cross-validation inside each half. Because the halves share no observations, the two results of one repetition are independent, while results from different repetitions remain correlated. That structure lets the variance of a split-half result and the across-repetition correlation both be estimated from the data instead of assumed.

Install

pip install sharp-cv

or the development version:

pip install git+https://github.com/TianCZeng/sharp-cv.git

Requires Python 3.9 or later, NumPy, SciPy, scikit-learn and joblib.

Quick start

Regression, repeated 5-fold with 30 repetitions:

from sklearn.datasets import make_regression
from sklearn.linear_model import Ridge
from sklearn.model_selection import RepeatedKFold
from sharp_cv import sharp_cross_val_test

X, y = make_regression(n_samples=200, noise=1.0, random_state=0)
result = sharp_cross_val_test(
    Ridge(alpha=0.1), Ridge(alpha=10.0),
    X, y,
    cv=RepeatedKFold(n_splits=5, n_repeats=30),
    random_state=0,
)
print(result.statistic, result.pvalue)   # z and two-sided p
print(result.mean_diff)                  # mean score difference, model 1 minus model 2
print(result.ci_low, result.ci_high)     # 95% interval for that difference
print(result.score_1_AB.mean(), result.score_2_AB.mean())  # each model's mean score

The tested difference is always model 1 minus model 2, so a negative statistic means the second estimator scored higher. result.diff_AB is the [30, 2] array of paired differences the test was run on.

Classification is stratified automatically when estimator_1 is a classifier. Repetitions are independent of each other, so n_jobs runs them in parallel as in cross_val_score, and verbose prints progress:

from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import RepeatedStratifiedKFold

X, y = make_classification(n_samples=400, n_classes=3, n_informative=5,
                           random_state=0)
result = sharp_cross_val_test(
    LogisticRegression(max_iter=1000), RandomForestClassifier(n_estimators=50),
    X, y,
    cv=RepeatedStratifiedKFold(n_splits=5, n_repeats=30),
    scoring="accuracy",
    n_jobs=-1,
    random_state=0,
)

Monte-Carlo cross-validation, 150 repetitions with a 20% test set inside each half (n_splits counts repetitions here, not inner folds):

from sklearn.model_selection import ShuffleSplit

result = sharp_cross_val_test(
    Ridge(alpha=0.1), Ridge(alpha=10.0), X, y,
    cv=ShuffleSplit(n_splits=150, test_size=0.2),
    random_state=0,
)

cv has no default: the number of repetitions is a choice, so it has to be passed. A single-run splitter — an int, KFold or StratifiedKFold — has no repetition to work with and is rejected.

The procedure

One repetition of the split-half procedure is:

1. Divide the data at random into two disjoint halves A and B (stratified by class for classifiers).
2. Inside each half, run the inner cross-validation. Both estimators are fit on the same training folds and scored on the same test folds, which gives score_1 and score_2.
3. Reduce each half to one mean score per estimator and one mean difference, the mean over score_1 - score_2.

Repeating this J times gives a [J, 2] array of paired differences. Values from the same repetition come from disjoint data and are uncorrelated; values from different repetitions share data and are correlated. SHARP estimates that correlation from the two halves and uses it to compute the variance of the overall mean difference.

Throughout the package, A and B always mean the two halves of the data, never the two models. The two models are 1 and 2, the tested difference is model 1 minus model 2, and the AB in diff_AB, score_1_AB and score_2_AB marks their columns as half A and half B.

The cv argument sets both the inner cross-validation and the number of repetitions:

cv	Inside each half, per repetition	Rows of diff_AB
RepeatedKFold(n_splits=K, n_repeats=J)	K-fold, folds averaged	J repetitions
RepeatedStratifiedKFold(n_splits=K, n_repeats=J)	stratified K-fold, folds averaged	J repetitions
ShuffleSplit(n_splits=J, test_size=t)	one train/test split	J repetitions
StratifiedShuffleSplit(n_splits=J, test_size=t)	one stratified train/test split	J repetitions
KFold(K), StratifiedKFold(K) or an int K	single run, no repetition	rejected

The halves are redrawn on every repetition. Running repeated cross-validation inside two fixed halves would give a different correlation structure and is not what SHARP assumes.

How many repetitions?

A reasonable starting point is around 30 repetitions for repeated K-fold and around 150 for Monte-Carlo. The stratified variants take the same numbers. Note which argument carries the repetition count: n_repeats for the repeated splitters, n_splits for the ShuffleSplit ones.

These are suggestions rather than requirements, but fewer repetitions may decrease statistical power. Each repetition fits both estimators 2K times (or twice for Monte-Carlo schemes), so the cost is 2 * J * K fits per estimator, and repetition is the main cost of the procedure. Pass n_jobs to spread the repetitions over several processes.

Estimator modes

mode selects how the noise variance and the correlation are estimated. We recommend the default, st.

mode	Description
st	Score test: variance estimated under the null of zero difference. Default.
lrt	Likelihood ratio test
ml	Maximum likelihood, Wald z-test
rml	Restricted maximum likelihood, Wald z-test
mm	Method of moments, Wald z-test
mmc	Method of moments with the correlation clipped to [0, rho_clip]

The four likelihood-based modes fit the correlation on [0, rho_max], so they never return a negative correlation. rho_max is 0.499 for SHARP, just inside the point at which its covariance stops being positive definite (it is singular at 0.5). mmc clips to rho_clip, one further step inside that bound (0.497). Only mm leaves the correlation unconstrained.

The likelihood is computed in closed form from the three eigenvalues of the correlation matrix, so no 2J x 2J matrix is built: the data are reduced to three sums once, after which the fit does not grow with J. This reformulation is due to nipype/pydra-ml#72. What remains is a function of the correlation alone, and its best value needs no search: for rml it is a formula, and for the other likelihood modes it is the best of at most five candidates, the two ends of the allowed range and the roots of a cubic.

fall_back_rho is used by mm and mmc only, and is ignored by every other mode including the default: when the estimated variance of the mean drops below the independent-samples minimum, the correlation is replaced by this value. sharp_cross_val_test picks 1 / (2K) for K-fold inner schemes and test_size / 2 for Monte-Carlo schemes. Both are the fraction of the full dataset held out by one inner test set (the halving converts a fraction of a half into a fraction of the whole), the heuristic used in the paper's simulations. sharp_test requires it for mm and mmc and accepts None otherwise. A value you pass must be at least 0 and below 0.5, the range in which the SHARP correlation model is valid. Both functions reject a value outside it, even in modes that ignore it, and sharp_cross_val_test checks it and confidence_level before fitting any model.

Using your own paired differences

If you already ran the procedure yourself, pass the [n, 2] array to sharp_test, with column 0 from half A and column 1 from half B. Each row must be one repetition of the split-half procedure, with the halves redrawn every time.

from sharp_cv import sharp_test

z, p = sharp_test(diff_AB)                 # score test, the default

# mm and mmc need fall_back_rho: 1 / (2K) for K-fold inside each half,
# test_size / 2 for a single Monte-Carlo split inside each half
z, p = sharp_test(diff_AB, fall_back_rho=1 / (2 * K), mode="mm")

sharp_test returns (nan, nan) when fewer than two rows are given or the two columns are identical.

Confidence intervals

sharp_confint puts an interval around the difference, in the units of the score. sharp_cross_val_test reports the same interval as ci_low / ci_high.

from sharp_cv import sharp_confint

lo, hi = sharp_confint(diff_AB, level=0.95)

The interval is the set of differences the test does not reject, so it agrees with the p-value by construction: it excludes 0 exactly when p < 1 - level. For st and lrt the variance is re-estimated under every hypothesised difference, which is what makes the two agree; the Wald modes have a standard error that does not depend on the hypothesis, so their interval is the usual mean_diff +/- z * se.

Pass confidence_level=None to sharp_cross_val_test to skip it.

The smallest p-value the test can return

With few repetitions, the p-value cannot go below a certain value, however large the difference between the two models. The default test (st) estimates the noise as if the true difference were zero, so a large difference also raises the estimated noise.

Repetitions J	Smallest p, difference < 10 SD	Smallest p, any difference
5	0.014	0.0016
10	9.1e-4	7.7e-6
20	4.6e-6	2.5e-10
30	2.6e-8	9.5e-15

SD is the standard deviation of diff_AB. Only a mean difference larger than about 10 SD can get past the first column. The two columns are 2 * Phi(-sqrt(J + 1)) and 2 * Phi(-sqrt(2J)), where Phi is the standard normal CDF.

In practice: to be able to get p below 0.05, 0.01 or 0.001, use at least 3, 6 or 10 repetitions. The recommended 30 repetitions are far from these limits.

The confidence interval has the same limit. If level asks for more confidence than the data can give, for example 99.9% with 5 repetitions, one or both ends are infinite. If it asks for slightly less, the ends are finite but can be far from mean_diff.

Result object

sharp_cross_val_test returns a SharpTestResult named tuple:

Field	Contents
statistic	z statistic; positive when model 1 scored higher, negative when model 2 did
pvalue	two-sided p-value
mean_diff	mean of diff_AB, the estimated difference in the units of the score
test	'sharp'
mode	the estimator mode that was used
fall_back_rho	the fallback correlation in effect; read by mm and mmc only
diff_AB	[J, 2] paired differences, one row per repetition, half A and half B
score_1_AB, score_2_AB	[J, 2] mean score of each model in half A and half B; diff_AB is their difference
confidence_level	coverage of the interval below, or NaN if confidence_level=None was passed
ci_low, ci_high	confidence interval for mean_diff, in the units of the score

Printing the result shows the scalar fields and the shapes of the arrays.

Notes

• With scoring=None both estimators are scored with their own score method, so they must measure the same thing (for example both R² or both accuracy). A classifier and a regressor are rejected in that case. Pass scoring= to be explicit.
• random_state seeds one random stream that draws the half-splits and, for repeated and Monte-Carlo schemes, the inner splits too. Pass it to sharp_cross_val_test, or set it on the RepeatedKFold, RepeatedStratifiedKFold, ShuffleSplit or StratifiedShuffleSplit you pass as cv; either alone makes the result reproducible. If both are set, the one given to sharp_cross_val_test wins and a warning is raised. Only n_splits, n_repeats, test_size, train_size and random_state are read from those splitters; their split method is never called.
• n_jobs runs repetitions in parallel through joblib, as cross_val_score does, and verbose prints progress. The splits and fits do not depend on n_jobs, so results agree up to floating-point rounding.
• There is no groups argument currently. The half-split is not group-aware, so samples from one subject or site can land in both halves, and group-aware splitters such as GroupKFold are rejected.
• X and y may be NumPy arrays, pandas objects or SciPy sparse matrices. Rows are selected the way train_test_split selects them, keeping the container, so pipelines that select pandas columns by name work.
• Every inner test set must hold at least 2 samples, since scores such as R² are undefined on one. With K-fold inside each half this means at least 4K samples in all. sharp_cross_val_test checks this, and whether the inner splitter can run inside each half at all, before fitting anything. If a score still comes out NaN or infinite, for example ROC AUC on a test set with one class, it warns, and the statistic, p-value and interval are NaN.
• Estimators are cloned before every fit; pass unfitted estimators or pipelines.
• For hyperparameter tuning, pass a GridSearchCV (or a pipeline that contains one) as the estimator. It is refit inside every training fold, which gives the nested cross-validation the paper describes.
• Cross-validation runs inside each half rather than on the full dataset, so relative model performance may differ from full-dataset estimates, particularly when the algorithms or biomarkers being compared scale differently with training size.

The single-split SHA test

The paper also describes SHA (Split-HAlf), a cheaper variant that splits the data once and keeps the individual folds. It is not included in this release: a single split gives only two half results however many folds are used, which leaves the test too little to estimate its own uncertainty from, and in practice it almost never rejects. This is why single-run splitters are rejected by sharp_cross_val_test.

Citation

If you use this package, please cite:

Zeng, T., Li, H., Zhang, S., Tan, Y. Q., Tian, F., Orban, C., ... Nichols, T. E., & Yeo, B. T. T. (2026). Spurious model comparisons are widespread in biomedical artificial intelligence. bioRxiv. https://doi.org/10.64898/2026.05.17.724301

The link resolves to the most recent version of the preprint.

License

MIT. See LICENSE.