StaMBO: Statistical model comparison with bootstrap

This package is aimed to be a one-stop-shop for statistical testing in machine learning when it comes to evaluating models on a test set and comparing whether our improved model is really beating the baseline. That is, we cover the following very typical use-case in machine learning:

Currently, we support the cases of classification, regresson, and semantic segmentation, including data with a block-diagonal (grouped/clustered) covariance structure, e.g. repeated measurements from the same subject. We do not yet support the significance of ranking. It is coming in future releases.

In practice

Install from PyPI:

pip install stambo

The use of the library is then straightforward:

import stambo
...
seed = 42
testing_result = stambo.compare_models(y_test, preds_1, preds_2, metrics=("ROCAUC", "AP", "QKappa", "BACC", "MCC"), seed=seed)
print(stambo.to_latex(testing_result))

The above will print a LaTeX table, which one can easily copy-paste. As an example, below is the rendered table, which was returned in notebooks/Classification.ipynb ():

Note: From version 0.1.5 we support block-diagonal structure of the data. That is, if you have data from the same patient in the test set, it can easily be adjusted for by specifying the groups argument.

The regression example can be found at notebooks/Regression.ipynb ( )

For more advanced explanation, see the documentation. By default, binary, multi-class, and multi-label classification, as well as regression are supported.

One can also use the library to perform a simple two-sample test. For example, to compare the means of two distributions:

import stambo
...
seed = 42
res = stambo.two_sample_test(sample_1, sample_2, statistics={"Mean": lambda x: x.mean()})

A more detailed and full example of the above is shown here: notebooks/Two_sample_test.ipynb ()

If you have more than two models (or samples) to compare, stambo.compare_models_pairwise (and its lower-level building block, stambo.pairwise_bootstrap_test) run the bootstrap test on every pair, with a Holm-Bonferroni correction for the multiple comparisons applied by default:

import stambo
...
seed = 42
results = stambo.compare_models_pairwise(y_test, (preds_1, preds_2, preds_3), ("ROCAUC", "AP"), seed=seed, n_bootstrap=1000)
print(stambo.pairwise_to_latex(results))

See notebooks/Pairwise_comparison.ipynb () for a full walkthrough, including why the correction matters and how it interacts with clustered/grouped data.

Built for AI coding agents

stambo is agent-ready: AGENTS.md is a concise, verified cheat sheet (which function to call, paired/non_paired/groups semantics, the two-tailed convention, return-format schema) that coding agents such as Claude Code or Codex pick up automatically as project context (CLAUDE.md is a pointer to it for Claude Code's own auto-load). stambo.to_dict(report) gives results as a JSON-serializable, named-field dict instead of a positional array, and the package ships a py.typed marker so type checkers and IDE/agent tooling trust its type hints.

Contributing

To setup a dev environment, you should use uv and install the project as follows:

uv venv
uv pip install -e ".[dev]"

Author

Dr. Aleksei Tiulpin, PhD