Leakly: Leakage checks for any machine-learning pipeline
Leakly uses label permutation to test whether a machine-learning pipeline performs above chance when no true signal is present.
Above-chance performance after permutation may indicate leakage from preprocessing, feature selection, tuning, or another step of the pipeline.
How it works
- Permute labels to remove the real feature-label association.
- Run the full pipeline exactly as in the original analysis.
- Compare the permuted score distribution with chance level.
- Above-chance permuted performance suggests possible leakage.
Leakly includes example configurations for a leaky pipeline and a non-leaky pipeline so users can inspect the effect directly.
Install
pip install Leakly
Quick Start on Colab: 
Key Python snippet
from leakly import (
MLPipeline,
SummaryPlotter,
load_example_leakage_config,
permute_label)
scores = []
for seed in range(100):
permuted_y = permute_label(y, random_state=seed)
score = (
# Replace with any user-defined pipeline
MLPipeline(
X,
permuted_y,
covariates=covariates,
config=load_example_leakage_config(),
).fit()
).evaluate()
scores.append(score)
SummaryPlotter(scores, chance_level=0.5).plot()
FAQ
Can Leakly check my own pipeline?
Yes. Leakly can evaluate any pipeline that takes X, y,
optional covariates, and returns a test score.
The key is to run the full pipeline exactly as in the real analysis,
including preprocessing, feature selection, tuning, and evaluation.
Why can a leaky pipeline score well on permuted labels?
If leakage occurs, information from test samples can enter the analysis before the train/test split or outside the cross-validation loop. Common sources include feature selection, scaling, imputation, covariate adjustment, dimensionality reduction, or hyperparameter tuning performed on all samples.
In high-dimensional data such as omics and neuroimaging, random features can appear predictive by chance. If a pipeline can retain these spurious patterns, it may perform above chance even after labels are permuted.
How many permutations should I run?
Use 100 for a quick check. Use 1,000 or more for publication-level evidence.
License
MIT. See LICENSE.
Metadata
Release files for Leakly 0.1.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| leakly-0.1.2.tar.gz | 257.3 kB | With provenance | Details for leakly-0.1.2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| leakly-0.1.2-py3-none-any.whl | Python 3 | none | any | Details for leakly-0.1.2-py3-none-any.whl |
Total release size: 286.4 kB
Release files / leakly-0.1.2.tar.gz
| Download URL | leakly-0.1.2.tar.gz |
|---|---|
| Size | 257.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
70cc5f0eed490ad11ab512567122e9a3e69dc6d25e95674a3a2c9c9f400b4f1b
|
|
BLAKE2b-256 checksum How to use checksums |
519abf276b2a259daccef8266f04f286abbd062b955906ce0b4f9afbffb00d1d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on May 14, 2026.
Transparency logRelease files / leakly-0.1.2-py3-none-any.whl
| Download URL | leakly-0.1.2-py3-none-any.whl |
|---|---|
| Size | 29.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
3c9d341ca1c33460d2d030f0d816f903473aa2220fef3f283f83eba65325c66b
|
|
BLAKE2b-256 checksum How to use checksums |
2928e4515ab6eb6ff18e9cb7fffeacf11491c9ea0091b4fa2c07d3a65bea0228
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on May 14, 2026.
Transparency logSecurity for Leakly 0.1.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for leakly-0.1.2.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on May 14, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.