Skip to main content

Leakly logo

PyPI Build License

Open in Google Colab

Leakly: Leakage checks for any machine-learning pipeline

Leakly uses label permutation to test whether a machine-learning pipeline performs above chance when no true signal is present.

Above-chance performance after permutation may indicate leakage from preprocessing, feature selection, tuning, or another step of the pipeline.

How it works

  1. Permute labels to remove the real feature-label association.
  2. Run the full pipeline exactly as in the original analysis.
  3. Compare the permuted score distribution with chance level.
  4. Above-chance permuted performance suggests possible leakage.

Leakly includes example configurations for a leaky pipeline and a non-leaky pipeline so users can inspect the effect directly.

Example permutation AUC summary

Install

pip install Leakly

Quick Start on Colab: Open example.ipynb in Colab

Key Python snippet

from leakly import (
    MLPipeline,
    SummaryPlotter,
    load_example_leakage_config,
    permute_label)

scores = []
for seed in range(100):
    permuted_y = permute_label(y, random_state=seed)
    score = (
        # Replace with any user-defined pipeline
        MLPipeline(
            X,
            permuted_y,
            covariates=covariates,
            config=load_example_leakage_config(),
        ).fit()
    ).evaluate()
    scores.append(score)

SummaryPlotter(scores, chance_level=0.5).plot()

FAQ

Can Leakly check my own pipeline?

Yes. Leakly can evaluate any pipeline that takes X, y, optional covariates, and returns a test score. The key is to run the full pipeline exactly as in the real analysis, including preprocessing, feature selection, tuning, and evaluation.

Why can a leaky pipeline score well on permuted labels?

If leakage occurs, information from test samples can enter the analysis before the train/test split or outside the cross-validation loop. Common sources include feature selection, scaling, imputation, covariate adjustment, dimensionality reduction, or hyperparameter tuning performed on all samples.

In high-dimensional data such as omics and neuroimaging, random features can appear predictive by chance. If a pipeline can retain these spurious patterns, it may perform above chance even after labels are permuted.

How many permutations should I run?

Use 100 for a quick check. Use 1,000 or more for publication-level evidence.

License

MIT. See LICENSE.

Metadata

Release files for Leakly 0.1.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for Leakly 0.1.2
File Size Uploaded Provenance Actions
leakly-0.1.2.tar.gz 257.3 kB With provenance Details for leakly-0.1.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for Leakly 0.1.2
File Interpreter ABI Platform Actions
leakly-0.1.2-py3-none-any.whl Python 3 none any Details for leakly-0.1.2-py3-none-any.whl

Total release size: 286.4 kB

Release files / leakly-0.1.2.tar.gz

Download URL leakly-0.1.2.tar.gz
Size 257.3 kB
Tags Source
SHA-256 checksum
How to use checksums
70cc5f0eed490ad11ab512567122e9a3e69dc6d25e95674a3a2c9c9f400b4f1b
BLAKE2b-256 checksum
How to use checksums
519abf276b2a259daccef8266f04f286abbd062b955906ce0b4f9afbffb00d1d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.12

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on May 14, 2026.

Transparency log

Release files / leakly-0.1.2-py3-none-any.whl

Download URL leakly-0.1.2-py3-none-any.whl
Size 29.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
3c9d341ca1c33460d2d030f0d816f903473aa2220fef3f283f83eba65325c66b
BLAKE2b-256 checksum
How to use checksums
2928e4515ab6eb6ff18e9cb7fffeacf11491c9ea0091b4fa2c07d3a65bea0228
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.12

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on May 14, 2026.

Transparency log

Security for Leakly 0.1.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for leakly-0.1.2.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on May 14, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page