flakerate

Measure how flaky a test actually is, find out why, and quarantine it with an expiry date.

The usual response to a flaky test is to retry it until it passes. That turns the build green and throws away the only evidence there was. You never learn whether it fails one run in fifty or one in three, and you never learn why, so it stays in the suite getting slowly worse.

flakerate runs the suite repeatedly, reports a failure rate with a confidence interval, diagnoses the cause by experiment, and writes a quarantine file that expires.

Installation

pip install flakerate

Requires Python 3.10+ and pytest 8+.

Quick start

flakerate run -n 20 -- tests

20 runs, 412 tests, 3 flaky

failed  rate  95% range  cause  test
------  ----  ---------  -----  ---------------------------------------------
  7/20   35%    18%-57%         tests/test_orders.py::test_cancel_refunds
  6/20   30%    15%-52%         tests/test_api.py::test_rate_limit_headers
  1/20    5%     1%-24%         tests/test_sync.py::test_backfill_idempotent

Every run gets a different shuffle seed, because a suite that always runs in the same order will never show you an order-dependent failure, and that is the most common kind.

The range is a Wilson score interval and it is the number to make decisions against. Note the last row: one failure in twenty runs is anywhere from 1% to 24%. That is not a measurement yet, it is a signal to run it more times. Wilson rather than the textbook normal approximation because the counts are small and the rates sit near zero, which is exactly where the normal approximation returns a negative lower bound.

Diagnosing the cause

flakerate run -n 30 --classify -- tests

failed  rate  95% range  cause             test
------  ----  ---------  ----------------  ---------------------------------------------
  9/30   30%    17%-48%  order-dependent   tests/test_orders.py::test_cancel_refunds
  8/30   27%    14%-45%  environment       tests/test_api.py::test_rate_limit_headers

tests/test_orders.py::test_cancel_refunds
  in suite: 9/30 failed, alone: 0/5 failed, fixed order: 0/3 failed
  passes on its own, so something earlier in the suite leaves state behind. Look for module-level
  globals, a cached singleton, an unclosed database transaction, or a fixture with a wider scope
  than it needs.

Classification runs experiments rather than pattern-matching tracebacks.

Experiment	What it answers
Run the test alone, several times	Is the rest of the suite involved at all?
Run the whole suite, several times, in one fixed order	Is the order the variable, or something else?

Cause	What it means
order-dependent	Passes alone. Something that runs before it leaves state behind.
nondeterministic	Fails alone too. An unseeded random source, a real clock, dict ordering, a thread.
environment	Fails alone, and the failures look like they come from outside the process.
unclear	The test could not be run on its own, so no conclusion is offered.

Failure messages only ever refine a conclusion the experiments already reached. A traceback mentioning a socket is suggestive, not proof, and it is never the sole basis for a verdict.

The isolation experiment is per test and cheap. The fixed-order runs are shared across every flaky test in the report, so classifying ten of them costs the same suite runs as classifying one.

Quarantine

flakerate run -n 30 --classify --quarantine --days 14 -- tests

Writes flakerate.toml:

[quarantine."tests/test_orders.py::test_cancel_refunds"]
reason = "tests/test_orders.py:88: AssertionError"
rate = 0.3
cause = "order-dependent"
added = "2026-08-24"
expires = "2026-09-07"

Commit it, then in CI:

- run: pip install flakerate
- run: flakerate check
- run: FLAKERATE_QUARANTINE=flakerate.toml pytest

Quarantined tests still run and still report. They are marked non-strict xfail, so one that starts passing again shows up as an xpass rather than silently becoming load-bearing.

An entry with no expiry date, or one that cannot be read, counts as expired. Failing the other way would make expires = "soon" a quarantine nobody ever sees the end of, which is the thing the date exists to prevent.

flakerate check fails the build once an entry is past its date:

$ flakerate check
fail 2 quarantine entry/entries have expired

expired     cause             test
----------  ----------------  ---------------------------------------------
2026-09-07  order-dependent   tests/test_orders.py::test_cancel_refunds
2026-09-07  environment       tests/test_api.py::test_rate_limit_headers

Fix the test, or renew it on purpose with 'flakerate run --quarantine'.

That date is the point. Quarantine without one is deletion with extra steps, and every codebase that has one has tests in it nobody has looked at in three years.

Configuration

All keys live under [tool.flakerate] in pyproject.toml. All are optional.

Key	Type	Default	Meaning
runs	integer	20	How many times run repeats the suite
quarantine	string	"flakerate.toml"	Path to the quarantine file
pytest_args	list of strings	[]	Arguments used when none are given on the command line
max_flake_rate	number, 0 to 1	0.0	Reserved for a future rate-based gate

Values are type-checked rather than coerced, so max_flake_rate = true is an error rather than quietly becoming 1.0.

Command reference

Command	What it does
flakerate run -n N -- <pytest args>	Run the suite N times and report flake rates
flakerate run --classify	Also diagnose the cause of each flaky test
flakerate run --quarantine --days N	Write findings to the quarantine file
flakerate run --no-shuffle	Keep pytest's own order
flakerate check	Exit non-zero if a quarantine entry has expired
flakerate list	Show what is quarantined and how long it has left

Every command takes --json.

pytest options

Option	Environment variable	Effect
--flakerate-report PATH	FLAKERATE_REPORT	Write a JSON outcome report
--flakerate-seed N	FLAKERATE_SEED	Shuffle test order with a reproducible seed
--flakerate-quarantine PATH	FLAKERATE_QUARANTINE	Apply a quarantine file

How it compares

Tool	Measures the rate	Confidence interval	Diagnoses cause	Quarantine expiry	Maintained
pytest-rerunfailures	no	no	no	no	yes
flaky	no	no	no	no	last release 2024
pytest-flakefinder	reruns only	no	no	no	last release 2022
flakerate	yes	yes	yes	yes	yes

Notes

How many runs you need depends on what you are trying to catch. Twenty finds anything failing more than about 1 in 10. A 1-in-50 flake needs a few hundred, and flakerate run says so when the sample is too small to conclude anything.

A test that fails on every run is broken, not flaky, and is reported separately.

Failures in setup and teardown count as failures. A fixture that intermittently explodes is a flaky test to anyone waiting on the build, whatever pytest calls it internally.

A pytest run that exits 2 or higher (a collection error, a bad argument) is discarded rather than counted as everything failing.

--flakerate-seed reproduces an exact order, so a failure found here can be re-run on its own.

The plugin is a pytest entry point, so it is imported by every pytest process on the machine. It registers nothing and returns immediately from every hook unless given a report path or a quarantine file.

pytest-xdist is not supported yet: outcomes from several worker processes are not merged.

Contributing

Bug reports and pull requests are welcome. uv sync then uv run pytest to get started.

License

MIT.