flake-exorcist

Find flaky tests. Name the cause. Reproduce on demand.

The problem

Your CI turns red on unchanged code. You re-run the build and it passes. Nobody learns why, nobody files a fix, and next week it happens again.

Google's engineering research found roughly 1 in 7 tests in large repositories eventually behaves this way. Most tools today detect flakiness by re-running failed tests. That tells you which test is flaky but not why. So you're left guessing.

What this does

flake-exorcist runs your pytest suite under controlled perturbations, one variable at a time, and watches which change flips a passing test to failing. The variable that flips the outcome is the root cause.

It needs no CI history, no paid account, and no setup beyond pip install.

$ exorcist hunt tests/ --fast

Exorcist Report | Runs: 41 | Seed: 20260808 | Flaky: 2 | Env-dependent: 1 | Failing: 1

Flaky Tests
  MEDIUM  tests/test_order_dependent.py::test_counter_is_zero   cause: order   confidence: 57%
    Polluter: tests/test_polluter.py::test_polluter_increments
    Fix: Isolate shared state per test; reset module-level globals in an autouse fixture.
  MEDIUM  tests/test_hash_seed.py::test_unseeded_rng_assumption  cause: seed   confidence: 43%
    Fix: Seed the RNG in a fixture; sort collections before asserting.

Non-portable (not flaky)
  tests/test_timezone.py::test_utc_date_format

Genuinely failing (not flaky)
  tests/test_real_failure.py::test_always_fails

Confidence reflects how often the flipping dimension actually reproduced the failure, so probabilistic flakes (order, parallel) report moderate confidence rather than a misleading 100%.

Exit code 1 means flaky tests were found. Wire it into CI as a gate.

Install

Requires Python 3.11 or newer.

pip install flake-exorcist

Or install from source for development:

git clone https://github.com/suletetes/flake-exorcist.git
cd flake-exorcist
pip install -e ".[dev]"

Quick start

# Run against the bundled sample repo (ships with the package)
exorcist hunt examples/sample_flaky_repo/tests --fast

# Run against your own project
exorcist hunt path/to/your/tests --fast

The four dimensions

The tool perturbs one thing at a time and observes what happens:

Dimension	What changes	What it catches
ORDER	Execution order (seeded shuffle)	Tests leaking shared state to later tests
ENVIRONMENT	Timezone, locale	Tests assuming the local clock or locale
PARALLEL	Number of workers (pytest-xdist)	Race conditions, port conflicts, file locks
SEED	PYTHONHASHSEED and random seed	Tests depending on set/dict ordering or unseeded RNG

How the diagnosis works

1. Baseline - run the suite once in default order, record every outcome.
2. Broad scan - run once per dimension, changing only that variable. Tests whose outcome flipped become candidates.
3. Deep dive - repeat the flipping dimension multiple times per candidate to confirm and score confidence.
4. Bisection (for ORDER) - binary search the preceding tests to name the exact polluter that contaminates the victim.
5. Report - ranked findings with cause, confidence, a reproduction command, and a one-line fix hint.

The classifier and attributor are pure functions: same inputs always produce the same output. They contain no I/O, no clock reads, no subprocess calls. This is what makes the tool itself testable with property-based methods (see below).

Verdicts

Verdict	Meaning
deterministic_pass	Always passes. Stable.
deterministic_fail	Always fails. A real bug, not a flake.
flaky	Outcome changes under a specific dimension. The tool names the cause.
environment_dependent	Fails under a specific TZ/locale but does so consistently. Non-portable, not flaky.
inconclusive	Something flipped but evidence is too weak to attribute. Reported honestly.

Commands

exorcist hunt <path>

Flag	What it does
--fast	Default. One pass per dimension in the broad scan, 3 repeats in deep dive.
--thorough	7 repeats per candidate, higher confidence, slower.
--json	Machine-readable JSON to stdout. Schema documented in design.md.
--only <node_id>	Skip the broad scan, deep-dive one specific test directly.
--max-runs N	Hard budget on total suite runs. Partial results on exhaustion.
--dimensions ORDER,SEED	Test only a subset of dimensions (comma-separated, case-insensitive).
--seed N	Base seed for reproducibility. Same seed = same run.
--workers N	Worker count for the parallel dimension.

exorcist repro <victim> --after <polluter>

Runs exactly two tests in the given order and reports whether the victim fails. Turns a "random" failure into a deterministic, scriptable one.

Exit codes

Code	Meaning
0	No flaky tests found
1	Flaky tests detected
2	Internal tool error
3	Bad arguments, missing pytest, no tests found

environment_dependent and deterministic_fail alone do not cause exit 1. Only actual flakiness does.

Configuration

Optional. Put this in your project's pyproject.toml:

[tool.exorcist]
mode = "fast"
dimensions = ["order", "environment", "parallel", "seed"]
base_seed = 42
max_runs = 50
workers = 4
timeout_s = 60

CLI flags override config values. Config values override built-in defaults.

How it was built (Kiro usage)

This project was built with Kiro's spec-driven workflow from start to finish. The .kiro/ directory at the root of this repo contains the full trail:

.kiro/
  steering/           <- persistent project context applied to every task
    product.md        <- vision, users, non-negotiable principles
    tech.md           <- stack decisions, pytest-driving mechanics
    structure.md      <- repo layout, layering rules, adapter seam
    flakiness-taxonomy.md <- the domain model the classifier encodes
  specs/flake-exorcist/
    requirements.md   <- 67 functional requirements in EARS notation
    design.md         <- architecture, algorithms (pseudocode), JSON schema, 7 ADRs
    tasks.md          <- 27 ordered TDD tasks with requirement traceability
  hooks/
    self-diagnose.json  <- runs exorcist on its own suite after test tasks

The spec-to-code chain:

1. Described the problem in plain language. Kiro generated EARS-format requirements, a design with data models and algorithms, and an ordered task list.
2. The classifier and attributor are pure functions (no I/O). Kiro's property-based tests verify they are deterministic: same run records in, same verdict out, regardless of input order or timestamp noise.
3. A self-diagnose hook runs exorcist hunt tests/ on this project's own suite after test-related tasks. If the tool introduces flakiness into itself, it catches it.

The one sentence version: I used Kiro's spec workflow to design the perturbation engine, and its property-based tests to prove the flakiness classifier is itself not flaky.

Testing

# Full suite
pytest tests/

# Only unit tests (fast, no subprocess)
pytest tests/unit/

# Only property tests (Hypothesis)
pytest tests/property/

# Integration (runs the real CLI against the sample repo)
pytest tests/integration/

# Lint and types
ruff check src/ tests/
mypy --strict src/exorcist

The test pyramid:

• Unit (tests/unit/) tests the pure classifier, attributor, config, models, and command-builder in isolation. No subprocesses.
• Property (tests/property/) uses Hypothesis to assert invariants over randomized inputs: classification is deterministic, confidence is monotonic, permuting input order changes nothing.
• Integration (tests/integration/) runs the full CLI against the bundled sample repo and asserts correct verdicts, causes, exit codes, and JSON schema.

Prior art and how this differs

This tool stands on the shoulders of prior work. The approach (perturb hidden variables to expose flakiness) is not new. What is new is packaging it as a single local command that crosses all four dimensions on a cold repo without requiring CI history or a paid account.

Tool	What it does	How this differs
NonDex (2016)	Perturbs JVM non-deterministic APIs	Python-native; adds multi-dimension triage
detect-test-pollution	Bisects order to find a polluter	Automates discovery (you don't need to know which test is flaky); adds dimensions beyond order
pytest-randomly	Shuffles test order	We credit the technique; our order-plugin owns order so it can be decoupled from RNG state
Shaker (2020)	Environment noise for concurrency flakiness	Combines env with order, parallel, and seed; classifies rather than just detecting
Datadog / Trunk / Mergify	SaaS flakiness dashboards	Local-only, no account needed, works on a repo you cloned ten seconds ago

Sample flaky repo

The examples/sample_flaky_repo/ directory ships a self-contained pytest suite with one planted flake per dimension plus stable and failing controls:

Test file	What it demonstrates
test_stable.py	Always passes (control)
test_real_failure.py	Always fails (control, not flaky)
test_order_dependent.py	Fails when run after test_polluter.py (shared state)
test_timezone.py	Fails under TZ=Pacific/Auckland (non-portable, not flaky)
test_hash_seed.py	Fails under some RNG/hash seeds (unseeded randomness)
test_concurrency.py	Fails under parallel workers (unlocked shared-file race)

Run it with plain pytest to see the flakiness manifest, or with exorcist to see it diagnosed.

Roadmap (out of scope for v1)

• Auto-fixing (suggest or apply patches for common patterns)
• Web dashboard for tracking trends over time
• IDE integration (VS Code, PyCharm inline annotations)
• Adapters for other frameworks (Jest, go test, JUnit)

License

MIT. See LICENSE.