flakeguard

Your test suite is lying to you.

Flakeguard finds the tests that randomly pass and fail, ranks them by how unreliable they are, and shows you why — so you can trust your tests again.

Free · private · works with any language. No account, no sign-up, and nothing leaves your machine.

The problem, in 30 seconds

You have a test that fails once in a while. You re-run it. It passes. Was that a real bug? A flaky test? Nobody knows — so everyone ignores it, and that is exactly how real bugs hide.

A flaky test costs a team hours, and slowly destroys trust in the entire test suite. Flakeguard fixes the hard half of the problem: it tells you exactly which tests are unreliable, and how unreliable.

Try it in 60 seconds

pipx install flakeguard
flakeguard scan ./reports        # a folder of test reports

No reports yet? Run it against the examples bundled in this repo:

git clone https://github.com/PrathameshKulkarni2k4/flakeguard
cd flakeguard
flakeguard scan examples/reports

What it does

• Reads your test results from many runs (the more runs, the sharper the picture).
• Finds the tests that disagree with themselves across those runs.
• Gives each one a score from 0 (solid) to 1 (a coin-flip).
• Shows the latest failure message, so you can go fix it.
• Runs in your CI and can leave a comment on the pull request — or block it when flakiness spikes.

Works with any language

Flakeguard reads JUnit XML, the report format almost every test framework can produce. One tool, every language:

Flakeguard also reads TAP (the Test Anything Protocol), produced by Node's built-in test runner, the tap/tape libraries, Perl's Test::More and others.

Use in CI (GitHub Actions)

- name: Flakeguard
  uses: PrathameshKulkarni2k4/flakeguard@main
  with:
    reports: reports      # dir/glob of JUnit XML files
    min-runs: 3           # ignore tests seen in fewer runs
    max-flaky: 0          # fail if any flaky test is found (-1 = never fail)
    comment: "true"       # post the report as a PR comment

See .github/workflows/example.yml for a complete workflow that runs a suite several times and feeds the reports in.

To gate locally or in any other CI, use the exit code:

flakeguard scan ./reports --fail-on-flaky   # exits 1 if anything is flaky

How the score works

For each test we count passes vs. non-passes across all runs (skips are ignored). The flakiness score is:

score = 2 * min(passes, failures) / (passes + failures)

• always passes, or always fails → 0.0 (consistent, not flaky)
• flips on every run (50/50) → 1.0 (maximally flaky)

It is deliberately explainable — no black box. You can read the number and know exactly how it was derived.

Roadmap

• JUnit XML parsing, flakiness scoring, text/JSON/Markdown reports
• GitHub Action with PR comments and a flakiness budget
• TAP report format
• Native JSON report formats
• CI history so it learns across builds
• Failure-signature clustering (group the same flake across tests)
• Quarantine file so known flakes are set aside, not deleted

Contributing

Contributions are very welcome — see CONTRIBUTING.md. Adding support for a new report format is a great first contribution and is designed to be small. Please be kind; this project follows a Code of Conduct.

License

MIT — see LICENSE.