Flaktor

Flaky test detection made simple.

Flaktor is a framework-agnostic CLI tool that tracks your test results over time and identifies flaky tests - tests that sometimes pass and sometimes fail without code changes.

Features

• Framework Agnostic: Works with any test framework that outputs JUnit/xUnit XML
• Simple CLI: Easy-to-use commands for uploading results and viewing reports
• Flaky Detection: Automatically detects tests with inconsistent results
• CI/CD Ready: Built-in support for GitHub Actions, GitLab CI, and more
• Beautiful Output: Rich terminal output with tables and colored status
• Lightweight: SQLite database with no external dependencies

Installation

pip install flaktor

Quick Start

# Initialize the database
flaktor init

# Run your tests with JUnit XML output
pytest --junitxml=results.xml tests/

# Upload results
flaktor upload results.xml

# View flaky tests
flaktor list --flaky

# Generate a report
flaktor report

Commands

CI/CD Integration

Flaktor is designed for CI/CD pipelines. Track test results across runs to detect flaky tests.

GitHub Actions

- name: Upload to Flaktor
  run: |
    flaktor upload results.xml \
      --branch "${{ github.ref_name }}" \
      --commit "${{ github.sha }}"

GitLab CI

script:
  - flaktor upload results.xml --branch "$CI_COMMIT_REF_NAME" --commit "$CI_COMMIT_SHA"

Webhook Alerts

Get notified when a new flaky test shows up, right after uploading results:

export FLAKTOR_WEBHOOK_URL=https://hooks.slack.com/services/...
flaktor upload results.xml
flaktor notify

flaktor notify only alerts on tests that weren't already flagged flaky, so re-running it in CI won't spam the same alert every build. The payload's text field works as-is with Slack Incoming Webhooks.

See docs/ci-cd-integration.md for complete examples for:

• GitHub Actions
• GitLab CI/CD
• Jenkins
• CircleCI
• Azure DevOps

Configuration

Put defaults in a .flaktorrc file (TOML) so you don't repeat flags on every command. Flaktor looks in the current directory, then each parent directory, then your home directory. Use flaktor --config path/to/file or the FLAKTOR_CONFIG environment variable to point at a specific file.

# Global settings
db = ".flaktor/flaktor.db"        # relative paths are relative to this file
webhook = "https://hooks.slack.com/services/..."

# Per-command defaults: use the command's option names (`--min-runs` -> min_runs)
[trend]
days = 14

[notify]
min_runs = 10

Precedence, highest first: command-line flag, environment variable (FLAKTOR_DB, FLAKTOR_WEBHOOK_URL), .flaktorrc, built-in default. Unknown commands or options in the file are reported as errors rather than silently ignored. Webhook URLs are secrets, so prefer the environment variable over committing one to a shared .flaktorrc. Run flaktor config to see which file is active.

MCP Server (for AI coding agents)

Flaktor can expose its flaky-test data to AI coding agents (Claude Code, Cursor, etc.) over the Model Context Protocol, so an agent can check whether a failing test is a known flake before debugging it as a real bug. The server is read-only — upload, init, and clean stay CLI-only.

No VS Code extension is required. Flaktor ships as a standard Python MCP server that connects through your MCP client.

pip install "flaktor[mcp]"

Then either run it directly:

flaktor mcp

Or add it to your MCP client config:

Claude Code

claude mcp add flaktor -- flaktor mcp

Cursor

{
  "mcpServers": {
    "flaktor": {
      "command": "flaktor",
      "args": ["mcp"]
    }
  }
}

VS Code

{
  "servers": {
    "flaktor": {
      "type": "stdio",
      "command": "flaktor",
      "args": ["mcp"]
    }
  }
}

If your client expects a config file instead of a JSON snippet, the important part is the same: run flaktor mcp as a stdio MCP server.

Available tools: list_flaky_tests, check_test_flakiness, list_quarantined_tests, list_tags, list_tests_by_tag, list_trending_tests, list_duration_trends, get_test_history, get_test_summary, get_database_stats.

Understanding Flakiness

Flaktor calculates a "flip rate" for each test:

flip_rate = 2 * min(pass_rate, fail_rate)

• A test that always passes: flip_rate = 0%
• A test that always fails: flip_rate = 0%
• A test that passes 50% of the time: flip_rate = 100% (most flaky)
• A test that passes 80% of the time: flip_rate = 40%

Tests with a flip rate above 20% (default threshold) are considered flaky.

Example Output

$ flaktor list --flaky

                    Flaky Tests (last 30 days)
+----------------------------------------+------+------+------+-----+--------+
| Test Name                              | Flip | Pass | Runs | P/F | Avg    |
+----------------------------------------+------+------+------+-----+--------+
| test_api.TestAuth.test_token_refresh   | 80%  | 60%  | 25   | 15/10 | 0.45s |
| test_db.TestConn.test_reconnect        | 60%  | 70%  | 20   | 14/6  | 1.23s |
| test_ui.TestLogin.test_remember_me     | 40%  | 80%  | 15   | 12/3  | 2.10s |
+----------------------------------------+------+------+------+-----+--------+

Development

# Clone the repository
git clone https://github.com/PanjatanCoders/flaktor.git
cd flaktor

# Install in development mode
pip install -e .

# Run tests
pytest tests/ -v

License

MIT License - see LICENSE file for details.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.