triagekit-mcp-flaky-test-detective

Part of the TriageKit MCP server suite.

Tells flaky tests from genuinely broken ones using real CI run history — instead of an engineer guessing "probably flaky, re-run it" or a broken test getting rubber-stamped past review because nobody has time to check.

Why

CI red gets treated one of two bad ways: blindly re-run until green (masking a real bug), or dismissed as "that test is always flaky" (hiding a regression). This server pulls a test's actual pass/fail history from GitHub Actions, classifies the pattern deterministically, clusters failures by root cause, and checks for infra correlation — so the decision is based on evidence, not vibes.

Tools

Install

Not yet published. This package is not on PyPI or GHCR yet (alpha, pre-release) - the commands below are the intended install path once triagekit-mcp-core and this package are published (core must publish first; see CONTRIBUTING.md). Until then, install from a local build: uv build --package triagekit-mcp-core --out-dir dist && uv build --package triagekit-mcp-flaky-test-detective --out-dir dist && uv pip install dist/*.whl.

uvx triagekit-mcp-flaky-test-detective
# or
pip install triagekit-mcp-flaky-test-detective
# or
docker run --rm -i -e TRIAGEKIT_FLAKY_TEST_DETECTIVE_GITHUB_TOKEN ghcr.io/Karthick-dev-cart/triagekit-flaky-test-detective

Quick Start (Claude Desktop / Claude Code)

Add to your mcp.json:

{
  "mcpServers": {
    "flaky-test-detective": {
      "command": "uvx",
      "args": ["triagekit-mcp-flaky-test-detective"],
      "env": {
        "TRIAGEKIT_FLAKY_TEST_DETECTIVE_GITHUB_TOKEN": "ghp_..."
      }
    }
  }
}

Example prompts

• "Is tests/test_checkout.py::test_apply_discount actually flaky, or just broken? Check the last 30 runs on myorg/myrepo."
• "Give me the flakiest tests in myorg/myrepo over the last 20 CI runs."
• "Are test_upload_large_file's failures all the same root cause, or several?"
• "Does test_db_connection fail more often on a specific runner?"
• "Which tests in myorg/myrepo are safe to quarantine right now?"
• "Generate 20 synthetic user records matching this example: {\"id\": 1, \"email\": \"a@b.com\", \"active\": true}."
• "Did this OpenAPI spec change introduce any breaking changes vs. the previous version?"

Authentication

generate_synthetic_test_data and diff_api_contract need no credentials — they operate on data you pass directly. See .env.example for a local-dev template.

Credentials are only ever read from the environment at startup — no tool accepts a token as a parameter. See CONTRIBUTING.md for why.

Development

uv sync
uv run pytest packages/flaky-test-detective -v
uv run ruff check packages/flaky-test-detective
uv run pyright packages/flaky-test-detective

License

MIT