triagekit-mcp-flaky-test-detective 0.1.0

MCP server that tells flaky tests from genuinely broken ones using real CI run history, and generates synthetic test data / API contract diffs for QA automation.

triagekit-mcp-flaky-test-detective

Part of the TriageKit MCP server suite.

Tells flaky tests from genuinely broken ones using real CI run history — instead of an engineer guessing "probably flaky, re-run it" or a broken test getting rubber-stamped past review because nobody has time to check.

Why

CI red gets treated one of two bad ways: blindly re-run until green (masking a real bug), or dismissed as "that test is always flaky" (hiding a regression). This server pulls a test's actual pass/fail history from GitHub Actions, classifies the pattern deterministically, clusters failures by root cause, and checks for infra correlation — so the decision is based on evidence, not vibes.

Tools

Tool	What it does
classify_test_failure	Classifies a test's recent history as consistently-failing / consistently-passing / intermittent (flaky) / insufficient-data, with run-by-run evidence
get_flaky_test_leaderboard	Ranks all tests in a repo's recent CI history by computed flake rate
diff_failure_stack_traces	Clusters a test's failures by stack-trace similarity — is "flaky" really 2-3 distinct root causes?
correlate_flake_with_infra	Checks whether flake rate correlates with the CI runner — infra contention vs. genuine non-determinism
suggest_quarantine_candidates	Returns genuinely intermittent tests above a flake-rate threshold
file_flaky_test_ticket	Opens a Jira ticket pre-filled with flake evidence (optional — requires Jira config)
generate_synthetic_test_data	Generates synthetic fixtures from a JSON-Schema fragment or example payload
diff_api_contract	Diffs two OpenAPI specs and classifies each change as breaking or non-breaking

Install

Not yet published. This package is not on PyPI or GHCR yet (alpha, pre-release) - the commands below are the intended install path once triagekit-mcp-core and this package are published (core must publish first; see CONTRIBUTING.md). Until then, install from a local build: uv build --package triagekit-mcp-core --out-dir dist && uv build --package triagekit-mcp-flaky-test-detective --out-dir dist && uv pip install dist/*.whl.

uvx triagekit-mcp-flaky-test-detective
# or
pip install triagekit-mcp-flaky-test-detective
# or
docker run --rm -i -e TRIAGEKIT_FLAKY_TEST_DETECTIVE_GITHUB_TOKEN ghcr.io/Karthick-dev-cart/triagekit-flaky-test-detective

Quick Start (Claude Desktop / Claude Code)

Add to your mcp.json:

{
  "mcpServers": {
    "flaky-test-detective": {
      "command": "uvx",
      "args": ["triagekit-mcp-flaky-test-detective"],
      "env": {
        "TRIAGEKIT_FLAKY_TEST_DETECTIVE_GITHUB_TOKEN": "ghp_..."
      }
    }
  }
}

Example prompts

• "Is tests/test_checkout.py::test_apply_discount actually flaky, or just broken? Check the last 30 runs on myorg/myrepo."
• "Give me the flakiest tests in myorg/myrepo over the last 20 CI runs."
• "Are test_upload_large_file's failures all the same root cause, or several?"
• "Does test_db_connection fail more often on a specific runner?"
• "Which tests in myorg/myrepo are safe to quarantine right now?"
• "Generate 20 synthetic user records matching this example: {\"id\": 1, \"email\": \"[e-mail]\", \"active\": true}."
• "Did this OpenAPI spec change introduce any breaking changes vs. the previous version?"

Authentication

Variable	Required for	Notes
TRIAGEKIT_FLAKY_TEST_DETECTIVE_GITHUB_TOKEN (or native GITHUB_TOKEN)	All 5 CI-analysis tools	A GitHub PAT (classic or fine-grained) with actions:read on the target repo
TRIAGEKIT_FLAKY_TEST_DETECTIVE_JIRA_BASE_URL	file_flaky_test_ticket	e.g. https://yourteam.atlassian.net
TRIAGEKIT_FLAKY_TEST_DETECTIVE_JIRA_EMAIL	file_flaky_test_ticket	Jira Cloud account email
TRIAGEKIT_FLAKY_TEST_DETECTIVE_JIRA_API_TOKEN (or native JIRA_API_TOKEN)	file_flaky_test_ticket	Create one here

generate_synthetic_test_data and diff_api_contract need no credentials — they operate on data you pass directly. See .env.example for a local-dev template.

Credentials are only ever read from the environment at startup — no tool accepts a token as a parameter. See CONTRIBUTING.md for why.

Development

uv sync
uv run pytest packages/flaky-test-detective -v
uv run ruff check packages/flaky-test-detective
uv run pyright packages/flaky-test-detective

License

MIT