---

Why ai-test-failure-analyzer

Manual test failure investigation takes 30–60 minutes: open the test output, grep through logs, dig through git history, check recent deploys, ask Slack. And you can still point at the wrong thing — especially when the test file itself has an "intentional failure" comment or a fixture designed to trigger the analyzer.

This tool does it automatically in seconds:

• Parses the test result file to extract failing tests with HTTP details
• Scans git history for high-risk commits (endpoint renames, migrations, auth changes)
• Scans application logs for ERROR/FATAL lines
• Reads config files (.env, docker-compose)
• Cross-correlates all evidence into clusters
• Forms ranked, evidence-cited hypotheses with file:line precision
• Never points to test fixtures or "intentional failure" comments as root causes

How it's different

	ai-test-failure-analyzer	Manual triage	Generic LLM
Evidence source	Real git/logs/config	Human memory	Training data
Fixture noise	Blocked by Tier-1 gate	No protection	No protection
file:line precision	✅	Sometimes	No
Works without source code	✅ API-only mode	✅	✅
Repeatable	✅	❌	❌
CI-integrated	✅	❌	❌

Supported frameworks

24 frameworks across all major ecosystems:

Framework	Format	Command
Playwright	JSON reporter	playwright test --reporter=json
Jest	JSON	jest --json --outputFile=results.json
Vitest	JSON	vitest run --reporter=json
Cypress	Mochawesome JSON	cypress run --reporter mochawesome
WebdriverIO	JSON	wdio run wdio.conf.js
Mocha	JSON	mocha --reporter json
Detox	JSON	detox test --reporter json
pytest	JUnit XML	pytest --junit-xml=results.xml
Newman (Postman)	JSON	newman run col.json --reporters json --reporter-json-export results.json
k6	Summary JSON	k6 run --summary-export=results.json script.js
Go test	JSON	go test ./... -json > results.json
RSpec	JSON	rspec --format json --out results.json
PHPUnit	XML	phpunit --log-junit results.xml
NUnit	XML	standard NUnit output
xUnit	XML	standard xUnit output
MSTest	TRX XML	standard MSTest output
Robot Framework	XML	robot --output output.xml
Artillery	JSON	artillery run --output results.json
Gatling	JSON	standard Gatling simulation log
Pact	JSON	standard Pact verification output
Allure	JSON	allure generate results directory
CTRF	JSON	any CTRF-compliant reporter
SARIF	JSON	any SARIF-compliant scanner
Any JUnit-compatible	XML	TestNG, Karate, REST Assured, Insomnia CLI

Install

npm (global — JS/CI devs):

npm install -g ai-test-failure-analyzer
ai-analyze analyze playwright-report.json

npx (zero install):

npx ai-test-failure-analyzer analyze playwright-report.json

pipx (Python devs):

pipx install ai-test-failure-analyzer
analyzer analyze playwright-report.json

Claude Code skill:

/plugin install ai-test-failure-analyzer

Install skill to all agents (Claude, Cursor, Codex, Gemini, Windsurf):

ai-analyze install

Docker:

docker run --rm -v $(pwd):/workspace \
  ghcr.io/aks-builds/ai-test-failure-analyzer \
  analyze /workspace/results.json

Usage

CLI

ai-analyze analyze results.json
ai-analyze analyze results.json --mode api-only    # force API-only (no source scan)
ai-analyze analyze results.json --out report.md    # write Markdown report to file
ai-analyze analyze results.json --format json --out report.json   # JSON output
ai-analyze analyze results.json --format ctrf --out report.ctrf.json  # CTRF output
ai-analyze analyze results.json --create-issue     # file GitHub issue for top hypothesis
ai-analyze analyze results.json --enrich           # add AI root-cause annotations
ai-analyze analyze results.json --no-cache         # bypass result cache
ai-analyze watch results.json                      # re-analyze on every file change

CTRF output

ai-analyze analyze results.json --format ctrf --out analysis.ctrf.json

The CTRF output embeds ai root-cause annotations per test and includes a summary block:

{
  "results": {
    "tool": { "name": "ai-test-failure-analyzer", "version": "2.0.0" },
    "summary": { "tests": 42, "passed": 38, "failed": 3, "skipped": 1, "other": 0 },
    "tests": [
      {
        "name": "POST /api/clips → 404",
        "status": "failed",
        "duration": 312,
        "ai": "Root cause [95%]: endpoint moved. Evidence: git+config.",
        "extra": { "hypothesis_confidence": 95, "flakiness_score": 0.1 }
      }
    ]
  }
}

MCP server (Claude Code / Cursor)

Add to your MCP config:

{
  "mcpServers": {
    "ai-test-failure-analyzer": {
      "command": "ai-analyze",
      "args": ["serve-stdio"]
    }
  }
}

Then ask Claude: "Analyze the failures in playwright-report.json"

MCP HTTP (OpenAI / Gemini)

ai-analyze serve-http --port 8765

API-only mode

No source code? No problem. When your workspace has no src/, app/, lib/, or api/ directory — or when you pass --mode api-only — the tool switches to API-only mode.

It analyzes HTTP contract evidence directly from the test results:

ai-analyze analyze newman-results.json
# > API_ONLY mode — no workspace source detected, analyzing HTTP contract only
# Root Cause [95%] — POST /api/clips → 404 Not Found
#   Endpoint moved or removed. Check API changelog or versioning.
#   Evidence: response status 404 + URL /api/clips

Supports Newman, k6, Playwright (API tests), Jest, and any framework that records HTTP status codes.

CI integration

# .github/workflows/analyze-failures.yml
- name: Analyze test failures
  if: failure()
  run: |
    npx ai-test-failure-analyzer analyze test-results/results.json \
      --non-interactive \
      --out failure-analysis.md
- uses: actions/upload-artifact@v4
  if: failure()
  with:
    name: failure-analysis
    path: failure-analysis.md

Security

• No shell injection: all subprocess calls use explicit argument lists
• Path traversal protection: all paths resolved relative to workspace root
• Size caps: 5 MB/file, 50 MB/scan, 200 commits max
• Secrets redacted: .env token/secret/key/password values masked in reports
• No outbound network from core analysis (GitHub issue creation is opt-in)

See SECURITY.md for the full threat model.

Repository layout

analyzer/                   Python package (MCP server + CLI + analysis)
  parsers/                  24 framework parsers (Playwright, Jest, Vitest, Cypress, WDIO,
                            Mocha, Detox, pytest, Newman, k6, Go, RSpec, PHPUnit, NUnit,
                            xUnit, MSTest, Robot Framework, Artillery, Gatling, Pact,
                            Allure, CTRF, SARIF, JUnit-compatible)
  evidence/                 Evidence collection (git, logs, config, flaky history)
  intelligence/             Flaky detector, clusterer, enricher
  render/                   Report rendering (Markdown, ANSI, CTRF)
  ui/                       User interfaces (CLI, TUI, Web)
  workspace_scanner.py      Phase 0 — mode detection, noise path discovery
  noise_filter.py           Evidence filtering and hypothesis deduplication
  orchestrator.py           10-phase analysis pipeline
  hypothesis.py             Confidence scoring and hypothesis formation
bin/cli.js                  Zero-dep Node wrapper (ai-analyze command)
skills/ai-test-failure-analyzer/SKILL.md  Claude Code agent skill
.claude-plugin/             Claude marketplace manifests
tests/analyzer/             pytest test suite
.github/workflows/          CI/CD (ci, release, publish, codeql)

Testing

pytest tests/analyzer -q    # Python: parsers, correlator, noise filter, workspace scanner
npm test                    # Node: CLI smoke tests

Contributing

See CONTRIBUTING.md.