playwright-tc-failure-ai-analyzer 1.0.0

Provider-agnostic AI failure-analysis engine and QA assistant that plugs into any pytest + Playwright project. Turns a failing test into an evidence-grounded root-cause analysis and a tracker-ready bug report. Works fully offline with zero API keys.

playwright-tc-failure-ai-analyzer

Drop-in AI failure analysis for any pytest + Playwright project.

playwright-tc-failure-ai-analyzer turns a failing test into an evidence-grounded root-cause analysis, a confidence score, a likely owning team, and a tracker-ready bug report — then attaches it all to your Allure / pytest-html reports. It works fully offline with zero API keys (deterministic heuristic engine + pure-Python embeddings + local JSON vector store), and transparently upgrades to a real LLM when you point it at one.

Install name: playwright-tc-failure-ai-analyzer — import name: qa_ai_engine

---

Install

pip install playwright-tc-failure-ai-analyzer                      # core (offline heuristic engine)
pip install "playwright-tc-failure-ai-analyzer[playwright,reports]"  # + live page evidence + report attachments
pip install "playwright-tc-failure-ai-analyzer[all]"              # everything, including LLM providers

1-minute integration

The package ships a pytest plugin that is auto-discovered — there is nothing to import. Just enable it:

# PowerShell
$env:AI_ENABLED = "true"
pytest

# bash
AI_ENABLED=true pytest

Any test that fails and uses the pytest-playwright page fixture is analysed automatically. Console output on failure:

AI analysis: A locator did not resolve to a visible element within the timeout.
             (Locator, confidence=80%) — owner=UI Automation / QA team

That's it. No conftest.py changes required. The plugin is a no-op unless AI_ENABLED=true, so it never affects normal runs.

Use the engine directly (any framework)

from qa_ai_engine import AIEngine

engine = AIEngine()
outcome = engine.analyze_failure(
    test_name="checkout::test_pay",
    exception=err,          # the caught exception
    page=page,              # optional Playwright page for live evidence
    assertion_message=str(err),
)
print(outcome.analysis.root_cause, outcome.analysis.confidence, outcome.analysis.owner)
print(engine.bug_gen.to_markdown(outcome.bug_report))

QA Assistant CLI

qa-ai status                 # provider / config status
qa-ai analyze-last-failure   # full analysis + bug report for the latest failure
qa-ai release-readiness      # 0-100 go/no-go score from failure history
qa-ai search "login timeout" # RAG search over past failures
qa-ai                        # interactive chat mode

Configuration (all via environment variables)

Variable	Default	Purpose
AI_ENABLED	false	Master switch.
AI_PROVIDER	heuristic	openai | azure | claude | gemini | ollama | heuristic.
AI_MODEL	gpt-4o-mini	Model name for the chosen provider.
AI_API_KEY	–	Key for the chosen provider (or the provider's own env var).
AI_VECTOR_BACKEND	json	json (offline) or chroma.
AI_BASE_DIR	current dir	Where failure_history/, vector_db/, ai_reports/ are written.
QA_AI_DISABLE_PLUGIN	false	Force the pytest plugin off (e.g. when wiring the engine manually).

Offline mode (the default) requires no keys and no extra dependencies.

What you get on every failure

• Root cause + failure category (Locator, Backend, Auth, Network, Data, …)
• Confidence score and severity
• Likely owning team (configurable routing)
• Recommended fix
• RAG search over similar past failures
• A structured, tracker-ready bug report (title, steps, priority, owner)
• JSON / Markdown / HTML artifacts + Allure & pytest-html attachments

Provider setup (opt-in)

All providers are lazily imported — you only need the SDK for the one you use. Set AI_ENABLED=true and AI_PROVIDER, then supply the provider's credentials.

OpenAI

pip install "playwright-tc-failure-ai-analyzer[openai]"
$env:AI_ENABLED="true"; $env:AI_PROVIDER="openai"
$env:AI_MODEL="gpt-4o-mini"; $env:AI_API_KEY="sk-..."

Azure OpenAI

$env:AI_ENABLED="true"; $env:AI_PROVIDER="azure"
$env:AI_API_KEY="<azure-key>"; $env:AI_MODEL="<deployment-name>"
$env:AZURE_OPENAI_ENDPOINT="https://<resource>.openai.azure.com"

Ollama (local, no key)

$env:AI_ENABLED="true"; $env:AI_PROVIDER="ollama"
$env:AI_MODEL="llama3"   # OLLAMA_HOST defaults to http://localhost:11434

If a provider call fails at runtime, the engine automatically falls back to the deterministic offline analyzer — a run is never blocked by a missing key or a network error.

Teach it about your app (optional)

The assistant answers are grounded in a small, generic knowledge base by default. Point it at your own application without touching code:

Variable	Purpose
QA_AI_APP_NAME	Friendly name of the app under test.
QA_AI_FRAMEWORK_CONTEXT	One-paragraph description used for RAG context.
QA_AI_KNOWLEDGE_FILE	Path to a JSON file with richer knowledge.

Example knowledge.json:

{
  "app_name": "Acme Checkout",
  "context": "E-commerce checkout built on React + a REST orders API.",
  "widgets": { "cart": "Cart summary panel on the right rail." },
  "apis": { "orders": "POST /api/orders creates an order and returns an id." }
}

$env:QA_AI_KNOWLEDGE_FILE="knowledge.json"

Troubleshooting

Symptom	Cause / fix
Plugin does nothing	AI_ENABLED is not true. The plugin is a no-op otherwise.
import qa_ai_engine fails	Package not installed in the active interpreter — pip install playwright-tc-failure-ai-analyzer.
No live page evidence	Install the playwright extra and use the page fixture.
Analysis is always "heuristic"	No provider configured — set AI_PROVIDER + credentials (see above).
Double analysis	The engine is wired both via the plugin and manually — set QA_AI_DISABLE_PLUGIN=true for manual wiring.
Artifacts written to the wrong place	Set AI_BASE_DIR to control where failure_history/, vector_db/, ai_reports/ go.

License

MIT