AIQA — an AI-powered Quality Engineering SDK

Turn a failing test from any automation framework into an evidence-grounded root-cause analysis, a confidence score, an owning team, and a tracker-ready bug report.

AIQA is not a Playwright tool. Its core understands only failures, evidence, context, analysis, and reports — never a specific application, framework, or DOM. Framework knowledge lives entirely in swappable adapters, so the same engine works with Playwright, Selenium, Cypress, Robot Framework, Appium, Requests, REST Assured, JUnit, NUnit, TestNG, pytest — or anything that can emit JSON.

It runs fully offline with zero API keys (deterministic heuristic engine + pure-Python similarity search) and upgrades transparently to an LLM when you configure one.

Install name: multi-framework-tc-failure-ai-analyzer · Import name: aiqa

Renamed package. This project was formerly published as playwright-tc-failure-ai-analyzer. It is now multi-framework-tc-failure-ai-analyzer to reflect its framework-agnostic, adapter-based design. Future releases ship only under the new name — see the Migration Guide.

---

Architecture

Dependencies point in one direction only. The core domain depends on nothing; everything depends on the core.

   Adapters                 (framework-specific: Playwright, Selenium, pytest…)
      │  produce
      ▼
   FailureContext           ← Core Domain (pure, framework-agnostic models)
      │  consumed by
      ▼
   AI Analysis Engine       analyze(context) -> AnalysisResult   (never imports a framework)
      │  produces
      ▼
   Reporting                (Markdown · JSON · HTML · Console · multi-run portal · your own)
      │
      ▼
   Output

Layer	Package	Knows a framework?	Depends on
Adapters	aiqa.adapters	Yes (only here)	core
Core domain	aiqa.core	No	standard library only
AI engine	aiqa.analysis	No	core (+ optional LLM SDK, lazy)
Reporting	aiqa.reporting	No	core

The engine's entire contract is one method: analyze(context: FailureContext) -> AnalysisResult.

---

Install

pip install multi-framework-tc-failure-ai-analyzer            # core SDK (offline)
pip install "multi-framework-tc-failure-ai-analyzer[openai]"  # + LLM analysis

Quick start

from aiqa import FailureAnalyzer, FailureContext, render

context = FailureContext.from_dict({
    "metadata": {"test_name": "checkout::test_pay", "framework": "cypress"},
    "exception": {"type": "AssertionError", "message": "server returned HTTP 500"},
    "evidence": {"network": [{"method": "POST", "url": "/api/pay", "status": 500}]},
})

result = FailureAnalyzer().analyze(context)          # offline by default
print(render(result, "markdown", context))
print(result.category.value, result.confidence.value, result.owner)

Using adapters

Every adapter exposes collect_failure_context(...) and returns a FailureContext.

Playwright

from aiqa import FailureAnalyzer, render
from aiqa.adapters import PlaywrightAdapter, PlaywrightEventRecorder

recorder = PlaywrightEventRecorder(page)   # at test start (captures console/network)
# ... on failure:
context = PlaywrightAdapter(page=page, recorder=recorder).collect_failure_context(
    exc, test_name="login::test_submit", screenshot="fail.png", browser="chromium",
)
print(render(FailureAnalyzer().analyze(context), "console", context))

Selenium

from aiqa.adapters import SeleniumAdapter
context = SeleniumAdapter(driver=driver).collect_failure_context(exc, test_name="orders::checkout")

pytest (any test type — UI, API, unit)

from aiqa.adapters import PytestAdapter
# in conftest.py pytest_runtest_makereport:
context = PytestAdapter().collect_failure_context(item=item, call=call, report=report)

Robot Framework

from aiqa.adapters import RobotFrameworkAdapter
context = RobotFrameworkAdapter().collect_failure_context(
    test_name="Login Works", message="Element not visible", suite="Login")

Anything (JSON) — Cypress, REST Assured, JUnit, NUnit, TestNG, CI scripts:

from aiqa.adapters import GenericAdapter
context = GenericAdapter().collect_failure_context("failure.json")   # dict | JSON str | path

Runnable scripts live in examples/.

CI / GitHub Action

A reusable GitHub Action analyzes failures in any workflow and publishes the results to the Job Summary, a PR comment, and an artifact — offline by default:

- name: AI Failure Analysis
  if: always()
  uses: amandeepsdet/multi-framework-tc-failure-ai-analyzer/.github/actions/analyze-failures@v1
  with: { report-path: reports/, framework: pytest, post-comment: true }

It is orchestration only — it reuses the same adapters, engine, and QualityPortal. See docs/github-action.md.

---

Core domain

Pure, JSON-serialisable dataclasses in aiqa.core, independent of every framework:

• FailureContext — the single input to the engine, composed of:
  • FailureMetadata (test id/name, suite, framework, tags, timing, git commit)
  • ExceptionInfo (type, message, stacktrace)
  • Evidence (screenshot, DOM snapshot, console, network, API responses, logs, artifacts, and open-ended custom evidence)
  • ExecutionContext (environment, browser, OS, URL, timestamp, configuration)
• AnalysisResult — RootCause, ConfidenceScore, Severity, owner, evidence, Recommendation[], SimilarFailure[]
• BugReport — a tracker-ready report

AI analysis

FailureAnalyzer.analyze(context):

1. Retrieves similar past failures (RAG) — optional, pure-Python by default.
2. If an LLM provider is available, asks for an evidence-grounded JSON verdict; any failure transparently falls back to…
3. …a deterministic heuristic classifier (HTTP 5xx → Backend, 401 → Auth, locator / timeout / assertion signals, console errors, …).
4. Assigns a category, confidence, severity, and owning team.

Every claim is grounded in the collected evidence — the model cannot invent signals that were not captured.

Reports

Reporters consume an AnalysisResult only. Built in: markdown, json, html, console.

from aiqa import get_reporter, available_formats
print(available_formats())                     # ['console', 'html', 'json', 'markdown']
html = get_reporter("html").render(result, context)

Bug reports:

from aiqa.reporting import BugReportBuilder
bug = BugReportBuilder().build(result, context)
print(BugReportBuilder.to_markdown(bug))

Quality Intelligence Platform (multi-run portal)

The reporters above render a single result. For a historical, multi-run view, QualityPortal aggregates every execution into an execution history, a knowledge base, trends, and a self-contained HTML dashboard — without touching the core models, adapters, or analysis engine, and with zero extra dependencies (pure standard library).

from aiqa import FailureAnalyzer, QualityPortal

portal = QualityPortal("reports")
portal.begin_run(framework="playwright", environment="staging")

# per analysed failure during the run:
portal.add_failure(result, context)
portal.add_success("suite::test_ok")   # optional, for an accurate pass rate

run = portal.finish_run()
# -> reports/run_*/ai_report.html  (this execution)
# -> reports/index.html            (dashboard across all executions)

Each run gets its own run_*/ folder with one HTML report; index.html is regenerated to discover every run. On top of the raw results the portal derives:

Capability	What it gives you
Quality Score & build health	Weighted 0–100 score (Excellent / Good / Warning / Poor) + Healthy / Warning / Critical signal.
Release readiness	READY / AT_RISK / NOT_READY verdict with reasons.
Run comparison	New / resolved / persisting failures and regressions, keyed by a stable failure signature.
Flaky detection	Pass/fail transition analysis over a sliding window.
Knowledge base	Recurring-failure recall, occurrence counts, most successful historical fix.
Trends & insights	Pass rate, failures, confidence, duration, and quality across runs, plus an AI executive summary.

---

Extensibility

Every seam is an interface (aiqa.core.interfaces) — extend without touching existing code.

Custom LLM provider — implement three members:

class MyProvider:
    name = "my-llm"
    def available(self): return True
    def complete_json(self, prompt, system=""): return {...}

FailureAnalyzer(llm=MyProvider())

Custom reporter — subclass and register:

from aiqa.core.interfaces import Reporter
from aiqa.reporting import register_reporter

@register_reporter
class SlackReporter(Reporter):
    format = "slack"
    def render(self, result, context=None): return f":rotating_light: {result.root_cause.summary}"

Custom adapter — subclass FrameworkAdapter and return a FailureContext. Custom similarity/RAG backend — implement the SimilarityIndex protocol.

Configuration (all optional, env-driven)

Variable	Default	Purpose
AIQA_PROVIDER	offline	offline (heuristic) or openai.
AIQA_MODEL	gpt-4o-mini	Model name for the chosen provider.
AIQA_API_KEY	–	Key (falls back to OPENAI_API_KEY).
AIQA_BASE_URL	–	Custom/compatible endpoint (Azure, local, …).
AIQA_ENABLE_HISTORY	false	Persist a local similarity index.
AIQA_BASE_DIR	.aiqa	Where history is written.

Offline mode requires no keys and no extra dependencies.

---

Design guarantees

• The core has zero framework and zero application knowledge.
• The engine accepts only a FailureContext and never imports Playwright.
• Reports consume only an AnalysisResult.
• All framework logic is isolated in adapters (framework libs lazy-imported).
• Full type hints, SOLID boundaries, no circular dependencies, unit-testable.

Also included: qa_ai_engine pytest plugin (backward-compatible)

The same distribution still ships the original auto-discovered pytest plugin for pytest + Playwright projects, unchanged. Enable it with AI_ENABLED=true and use from qa_ai_engine import AIEngine. New projects should prefer the framework-agnostic aiqa SDK above.

License

MIT © Aman Deep