playwright-analyzer 0.1.2

AI-powered Playwright test report analyzer using PydanticAI

Playwright Test Report Analyzer 🤖

An AI-powered tool that analyzes Playwright test reports to provide intelligent insights, failure analysis, and actionable recommendations. The tool uses PydanticAI with OpenAI-compatible models to understand test failures and inject the analysis directly into your Playwright HTML reports.

Features ✨

• Automatic Test Report Parsing: Extracts test results from Playwright HTML reports
• AI-Powered Analysis: Uses reasoning models to analyze failures and identify patterns
• Failure Root Cause Analysis: Identifies why tests failed with categorization
• Pattern Recognition: Detects common failure patterns across multiple tests
• Flaky Test Detection: Identifies tests with intermittent failures
• Performance Insights: Highlights slow tests and performance issues
• HTML Report Enhancement: Injects AI insights directly into the Playwright report
• Flexible Model Support: Works with OpenAI, Ollama, or any OpenAI-compatible API

Prerequisites 🔧

• Python 3.9 or higher
• UV package manager

Installing UV

# Install UV (if not already installed)
curl -LsSf https://astral.sh/uv/install.sh | sh

# Or using Homebrew on macOS
brew install uv

Installation 📦

# Navigate to the analyzer directory
cd playwright-analyzer

# Install dependencies using UV
uv sync

# For development dependencies
uv sync --dev

Configuration ⚙️

1. Copy the example environment file:

cp .env.example .env

2. Edit .env with your configuration:

# For OpenAI
OPENAI_API_KEY=your_api_key_here
MODEL_NAME=gpt-4o-mini

# For local models (e.g., Ollama)
OPENAI_API_BASE=http://localhost:11434/v1
MODEL_NAME=llama3.2

Usage 🚀

Basic Usage

After running your Playwright tests:

# Run Playwright tests first
pnpm test:e2e

# Analyze the report
cd playwright-analyzer
uv sync

# Use the CLI command
uv run playwright-analyzer --help

Command Line Options

uv run playwright-analyzer --help

Options:
  --report PATH      Path to Playwright HTML report (default: ../playwright-report/index.html)
  --context PATH     Path to context file (default: ../CONTEXT-TEST.md)
  --model NAME       Model name (default: from .env or gpt-4o-mini)
  --api-key KEY      API key (default: from .env)
  --api-base URL     API base URL (default: from .env or OpenAI)
  --no-inject        Don't inject insights into HTML report

Examples

# Use GPT-4 for more detailed analysis
uv run playwright-analyzer --model gpt-4o

# Use local Ollama model
uv run playwright-analyzer --model llama3.2 --api-base http://localhost:11434/v1

# Analyze without modifying the HTML report
uv run playwright-analyzer --no-inject

# Custom report path
uv run playwright-analyzer --report /path/to/playwright-report/index.html

Integration with CI/CD 🔄

Add to your GitHub Actions workflow:

- name: Run Playwright Tests
  run: pnpm test:e2e
  continue-on-error: true

- name: Install UV
  if: always()
  uses: astral-sh/setup-uv@v2
  with:
    enable-cache: true
    cache-dependency-glob: "playwright-analyzer/pyproject.toml"

- name: Analyze Test Report
  if: always()
  run: |
    cd playwright-analyzer
    uv sync
    uv run playwright-analyzer
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

- name: Upload Enhanced Report
  if: always()
  uses: actions/upload-artifact@v3
  with:
    name: playwright-report-with-insights
    path: playwright-report/

Output 📊

The analyzer provides:

1. Console Output

• Test summary statistics
• Detailed failure analysis
• Recommendations and patterns
• Risk areas and performance issues

2. Enhanced HTML Report

The original Playwright report is enhanced with:

• AI insights section with visual metrics
• Failure analysis with root causes
• Recommendations panel
• Common patterns detection
• Flaky test identification

3. JSON Export

Insights are also saved to playwright-report/ai-insights.json for programmatic access.

Architecture 🏗️

playwright-analyzer/
├── src/
│   └── playwright_analyzer/
│       ├── __init__.py          # Package initialization
│       ├── analyze_report.py    # Main CLI entry point
│       ├── report_parser.py     # Parses Playwright HTML reports
│       ├── agent.py            # PydanticAI agent for analysis
│       └── report_injector.py  # Injects insights into HTML
├── pyproject.toml              # UV/Python project configuration
├── .python-version             # Python version for UV
├── .env.example                # Environment template
└── README.md                   # This file

How It Works 🔍

1. Report Parsing: Extracts test data from Playwright's embedded JSON
2. Context Loading: Loads application context (optional)
3. AI Analysis: Sends failure data to AI model for analysis
4. Insight Generation: Creates structured insights with PydanticAI
5. HTML Injection: Adds insights to the original report
6. Export: Saves insights as JSON for further processing

Models Supported 🤖

• OpenAI: GPT-4, GPT-4 Turbo, GPT-3.5 Turbo
• Anthropic: Claude (via OpenAI-compatible API)
• Ollama: Llama, Mistral, CodeLlama (local)
• Any OpenAI-compatible API

Features in Detail 📝

Failure Analysis

• Categorizes failures (UI, Network, Timing, Logic)
• Identifies root causes
• Provides specific fix suggestions
• Assesses impact level

Pattern Detection

• Groups similar failures
• Identifies systemic issues
• Detects configuration problems
• Highlights regression patterns

Performance Insights

• Identifies slow tests
• Detects timeout issues
• Suggests optimization opportunities
• Monitors test duration trends

Flaky Test Detection

• Identifies tests with retries
• Detects intermittent failures
• Suggests stabilization strategies
• Tracks flakiness patterns

Development 🧪

Running Tests

# Run tests with UV
uv run pytest

# Run with coverage
uv run pytest --cov

# Format code
uv run black .

# Lint code
uv run ruff check .

Installing Development Dependencies

# Install all dependencies including dev
uv sync --dev

Troubleshooting 🛠️

"Report file not found"

• Ensure Playwright tests have been run: pnpm test:e2e
• Check the report path: ls ../playwright-report/index.html

"API key is required"

• Set OPENAI_API_KEY in .env file
• Or use --api-key argument
• Or use a local model with --api-base

"Could not extract test data"

• Ensure you're using a recent version of Playwright
• Check that the HTML report contains embedded JSON data

"Module not found" errors

• Ensure UV is installed: brew install uv or curl -LsSf https://astral.sh/uv/install.sh | sh
• Run uv sync to install dependencies
• Use uv run prefix for all Python commands

Contributing 🤝

Contributions are welcome! Areas for improvement:

• Support for more test frameworks
• Additional analysis patterns
• Performance optimizations
• More model integrations

License 📄

MIT