qa-agent 0.1.1

Agentic exploratory QA testing for web applications

QA Agent

Automated exploratory QA testing for web applications — powered by Playwright and, optionally, Claude.

Point QA Agent at a URL and it explores your application like a real user: clicking buttons, filling forms, navigating with the keyboard, and checking for accessibility issues. Then it reports what it finds. No test scripts to write or maintain.

Need targeted tests? Pass plain-English instructions and Claude generates custom Playwright steps that run alongside the standard suite.

---

Table of Contents

• Features
• Installation
• Quick Start
• Agentic Testing
• Web Interface
• CLI Reference
• Programmatic Usage
• Test Categories
• Output Formats
• CI/CD Integration
• Architecture
• Development
• Contributing
• Exit Codes
• Troubleshooting
• License

---

Features

Category	What it does
Agentic testing	Give Claude a bug report or feature spec; it generates custom Playwright test steps automatically
Two modes	focused tests only given URLs; explore crawls and discovers pages
Six test suites	Keyboard · mouse · forms · accessibility · error detection (on by default) + WCAG 2.1 AA compliance (opt-in)
Auth support	Username/password, cookies, Bearer tokens, custom headers
Four output formats	Console, Markdown, JSON, PDF
Screenshots & video	On-error or every-interaction screenshots; full session recording
Web UI	Dashboard for launching runs, live output, and browsing past sessions
CI/CD ready	Exit codes map to pass/fail; JSON output integrates with any pipeline

---

Installation

Requires Python 3.10+. Check with python --version.

pip install qa-agent            # standard testing (Playwright only)
playwright install chromium     # required — downloads browser binaries

Optional extras:

pip install "qa-agent[ai]"     # agentic testing (adds Anthropic SDK)
pip install "qa-agent[pdf]"    # PDF reports (adds WeasyPrint)
pip install "qa-agent[web]"    # web UI (adds Flask)
pip install "qa-agent[all]"    # everything above

Agentic testing also requires an API key:

export ANTHROPIC_API_KEY=sk-ant-...

playwright install chromium must run once after every fresh install. See Troubleshooting if anything goes wrong.

---

Quick Start

# Test a single URL
qa-agent https://example.com

# Test multiple URLs
qa-agent https://example.com https://example.com/about

# Crawl and test discovered pages
qa-agent --mode explore --max-depth 2 https://example.com

# Generate reports in a custom directory
qa-agent --output json,markdown --output-dir ./reports https://example.com

# Run via module
python -m qa_agent https://example.com

---

Agentic Testing

Pass natural-language instructions and Claude generates custom test steps that run alongside the standard suite.

# From a bug report
qa-agent --instructions "The login button does nothing when email is blank" \
  https://example.com/login

# From a feature spec
qa-agent --instructions "The 'Remember me' checkbox should be unchecked by default \
  and persist the session across browser restarts." \
  https://example.com/login

# From a file
qa-agent --instructions-file feature-spec.txt https://example.com

How it works

1. Claude receives your instructions and the target URL.
2. It returns a structured plan: summary, focus areas, and custom Playwright test steps.
3. The agent runs those steps on every tested page alongside the standard suites.
4. Assertion failures become findings in the report with the severity Claude assigned.

If the API call fails (or the [ai] extra isn't installed), a warning is printed and the run continues with standard tests only.

Model & caching

# Use a different model (default: claude-sonnet-4-6)
qa-agent --ai-model claude-opus-4-6 --instructions "Test checkout" https://shop.example.com

# Bypass the plan cache
qa-agent --no-cache --instructions "..." https://example.com

Plans are cached to ~/.qa_agent/cache/ (24-hour TTL). Pass --no-cache to force a fresh API call.

---

Web Interface

python -m qa_agent web              # http://127.0.0.1:5000
qa-agent-web --host 0.0.0.0 --port 8080  # custom bind

• Configuration form with all CLI options
• Real-time streaming output (Server-Sent Events)
• Stop a running test mid-run
• Browse past sessions grouped by domain
• Session detail: findings table, severity breakdown, screenshot gallery, report downloads

No authentication — intended for local or internal use only.

Output is written to output/ by default. CLI sessions appear in the web UI automatically (JSON is always written).

---

CLI Reference

qa-agent --version
qa-agent --help

Modes

qa-agent --mode focused https://example.com   # default — test only given URLs
qa-agent --mode explore https://example.com    # crawl and test discovered pages

Exploration (explore mode)

Flag	Default	Description
--max-depth N	3	Max link depth
--max-pages N	20	Max pages to test
--allow-external	off	Follow links to other domains
--ignore PATTERN	—	URL regex to skip (repeatable)

Authentication

qa-agent --auth "user:pass@https://example.com/login" https://example.com/dashboard
qa-agent --auth-file auth.json https://example.com
qa-agent --cookies cookies.json https://example.com
qa-agent --header "Authorization: Bearer token123" https://example.com

auth.json schema

{
  "username": "testuser",
  "password": "testpass",
  "auth_url": "https://example.com/login",
  "username_selector": "input#email",
  "password_selector": "input#password",
  "submit_selector": "button[type=submit]"
}

Output

qa-agent --output console,markdown,json,pdf https://example.com
qa-agent --output-dir ./reports https://example.com

Default: console,markdown. JSON is always written regardless of --output (for web UI discovery). Output is organized as output/{domain}/{session_id}/qa_reports|screenshots|recordings.

PDF requires the [pdf] extra. Falls back to Markdown if WeasyPrint is not installed.

Screenshots & recording

qa-agent --screenshots       https://example.com  # on errors
qa-agent --screenshots-all   https://example.com  # every interaction
qa-agent --full-page         https://example.com  # full-page captures
qa-agent --record            https://example.com  # session video

Browser

qa-agent --no-headless                  # visible browser window
qa-agent --viewport 1920x1080           # default: 1280x720
qa-agent --timeout 60000                # ms, default: 30000

Test suites

# Skip standard suites
qa-agent --skip-keyboard      https://example.com
qa-agent --skip-mouse         https://example.com
qa-agent --skip-forms         https://example.com
qa-agent --skip-accessibility https://example.com
qa-agent --skip-errors        https://example.com

# Enable opt-in suites
qa-agent --wcag-compliance    https://example.com

---

Programmatic Usage

from qa_agent import QAAgent, TestConfig, TestMode, OutputFormat

config = TestConfig(
    urls=["https://example.com"],
    mode=TestMode.EXPLORE,
    output_formats=[OutputFormat.CONSOLE, OutputFormat.JSON],
    max_depth=2,
    max_pages=10,
    instructions="Verify the password reset flow.",  # optional
)

agent = QAAgent(config)
session = agent.run()

print(f"Pages tested:   {len(session.pages_tested)}")
print(f"Total findings: {session.total_findings}")

for finding in session.get_all_findings():
    print(f"  [{finding.severity.value.upper()}] {finding.title}")

---

Test Categories

Keyboard Navigation

TAB order and focusability · Arrow key navigation · Enter key activation · Escape key for modals · Keyboard trap detection · Focus visibility

Mouse Interaction

Click targets · Hover states · Double-click · Right-click/context menus · Target size (WCAG 2.5.5, 44×44 px min) · Overlapping elements

Form Handling

Required field indicators · Validation feedback · Error message accessibility · Label associations · HTML5 input types · Autocomplete attributes

Accessibility (WCAG)

Alt text · Heading structure (h1–h6) · Link text quality · Color contrast · ARIA usage · Landmark regions · Language attributes · Skip navigation

Error Detection

Console errors/warnings · Network errors (4xx, 5xx) · JavaScript exceptions · Broken images · Broken anchors · Mixed content

WCAG 2.1 AA Compliance (opt-in: --wcag-compliance)

Non-text contrast (1.4.11) · Use of color (1.4.1) · Content on hover/focus (1.4.13) · Meaningful sequence (1.3.2) · Input purpose (1.3.5) · Focus visible (2.4.7) · Label in name (2.5.3) · Target size (2.5.5) · Language of parts (3.1.2) · Error identification (3.3.1) · ARIA role/property validation

---

Output Formats

Console

JSON

{
  "meta": {
    "session_id": "a1b2c3d4",
    "start_time": "2024-01-15T10:30:00",
    "duration_seconds": 45.2
  },
  "summary": {
    "pages_tested": 5,
    "total_findings": 12,
    "findings_by_severity": { "high": 2, "medium": 5, "low": 5 }
  },
  "findings": [...]
}

Severity levels

Level	Meaning
CRITICAL	Security issues, data loss
HIGH	Major usability blockers
MEDIUM	UX problems, accessibility issues
LOW	Minor improvements, best practices
INFO	Informational observations

---

CI/CD Integration

# GitHub Actions example
- name: Run QA Tests
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
  run: |
    pip install "qa-agent[ai]"
    playwright install chromium
    qa-agent --output json --output-dir ./qa-results https://staging.example.com

- name: Upload Results
  uses: actions/upload-artifact@v4
  with:
    name: qa-results
    path: ./qa-results/

Exits with code 1 when critical or high severity issues are found, failing the CI step automatically. See Exit Codes.

Omit [ai] and the ANTHROPIC_API_KEY env var if you only need standard tests.

---

Architecture

qa_agent/
├── cli.py                   # CLI entry point
├── agent.py                 # Core orchestrator
├── config.py                # Configuration dataclasses
├── models.py                # Finding, PageAnalysis, TestSession, TestPlan
├── ai_planner.py            # Claude integration for plan generation
├── plan_cache.py            # Filesystem cache for test plans
├── testers/
│   ├── base.py              # BaseTester abstract class
│   ├── keyboard.py          # Keyboard navigation
│   ├── mouse.py             # Mouse interaction
│   ├── forms.py             # Form handling
│   ├── accessibility.py     # WCAG accessibility
│   ├── wcag_compliance.py   # WCAG 2.1 AA compliance (opt-in)
│   ├── errors.py            # Console & network errors
│   └── custom.py            # AI-generated test steps
├── reporters/
│   ├── console.py           # Colored terminal output
│   ├── markdown.py          # Markdown report
│   ├── json_reporter.py     # JSON report
│   └── pdf.py               # PDF report (requires weasyprint)
└── web/
    ├── server.py             # Flask app with SSE streaming
    ├── templates/            # Jinja2 templates
    └── static/               # CSS and JavaScript

Adding a custom tester

1. Create testers/my_tester.py extending BaseTester, implement run() -> list[Finding]
2. Export from testers/__init__.py
3. Add a test_my_feature: bool flag to TestConfig
4. Call from agent.py in _test_page()

---

Development

git clone https://github.com/billrichards/qa-agent.git
cd qa-agent
pip install -e ".[dev,web,ai]"
playwright install chromium

# Unit tests (no browser needed)
pytest -v -m "not integration and not network"

# Integration tests (real Playwright)
pytest -v -m integration --no-cov

# Lint & type check
ruff check .
mypy qa_agent

CI runs unit tests across Python 3.10–3.12 on Ubuntu, macOS, and Windows. Integration tests run on Ubuntu with Playwright. See .github/workflows/test.yml.

---

Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b my-feature)
3. Make your changes and add tests
4. Run pytest -v -m "not integration and not network"
5. Open a pull request against main

Code style: Ruff + Black, line length 100.

---

Exit Codes

Code	Meaning
0	All tests passed (no critical/high findings)
1	Critical or high severity issues found
2	Error running tests
130	Interrupted (Ctrl+C)

---

Troubleshooting

Playwright browser not found

playwright install chromium

Must run once after every fresh install. Easy to forget in CI.

Web UI not working

pip install "qa-agent[web]"

Required for qa-agent-web and python -m qa_agent web.

PDF reports missing

pip install "qa-agent[pdf]"

Falls back to Markdown silently if WeasyPrint is absent.

Agentic testing skipped

pip install "qa-agent[ai]"
export ANTHROPIC_API_KEY=sk-ant-...

If the package or key is missing, qa-agent prints a warning and continues with standard tests.

Python version too old

Requires 3.10+. Check with python --version.

---

License

MIT — Copyright (c) 2026 Bill Richards. See LICENSE.