aat-devqa 1.7.1

AWT — AI Watch Tester
I got tired of writing E2E tests. So I built something that writes and fixes them for me.

_{30-second demo: install → run test → AI self-heals failure}

Why I built this

I was building a Flutter web app. Every time the UI changed, my Playwright tests broke. Fixing selectors, re-recording flows, updating assertions — it ate hours every week.

So I started hacking on a tool that could look at a page, write its own tests, and when those tests broke, figure out why and fix itself.

That became AWT.

It's not perfect. OCR-based matching can be flaky on certain fonts. The self-healing loop sometimes needs a nudge. But for my day-to-day QA work — especially on Canvas and Flutter Web apps that Cypress can't even touch — it works well enough that I stopped writing test code entirely.

I'm sharing it because I suspect I'm not the only one who got fed up. If you've ever spent an afternoon fixing broken selectors, this might save you some time.

What it does

You give AWT a URL. It opens a real Chrome window, reads the page, writes test steps, runs them, and reports what passed and what failed. If something breaks, the DevQA Loop kicks in — AI reads the error, patches the test, and retries automatically (up to 5 times).

No test code to write. No recording sessions. No manual selector updates.

Start in 5 Minutes

Option 1 — Agent Skill (works with Claude Code, Cursor, Copilot...)

# One-line install
npx skills add ksgisang/awt-skill --skill awt -g

# Then just ask your AI tool:
"Test the login flow on http://localhost:3000"
"Check if the signup form works"
"Run regression tests after my last commit"
# → AWT scans, generates steps, runs them, and reports back

Option 2 — MCP Server (Claude Desktop, Cursor, Windsurf)

# Install
pip install aat-devqa

# Add to Claude Code
claude mcp add awt -- python mcp/server.py

# Tools available: aat_devqa, aat_run, aat_doctor, aat_list_scenarios,
#                  aat_validate, aat_generate_from_doc,
#                  aat_snapshot, aat_diff, aat_watch

Cloud & Local CLI versions are in active development and will be announced separately.

How It Works

You give AWT a URL
        │
        ▼
  🔍 SCAN — AWT opens Chrome and reads every button, input, and link
        │
        ▼
  📝 GENERATE — AI writes a step-by-step test plan (you review & approve)
        │
        ▼
  ▶️  RUN — AWT clicks, types, and navigates like a real user
        │
        ├── ✅ All passed → screenshot report saved
        │
        └── ❌ Something failed
                    │
                    ▼
            🔄 DEVQA LOOP — AI reads the failure,
               fixes the test (or your code),
               and tries again (up to 5 times)

The DevQA Loop — AWT's Core Feature

Most testing tools stop when a test fails and wait for a human. AWT keeps going.

When a step fails, AWT:

1. Takes a screenshot of exactly what the browser shows
2. Reads the error message and the visible page content
3. Re-scans the page to check if anything moved or changed
4. Patches the specific failing step and retries

If the failure is a bug in your source code (not just a wrong selector), AWT can trace it — finding the route handler, component, or API endpoint that's misbehaving — and suggest or apply a fix.

# Watch the loop run live
aat devqa "checkout flow test" --url http://localhost:3000

# Or use it with your AI coding tool (Claude Code, Cursor, Copilot...)
# "Test the registration page" → AWT scans, generates, runs, fixes

Four Ways to Use AWT

Agent Skill — Let your AI coding tool drive AWT

# One-line install
npx skills add ksgisang/awt-skill --skill awt -g

# Then ask your AI tool:
"Test the login flow on http://localhost:3000"
"Check if the signup form works"
"Run regression tests after my last commit"
# → AWT scans, generates test steps, runs them, and reports back

MCP Server — Protocol-native

# Add to Claude Code
claude mcp add awt -- python mcp/server.py

# Tools available: aat_run, aat_doctor, aat_list_scenarios, aat_validate,
#                  aat_cost, aat_generate_from_doc,
#                  aat_snapshot, aat_diff, aat_watch

Use AWT as a Quality Gate in Your AI Harness

If you use GSD (Get Shit Done), Cursor Rules, CLAUDE.md, or any AI coding workflow — AWT plugs in as the Verify step that catches what your AI missed.

GSD Workflow Integration

Discuss → Plan → Execute → Verify (AWT)
                              │
                    aat_generate_from_doc(PLAN.md)
                              │
                    → scenarios generated
                    → aat_run_skill_mode
                    → pass/fail report

# MCP: pass your GSD plan file to generate test scenarios
"Generate test scenarios from my PLAN.md and run them"
# → AWT reads the plan, creates E2E tests, runs in real Chrome

# CLI: same thing
aat generate --from PLAN.md
aat run scenarios/

Add AWT to Your CLAUDE.md / Cursor Rules

## QA Verification (AWT)
- After writing code, verify with AWT before committing.
- Run: aat scan --url {URL} → review scenario → aat run
- All tests must pass before marking a task as done.

This turns AWT into an automated quality gate — your AI writes code, AWT verifies it works, and the loop continues until everything passes.

Visual Regression — Catch UI Changes Automatically

AWT detects unintended UI changes by comparing screenshots before and after code changes. Zero AI tokens — pure Playwright + OpenCV.

# Step 1: Capture baselines (before code change)
aat snapshot scenarios/login.yaml --url http://localhost:3000

# Step 2: After code changes, compare
aat diff scenarios/login.yaml --url http://localhost:3000

# Result:
# step001  99.8%  ✅ PASS
# step002  87.3%  ❌ FAIL  ← UI change detected
# step003 100.0%  ✅ PASS

Responsive — Test 3 Viewports at Once

# Capture mobile + tablet + desktop baselines
aat snapshot scenarios/ --url http://localhost:3000 --responsive

# Compare all 3 viewports
aat diff scenarios/ --url http://localhost:3000 --responsive

# Or test a single custom viewport
aat snapshot scenarios/ --viewport 414x896

Console Error Collection

# Capture baselines and check for JS errors
aat snapshot scenarios/ --url http://localhost:3000 --console

# Fail the run if console errors are found
aat snapshot scenarios/ --url http://localhost:3000 --console-fail

Catches hidden JavaScript errors that pass visual checks — TypeError, network 404s, uncaught exceptions.

Auto-Open Diff Images

# Open failed diff images in your system viewer
aat diff scenarios/ --url http://localhost:3000 --open

# → macOS: opens in Preview
# → Linux: opens with xdg-open

Watch Mode — Auto-Test on File Changes

aat watch scenarios/ --url http://localhost:3000
# → Saves a file? Tests re-run automatically
# → Baselines exist? Visual diff included
# → Ctrl+C to stop

• Scenario file changed → re-run that scenario only
• Source file changed → re-run all scenarios (full regression)
• Powered by watchfiles (Rust-based), with polling fallback

PR Comments (GitHub Action)

# Output GitHub-flavored markdown for PR comments
aat diff scenarios/ --format=github

# JSON output for other tools
aat diff scenarios/ --format=json

A .github/workflows/visual-regression.yml template is included — drop it into your repo and every PR gets automatic visual regression comments.

PDF Reports — Something You Can Send to Someone

aat run scenarios/ --report pdf
# → reports/<scenario id>/report.pdf  (+ the report.html it was printed from)

aat run scenarios/ --report pdf --report-screenshots all   # picture the passing steps too
aat loop scenarios/ --report-format pdf   # each iteration's analysis and fix too

Every step with its status and timing, and the screenshots of failed and warned steps embedded in the file — so the report explains itself outside your terminal. --report-screenshots all embeds every screenshot instead, which is what you want when the report is meant to show that the product works rather than to explain a failure. A run whose steps all passed but produced a warning is titled PASS WITH WARNINGS, never PASS. Printing uses the Chromium that Playwright already installed; there is no PDF library to add.

What AWT Is Great At

Known Limitations (being honest)

AWT works well for me, but it has rough edges I haven't fully solved yet:

• OCR matching can misfire on custom fonts or low-contrast UIs — fallback to DOM selectors when that happens
• The self-healing loop occasionally fixates on the wrong element. Setting --max-attempts 3 helps avoid spinning wheels
• Complex SPAs with heavy animation may need --verbosity=detailed and a longer wait time
• First-run scenario generation is only as good as the page's accessibility tree — poorly structured HTML gives poor results

If you hit something broken, please open an issue. I fix reported bugs fast.

AWT vs Other Tools

vs Playwright / Cypress

Playwright and Cypress are excellent — and AWT is built on top of Playwright. The difference is who writes the tests:

Use Playwright/Cypress when you want full programmatic control. Use AWT when you want tests without writing them.

vs testRigor

vs Applitools

Applitools specializes in visual regression with AI-powered comparison. AWT now includes its own visual regression (SSIM-based, zero AI tokens) plus functional testing. For pixel-perfect comparison across browsers and viewports, Applitools is still more mature. But AWT covers both functional + visual in a single free tool.

Speed & Screenshot Modes

Control the trade-off between thoroughness and speed:

# CI/CD — fastest, minimal storage
aat run --verbosity=concise --screenshots=on-failure scenarios/

# Standard QA — balanced (recommended)
aat run --verbosity=concise --screenshots=before-after scenarios/

# Full audit — every step recorded
aat run --verbosity=detailed --screenshots=all scenarios/

Supported AI Providers

# aat.yaml
ai:
  provider: openai        # openai | anthropic | ollama
  model: gpt-4o
  api_key: ${OPENAI_API_KEY}

Architecture

aat devqa / aat run / aat snapshot / aat diff / aat watch
              │
              ▼
    ┌─────────────────────────────────────┐
    │           CLI (Typer)               │
    ├─────────────────────────────────────┤
    │         Core Orchestrator           │
    │  Executor · Comparator · DevQALoop  │
    ├────────────┬────────────┬───────────┤
    │   Engine   │  Matcher   │ AI Adapter│
    │ web/desktop│ ocr/cv/ai  │openai/etc.│
    ├────────────┴────────────┴───────────┤
    │  Visual Regression · Watch Mode     │
    │  Pydantic v2 Models · SQLite Learn  │
    └─────────────────────────────────────┘

All modules follow a plugin registry pattern — add a new engine, matcher, or AI provider by implementing one base class and registering it in __init__.py.

Development

Prerequisites

• Python 3.11+
• Tesseract OCR: brew install tesseract / apt install tesseract-ocr

Commands

git clone https://github.com/ksgisang/AI-Watch-Tester.git
cd AI-Watch-Tester
python -m venv .venv && source .venv/bin/activate
make dev
make test        # verify everything works
aat dashboard    # launch at http://localhost:9500

Contributing

Contributions, bug reports, and new plugins are welcome — see CONTRIBUTING.md.

If something doesn't work for your use case, please open an issue before spending time on a PR. Happy to help scope the right fix.

git checkout -b feat/my-feature
make format && make lint && make typecheck && make test
git commit -m "feat(scope): description"

FAQ

pip install aat-devqa
playwright install chromium
aat dashboard     # opens at http://localhost:9500

git clone https://github.com/ksgisang/AI-Watch-Tester.git
cd AI-Watch-Tester
make dev && aat dashboard

# Recommended for daily QA
aat run --verbosity=concise --screenshots=before-after scenarios/

# For CI/CD pipelines
aat run --verbosity=concise --screenshots=on-failure scenarios/

aat snapshot scenarios/ --url http://localhost:3000   # before change
# ... make code changes ...
aat diff scenarios/ --url http://localhost:3000        # compare

curl -X POST https://your-awt-server.com/api/v1/run \
  -H "X-API-Key: awt_your_key" \
  -H "Content-Type: application/json" \
  -d '{"target_url": "https://staging.example.com"}'

License

MIT — free for personal and commercial use.

_{Built with Playwright, OpenCV, and too much frustration with broken selectors.
Made by @ksgisang — feedback and issues very welcome.}