qa-screens

An MCP server for visual QA during AI refactoring. It captures pages with Playwright, compares them with reference screenshots using SSIM, and returns the score, the changed regions and a labelled REFERENCE | LIVE preview image, so the agent can see what broke and fix it.

• Design conformance: compare the running site with golden or Figma screenshots.
• Refactor safety: capture before and after a change and prove it is pixel-neutral.
• A/B: compare two running servers, such as main and a worktree.
• Apps behind a login: OAuth (client credentials, password, refresh, authorization code + PKCE), bearer tokens, cookies, sessions, localStorage/sessionStorage and HTTP basic auth.
• Self-reporting: significant errors and crashes are filed as GitHub issues automatically, with secrets scrubbed. See Error reporting.

Install

pip install qa-screens          # or: uv tool install qa-screens

Chromium is downloaded automatically on first use. To do it up front, run python -m playwright install chromium.

Claude Code

claude mcp add qa-screens -- uvx qa-screens

or in a project's .mcp.json:

{
  "mcpServers": {
    "qa-screens": { "command": "uvx", "args": ["qa-screens"] }
  }
}

To run the latest main without PyPI, use uvx --from git+https://github.com/astuanax/qa-screens qa-screens.

Copy skills/qa-screenshots into .claude/skills/ so the agent runs QA on its own after every UI change.

Other MCP clients

The server speaks MCP over stdio. The command is qa-screens (or python -m qa_screens), and it runs in the project directory, or wherever QA_SCREENS_ROOT points.

Configure

Put a .qa-screens.json in the project root. Every key is optional:

{
  "base_url": "http://localhost:8080",
  "references_dir": "qa/screenshots",
  "route_template": "/nl-be/{name}/",
  "routes": { "home": "/", "nl-be": "/nl-be/" },
  "threshold": 0.9,
  "mobile_device_scale_factor": 1,
  "mask_selectors": [".carousel", "[data-testid=clock]"],
  "default_profile": null
}

The environment variables QA_SCREENS_BASE_URL, QA_SCREENS_REFERENCES_DIR, QA_SCREENS_ROUTE_TEMPLATE, QA_SCREENS_THRESHOLD and QA_SCREENS_PROFILE override the file.

Tools

Page results are PASS, FAIL or ERROR. ERROR means the comparison is meaningless: an HTTP 4xx/5xx, a navigation failure or a missing reference. The agent should fix the environment, not the CSS.

Authentication for apps

A profile is a named session that is applied to every browser context that uses it:

auth_update_profile(
  name="staging",
  origins=["https://app.staging.example.com"],
  oauth={"grant_type": "client_credentials",
         "token_url": "https://idp.example.com/oauth/token",
         "client_id": "qa-bot", "client_secret": "env:QA_CLIENT_SECRET",
         "audience": "https://api.example.com"},
  apply_token_as=["header", "local_storage:access_token"])

run_qa(profile="staging", base_url="https://app.staging.example.com")

• Secrets: any value can be "env:VAR". It is resolved from the server's environment at use time, so it never passes through the AI conversation.
• Scoping: headers and tokens go only to the profile's origins (default: the base URL's origin), never to CDNs or third parties. Cookies follow normal browser rules.
• Token placement (apply_token_as): header (Authorization: Bearer …), header:X-Api-Key, local_storage:<key>, session_storage:<key>, local_storage_json:<key> (the whole token object), cookie:<name>.
• Refresh: expired tokens are refreshed through refresh_token or by re-running the client-credentials/password grant, including mid-session in the long-lived server.
• Storage: profiles live in ~/.local/state/qa-screens/profiles/ with mode 0600, outside the project, so they are never committed. QA_SCREENS_STATE_DIR moves the location.

Error reporting

qa-screens reports its own bugs, so they reach the maintainers without a manual bug report.

• Significant errors (unexpected exceptions in a tool, not user errors like a missing reference or bad credentials) are written to ~/.local/state/qa-screens/reports/pending/ and posted in the background as a GitHub issue on astuanax/qa-screens.
• Crashes: uncaught exceptions are logged to a crash file, and hard crashes (segfault, abort) are captured by faulthandler. On the next start-up the server scans for these files and posts them.
• A report stays pending until it has been posted, so reports made offline or without a token are sent later.
• De-duplication: each error has a stable fingerprint. A repeat within 24h is not re-posted, and an open issue with the same fingerprint gets a comment instead of a new issue. At most 10 posts per day.
• Privacy: tokens, JWTs, cookies, passwords, URL query strings and credentials, and the home directory path are redacted. Tool arguments that hold secrets are dropped entirely.

Without a token nothing is posted; reports wait in pending/. qa-screens reports --flush posts them by hand.

CLI (CI-friendly)

qa-screens                       # MCP server over stdio (same as `qa-screens serve`)
qa-screens run                   # QA every reference; exit 1 on FAIL/ERROR
qa-screens run nl-be faq --viewport desktop --base-url http://localhost:8080 --json
qa-screens reports --flush       # post pending error reports

A report is written to .qa-screens/runtime/reports/latest.json on every run.

How it works

1. The reference name maps to a URL (route_template / routes). Names ending in -mobile use a mobile context (390px, touch, mobile UA).
2. The page is loaded with the HTTP cache disabled (so the CSS you just edited is what gets measured), with animations, transitions and scrollbars disabled, and after document.fonts.ready. It is captured full-page.
3. SSIM is computed on grayscale images. Captures above 40MP are decoded at reduced resolution, so long pages never run out of memory. Different sizes are handled by align: resize (legacy default), crop or pad (height changes count as differences).
4. Changed pixels are grouped into region boxes. A red heatmap and a cropped side-by-side preview are written, and the preview is returned to the agent as an image.

Development

uv venv && uv pip install -e ".[dev]"
python -m playwright install chromium
pytest

Releases are published to PyPI by GitHub Actions when a v* tag is pushed (trusted publishing).

License

MIT