ui-blackbox-mcp 0.1.0

UI Blackbox Tester MCP — natural-language UI testing for Claude Desktop via Playwright

UI Blackbox Tester MCP

🌐 Language: English · 한국어

Test your UI in plain language — an MCP server. Give Claude Desktop (and other MCP clients) the ability to drive a browser. Say "check that the login flow works" and the agent opens a browser, clicks, types, asserts, and leaves a QA report (HTML/MD/JSON) behind — no test code required.

Auto-generated report — pass rate · per-step screenshots · failure cause · regression · accessibility · credential masking

Python 3.11+ · Playwright (Chromium, async) · official MCP SDK (FastMCP) · stdio · tests green in CI

---

✨ What's different (vs a generic browser MCP)

Plenty of tools can drive a browser. This one is built around the QA workflow.

	Generic browser MCP	UI Blackbox MCP
Authoring	Write selectors by hand	Natural language → kit → reusable scenario (generate_scenario)
Reuse	Start over each time	Save & load by name (scenario library)
Output	Text / logs	QA report: pass rate · step screenshots · AI failure cause + fix suggestion · regression diff · a11y findings · severity
Selector stability	Breaks every build	D2 priority chain (data-testid → role+name → text → css) + resolved_by transparency
Security	—	Credential masking (${VAR} injected from env, never written to reports)

→ Not a developer tool, but regression-test automation that non-developer QA/PMs can drive in natural language.

---

🚀 Quick start

1) Install

Option A — one-liner, no clone (recommended). uvx (Python's npx; ships with uv) fetches, isolates, and runs the server in one step — this is the whole Claude Desktop config, nothing else to set up:

{
  "mcpServers": {
    "ui-blackbox": {
      "command": "uvx",
      "args": ["--from", "git+https://github.com/WindowHyun/blackBoxTesting-MCP.git", "ui-blackbox-mcp"]
    }
  }
}

The same one-liner works for the CLI and for pipx/pip users:

uvx --from git+https://github.com/WindowHyun/blackBoxTesting-MCP.git ui-blackbox doctor
pipx install git+https://github.com/WindowHyun/blackBoxTesting-MCP.git   # or:
pip install git+https://github.com/WindowHyun/blackBoxTesting-MCP.git

Chromium auto-installs on the server's first run (or run playwright install chromium once). Once the package is published to PyPI, this shortens further to uvx ui-blackbox-mcp / pip install ui-blackbox-mcp — a release workflow is included (.github/workflows/release.yml, PyPI trusted publishing).

Option B — from a clone (development):

git clone https://github.com/WindowHyun/blackBoxTesting-MCP.git
cd blackBoxTesting-MCP

python -m venv .venv
.venv/bin/pip install -e .              # installs deps (mcp, playwright)
.venv/bin/playwright install chromium   # browser (first time). Skipped? the server auto-installs on first run

On networks where the browser CDN is blocked (corp/CI), point at a pre-installed binary with CHROMIUM_EXECUTABLE=/path/to/chrome.

Wherever a config below uses <ABS>, replace it with the absolute path of this repo, e.g. /home/you/blackBoxTesting-MCP. On Windows the interpreter is <ABS>\.venv\Scripts\python.exe. (With Option A, uvx replaces all of that.)

---

🔌 Client setup

The server speaks stdio, so every MCP client launches it the same way: run the venv Python with -m blackbox_mcp.server. Only the config file/format differs per client. Pick yours.

Claude Desktop

Edit the config file (create it if missing):

• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
• Windows: %APPDATA%\Claude\claude_desktop_config.json

{
  "mcpServers": {
    "ui-blackbox": {
      "command": "<ABS>/.venv/bin/python",
      "args": ["-m", "blackbox_mcp.server"],
      "env": {
        "HEADLESS": "true",
        "REPORT_DIR": "<ABS>/reports"
      }
    }
  }
}

Restart Claude Desktop. Then in chat: "Open https://example.com/login, test the login flow and make a report."

Claude Code / Claude CLI (same product — the claude CLI)

Easiest — one command (user scope, available in every project):

claude mcp add ui-blackbox \
  --scope user \
  --env HEADLESS=true \
  --env REPORT_DIR=<ABS>/reports \
  -- <ABS>/.venv/bin/python -m blackbox_mcp.server

Verify with claude mcp list → it should show ui-blackbox.

Or, to commit it alongside a repo, drop a project-scoped .mcp.json at the repo root:

{
  "mcpServers": {
    "ui-blackbox": {
      "command": "<ABS>/.venv/bin/python",
      "args": ["-m", "blackbox_mcp.server"],
      "env": { "HEADLESS": "true", "REPORT_DIR": "<ABS>/reports" }
    }
  }
}

Codex CLI

One command:

codex mcp add ui-blackbox \
  --env HEADLESS=true --env REPORT_DIR=<ABS>/reports \
  -- <ABS>/.venv/bin/python -m blackbox_mcp.server

Or edit ~/.codex/config.toml directly (TOML, not JSON):

[mcp_servers.ui-blackbox]
command = "<ABS>/.venv/bin/python"
args = ["-m", "blackbox_mcp.server"]

[mcp_servers.ui-blackbox.env]
HEADLESS = "true"
REPORT_DIR = "<ABS>/reports"

Gemini CLI

Edit ~/.gemini/settings.json:

{
  "mcpServers": {
    "ui-blackbox": {
      "command": "<ABS>/.venv/bin/python",
      "args": ["-m", "blackbox_mcp.server"],
      "env": { "HEADLESS": "true", "REPORT_DIR": "<ABS>/reports" },
      "timeout": 30000
    }
  }
}

Values can reference shell env vars as $VAR / ${VAR}. Run /mcp inside Gemini CLI to confirm the server connected.

Google Antigravity (IDE & CLI)

Antigravity (IDE + CLI) share one MCP config: ~/.gemini/config/mcp_config.json. In the IDE you can open it via the ⋯ menu at the top of the agent panel → MCP Servers → Manage MCP Servers → View raw config.

{
  "mcpServers": {
    "ui-blackbox": {
      "command": "<ABS>/.venv/bin/python",
      "args": ["-m", "blackbox_mcp.server"],
      "env": { "HEADLESS": "true", "REPORT_DIR": "<ABS>/reports" }
    }
  }
}

Save the file and Antigravity reloads the server automatically.

Tips that apply to every client. Use the venv Python absolute path for command (avoids system-Python dependency clashes). The MCP server's cwd may be a system path, so set REPORT_DIR to an absolute path or reports fall back to ~/ui-blackbox/reports. Slash commands (below) are Claude-specific; other clients drive the same tools via natural language.

---

⌨️ Slash commands (Claude — recommended, avoids clashing with other browser tools)

Type / in Claude to see these. Each instructs the agent to use only the ui-blackbox tools, so another browser tool (e.g. "Claude in Chrome") can't hijack the request.

Command	Args	Purpose
/ui-test	task	Run a natural-language task with the ui-blackbox tools
/ui-scenario	description, url	Build → run → report (all formats)
/ui-login	task, url	Switch to real Chrome (persistent login) then test a site that needs auth
/ui-generate	description, url, name	Analyze a page → generate & save a reusable scenario
/ui-sync	name, url	Change detection: diff a saved scenario against the current page, update & re-run

Example: /ui-test → open example.com, click the login button, take a screenshot

💬 Usage examples (natural language)

• "On this page, check that the signup form shows an error when fields are empty."
• "Make a login scenario and save it as 'smoke_login'." → later: "run smoke_login."
• "Compared to yesterday, what broke in that last test?" (regression)
• "Were there any console errors or 4xx responses?"

---

🧰 MCP Tools (25)

Group	Tools
Core	navigate · snapshot (a11y/dom) · screenshot · interact · assert_ · get_console_logs · get_network_errors
Extended	wait · switch_frame · expect_dialog · reset_session · use_real_browser · dismiss_banners · status
Auth state	save_state · load_state · list_states — export login (cookies+localStorage) once, reuse headless/in CI, swap roles
Network mock	mock_route · unmock_route — deterministic offline responses for flaky/unbuilt APIs
Scenario & report	run_scenario (incl. trace_on_failure) · generate_scenario · save_report
Library	save_scenario · load_scenario · list_scenarios

Every test flow ends with a report. Ad-hoc tool calls (navigate/interact/assert…) are recorded automatically, and a final save_report writes the JSON/MD/HTML report (slash commands instruct this automatically). run_scenario saves its own report.

Adding a tool = 1 file in tools/ + 1 import line. server.py is never touched.

---

🖥️ CLI / CI (no MCP client needed)

Scenarios you built in chat can be replayed headlessly in a pipeline — same runner, same reports, plus an exit code and JUnit XML:

ui-blackbox run smoke_login                     # library scenario → exit 0/1
ui-blackbox run ./steps.json --format all       # a steps .json file
ui-blackbox run a b c --junit results.xml       # suite + JUnit for CI
ui-blackbox run a b c --parallel 3              # one isolated subprocess each
ui-blackbox run a b c --parallel 3 --timeout 300  # per-scenario watchdog (sec)
ui-blackbox run smoke --trace-on-failure        # keep a Playwright trace.zip only on failure
ui-blackbox doctor                              # browser/dirs/config self-check

GitHub Actions sketch:

- run: pip install . && playwright install chromium
- run: ui-blackbox run smoke_login --junit results.xml
- uses: actions/upload-artifact@v4
  with: { name: ui-reports, path: ~/ui-blackbox/reports }

Exit codes: 0 all passed · 1 a step failed · 2 usage/infra error. In chat, the status tool reports version/mode/liveness for debugging.

Good to know:

• navigate fails on HTTP ≥ 400 (a 500/404 is not a green step). Add "expect_status": 404 to a step to assert an error page on purpose.
• Failure reason/suggestion in a CLI report are rule-based hints; in chat, Claude (the host LLM) enriches them with real analysis.
• The server is single-tenant — one browser/session per process. True parallelism is via --parallel (one subprocess per scenario), not a shared server.

---

🧪 Reports

See examples/ for real output (open sample_report.html in a browser). A single self-contained HTML (screenshots inlined as base64, zero external deps) — per-step results · failure screenshots · AI fix suggestions · regression (vs the previous run) · a11y findings · environment metadata · masking badges.

---

🏗️ Structure

blackbox_mcp/
  server.py        # FastMCP boot + ensure_chromium + lifespan + register_all
  bootstrap.py     # Chromium auto-install (D1)
  config.py        # environment variables
  browser/         # session singleton · event buffers · D2 selector chain
  testing/         # scenario runner · report (JSON/MD/HTML) · library · masking
  tools/           # MCP tool = 1 file (registry auto-registers)

Design: DESIGN.md · Milestones: ROADMAP.md · Run playbook: HARNESS.md · Agent context: CLAUDE.md

---

🔧 Development

.venv/bin/pip install -e ".[dev]"
.venv/bin/python -m pytest -q                    # full suite (unit + file:// integration + E2E)
.venv/bin/python -m pytest -q -m "not browser"   # fast unit lane, no browser needed

⚙️ Environment variables

HEADLESS (default true) · BROWSER (chromium) · CHROMIUM_EXECUTABLE · BROWSER_CHANNEL (chrome/msedge — real browser) · BROWSER_CDP (attach to a running browser) · STEALTH (reduce bot false-positives) · REPORT_DIR (default /ui-blackbox/reports) · SCENARIO_DIR (/ui-blackbox/scenarios) · SELECTOR_TIMEOUT_MS (2000) · DEFAULT_WAIT_UNTIL (networkidle) · NAV_TIMEOUT_MS (30000) · IGNORE_HTTPS_ERRORS (false) · REPORT_RETENTION (keep newest N runs, default 100, 0=unlimited). Details in .env.example.

Testing live/deployed sites. ① Ad/polling-heavy sites may never reach networkidle — navigate proceeds on timeout (settled:false), and DEFAULT_WAIT_UNTIL=domcontentloaded is faster. ② For slow-appearing elements, raise SELECTOR_TIMEOUT_MS to 5000–10000. ③ Filter ad/tracker 4xx noise with get_network_errors(same_origin=True). ④ Cookie-consent banners: dismiss_banners (auto-suggested when a click is blocked). ⑤ Login/bot-walls: use_real_browser. ⑥ Staging certs: IGNORE_HTTPS_ERRORS=true. ⑦ New tabs/popups are tracked automatically (the session follows a click that opens a new window and returns to the original tab when the popup closes — e.g. OAuth popups).

About bot detection. This tool targets your own UI / staging. Third-party sites may block automation with anti-bot measures, and bypassing those for login automation can violate their terms of service. False-positives on legitimate tests can be reduced with BROWSER_CHANNEL=chrome + STEALTH=true.

🔗 Sites that require login — real browser (no manual setup)

Recommended (automatic): say /ui-login or "log in with a real browser and …" and the agent calls the use_real_browser tool to launch real Chrome with a persistent profile. Log in once, by hand, in that window; the profile is saved to ~/ui-blackbox/chrome-profile and reused on later runs. Far less likely to trip bot detection than the bundled headless browser.

If real Chrome isn't found, it falls back to the bundled browser automatically. Pin a channel with BROWSER_CHANNEL=msedge, etc.

Advanced (manual CDP): to attach to a Chrome you already have open, launch it with a debug port and set BROWSER_CDP:

chrome --remote-debugging-port=9222 --user-data-dir="C:\cdp-profile"   # log in there

config env: "BROWSER_CDP": "http://localhost:9222" → attach. Closing the session leaves your browser open.