steve-qa-tester 1.0.0

Steve: Visual QA Agent — Figma vs Built comparison engine

Steve — Visual QA Agent

Steve (Systematic Testing & Evaluation Verification Engine) compares your live web build against a Figma design frame and reports exactly what doesn't match — at the pixel, property, and semantic level.

---

How It Works

Figma Node (design source of truth)          Your Live URL
        │                                         │
        ▼ Stage 1                                 ▼ Stage 2
 ┌─────────────────┐                    ┌──────────────────────┐
 │  figma-developer │                    │   Playwright MCP     │
 │  -mcp (PAT auth) │                    │   (browser capture)  │
 │                  │                    │                      │
 │  • Rendered PNG  │                    │  • Full-page load    │
 │  • Property tree │                    │  • Selector wait     │
 │    (colors,fonts,│                    │  • PNG screenshot    │
 │     sizes, etc.) │                    │  • Computed CSS      │
 └────────┬─────────┘                    └──────────┬───────────┘
          │                                         │
          └────────────────┬────────────────────────┘
                           ▼ Stage 3
              ┌────────────────────────────┐
              │      Three-Layer Diff      │
              ├────────────────────────────┤
              │ 1. Pixel Diff             │
              │ 2. Property Diff (CIEDE2000│
              │    color delta, font/size) │
              │ 3. Vision Diff (Gemini)   │
              └────────────┬───────────────┘
                           ▼ Stage 4
                   Merged Findings List
                   (severity + fix hint)

• Figma capture — uses figma-developer-mcp over stdio with your Personal Access Token. No OAuth, no rate-limit issues.
• Browser capture — uses @playwright/mcp to navigate, wait for full DOM load, screenshot, and read computed CSS.
• Pixel diff — raw pixel-level drift scanning via pixelmatch.
• Property diff — font size, weight, border radius, padding, and perceptually uniform CIEDE2000 color delta.
• Vision diff — multimodal Gemini 2.5 Flash compares both screenshots and lists semantic discrepancies.
• Reporter — deduplicates findings, assigns severity (High / Medium / Low), optionally files GitHub issues.

---

Prerequisites

Requirement	Scope/Role	Where to Get
Python 3.11+	Main engine runtime	python.org
Node.js 18+	Runs Figma & Playwright MCP drivers	nodejs.org
Figma Access Token	Reads design specs & captures assets	Figma Settings
Gemini API Key	Compares mock vs actual using Vision AI	Google AI Studio
GitHub Token (optional)	Automatically creates visual QA issue reports	GitHub Developer Settings

---

How to Get Your API Credentials

Follow these steps to obtain the necessary credentials before running Steve locally:

1. Getting your Figma Personal Access Token

Steve uses a Personal Access Token (PAT) to connect to your design files:

1. Open Figma in your browser or desktop app.
2. Click your profile icon/avatar in the top-left corner and open Settings.
3. Select the Security tab.
4. Scroll down to the Personal Access Tokens section.
5. Click Generate new token, enter a name (e.g., Steve Visual QA), and check the following scope:
   • File content (Read) - required to read nodes & download images.
6. Copy the generated token immediately and save it in your .env file under FIGMA_ACCESS_TOKEN.

2. Getting your Gemini API Key

Steve uses Gemini 2.5 Flash to perform intelligent, semantic layout comparison:

1. Navigate to Google AI Studio.
2. Log in with your Google Account.
3. Click Get API Key in the top-left sidebar.
4. Click Create API Key, select or create a project, and copy the new key.
5. Save this key in your .env file under GEMINI_API_KEY.

3. Getting your GitHub Personal Access Token (Optional)

If you want Steve to automatically file bug report tickets for visual differences:

1. Go to your GitHub account and open Settings → Developer Settings.
2. Select Personal Access Tokens → Tokens (classic).
3. Click Generate new token (classic), name it, and check:
   • repo (Full control of private & public repositories).
4. Save this token in your .env file under GITHUB_TOKEN and specify your target repository under STEVE_GITHUB_REPO=owner/name.

---

Setup

1. Clone and enter the project:

git clone https://github.com/your-org/Steve-QA-Tester.git
cd Steve-QA-Tester

2. Create your .env file:

cp .env.example .env

Fill in .env:

FIGMA_ACCESS_TOKEN=your_figma_personal_access_token
GEMINI_API_KEY=your_gemini_api_key

# Optional — for filing GitHub tickets
GITHUB_TOKEN=your_github_token
STEVE_GITHUB_REPO=owner/repo

3. Install Python dependencies:

pip install -r requirements.txt
pip install -e .

The pip install -e . registers the steve CLI command globally in your Python environment.

4. Install Playwright browsers (one-time):

npx playwright install chromium

figma-developer-mcp and @playwright/mcp are installed automatically via npx -y on first run — no manual npm install needed.

---

Run Command

steve run --url <YOUR_BUILD_URL> --figma <FILE_KEY>:<NODE_ID> --selector "<CSS_SELECTOR>"

Example:

steve run \
  --url https://staging.yourapp.com/checkout \
  --figma i6V4j6XDmc6QL6796C4uMu:308-939 \
  --selector "#checkout-btn"

You can also paste a full Figma URL:

steve run \
  --url https://staging.yourapp.com \
  --figma "https://www.figma.com/design/i6V4j6XDmc6QL6796C4uMu/App?node-id=308-939" \
  --selector "body"

---

All CLI Options

steve run [OPTIONS]

  --url        URL      Target build URL to screenshot (required)
  --figma      REF      Figma reference: 'file_key:node_id' or full Figma URL (required)
  --selector   STR      CSS selector of the element to compare (default: body)
  --source     URL      GitHub issue URL as source context (optional)
  --dry-run             Skip ticket filing, save reports locally only
  --no-cache            Bypass local Figma cache and force a fresh API fetch

steve replay <RUN_ID>   Replay and print the summary of a past run

steve --help            Show all options

---

Output

On each run Steve prints:

┌─ 🎨 Visual Agent Status ───────────────────┐
│ ✓ Completed Visual check. Findings: 3      │
└─────────────────────────────────────────────┘

🔍 Visual Findings:
  1. [🔴 HIGH] #checkout-btn — background_color_mismatch
       Expected : rgb(0, 122, 255)
       Actual   : rgb(255, 0, 0)
       Fix      : Set background-color to #007AFF

  2. [🟡 MEDIUM] #checkout-btn — font_size_mismatch
       Expected : 16px
       Actual   : 14px
       Fix      : Set font-size to 16px

┌─ Summary ───────────────────────────────────┐
│ 🔴 HIGH    1   visual issues                │
│ 🟡 MEDIUM  2   visual issues                │
│ 🟢 LOW     0   visual issues                │
└─────────────────────────────────────────────┘

Run artifacts are saved to runs/<run_id>/:

• visual_findings.json — structured findings
• expected.png — Figma design screenshot
• actual.png — live page screenshot
• diff.png — pixel diff overlay
• orchestrator_summary.json — full run summary

---

Project Structure

steve/
├── visual_agent/       Core comparison pipeline
│   └── agent.py
├── reporter_agent/     Deduplicates findings, files tickets
│   └── agent.py
├── orchestrator/       Routes and drives the visual run
│   └── agent.py
├── mcp_client.py       Figma MCP + Playwright MCP clients
├── config.py           Environment configuration
├── cli.py              CLI entry point
└── llm_client.py       Groq LLM client (optional enrichment)

---

Security

• Read-only — Steve never writes to your Figma files or codebase.
• Prompt-injection safe — Vision model instructions treat screenshots as pure visual data only.
• No hardcoded secrets — all credentials are loaded from .env, never committed.

---

License

MIT © 2026 Steve QA Tester