visualcheck 0.3.16

Visual regression checking with figma-first baselines, pixel+SSIM diffing, ignore regions, and HTML reports.

visualcheck

Visual regression checking with:

• Figma-first baselines (optionally synced via Figma API)
• Runtime baselines (auto-create when missing; never overwrite by default)
• Pixel diff % + SSIM (with optional resize-on-mismatch + WARN)
• Ignore regions (mask dynamic areas via selectors + explicit rects)
• Self-contained HTML report + report.json

Install

pip install visualcheck
playwright install chromium

Config (visualcheck.yaml)

project: consumer_website
suite: daily_sanity

envs:
  prod: "https://example.com"

views:
  desktop_profiles: ["desktop_1440x900", "macbook_1440x900"]
  mobile_devices: ["iPhone 15 Pro Max", "iPhone 13 mini", "Pixel 7"]

pages:
  - id: home
    url: "/"
    wait_for: "body"
    full_page: true

# Optional flows (multi-step user journeys)
flows:
  - id: search_flow
    start_url: "/"
    steps:
      - action: click
        selector: "text=Search"
      - action: wait
        ms: 500
    snapshots:
      - id: after_search
        wait_for: "body"
        full_page: true

baseline:
  # Locked resolution order:
  # figma -> runtime -> create runtime baseline (if enabled)
  priority: ["figma", "runtime"]
  create_if_missing: true
  never_overwrite: true
  on_created: "INFO"  # INFO|WARN|FAIL
  # Optional legacy mode: copy figma image into runtime when figma is used.
  # Default behavior keeps runtime as website-captured truth.
  seed_runtime_from_figma: false

compare:
  resize_on_mismatch: true
  mismatch_level: "WARN"  # WARN|FAIL
  thresholds:
    max_pixel_diff_pct: 0.10
    min_ssim: 0.995
  # Optional per-snapshot override + region-level checks
  by_snapshot:
    home:
      max_pixel_diff_pct: 0.15
      min_ssim: 0.992
      regions:
        - name: header_strict
          x: 0
          y: 0
          width: 1440
          height: 120
          max_pixel_diff_pct: 0.03
          min_ssim: 0.998
  # Optional ignore-region suggestions from diff output
  suggest_ignore_regions:
    enabled: true
    min_area: 2500
    max_regions: 5

capture:
  # prefer Chrome channel (fallback to bundled Chromium if unavailable)
  browser_channel: "chrome"
  extra_wait_ms: 1000
  scrolls: 5
  scroll_delay_ms: 700
  retries: 1
  retry_delay_ms: 700
  # Optional page stabilization before screenshot
  stability_checks: 2
  stability_interval_ms: 250
  stability_timeout_ms: 3000
  # Strong load gate: wait for document complete + network idle best-effort
  # + images/lazy-images settled before screenshot.
  wait_for_full_load: true
  full_load_timeout_ms: 120000
  full_load_check_interval_ms: 500
  full_load_stable_checks: 2
  # Additional lazy-load settling rounds (scroll to bottom and settle).
  lazy_load_scroll_rounds: 6
  screenshot_timeout_ms: 90000

stability:
  # auto-rerun unstable snapshots and choose final outcome by consensus
  reruns_on_warn: 1
  reruns_on_fail: 2
  require_consensus: "best"   # best|median|majority

quarantine:
  snapshots:
    - "search_flow__after_search"

owners:
  by_snapshot:
    home: "web-platform-team"
    "desktop_1440x900:pricing": "checkout-team"

ignore_regions:
  global:
    selectors: ["#cookie-banner", ".chat-widget"]
  by_snapshot:
    home:
      rects:
        - {x: 0, y: 0, width: 300, height: 120}

# Optional Figma sync (writes into visual_baseline/<project>/<suite>/figma/...)
# figma:
#   token_env: FIGMA_TOKEN
#   file_key: "<FIGMA_FILE_KEY>"
#   frames:
#     - id: home
#       node_id: "123:456"
#       view_id: "desktop_1440x900"   # optional

Commands

Run full check

visualcheck run --env prod
# force-refresh matched figma baselines during this run:
visualcheck run --env prod --force-reset-figma-baseline
# skip browser auto-open:
visualcheck run --env prod --no-open-report

Programmatic auto-open from .env (non-CLI)

VISUALCHECK_OPEN_REPORT=true

• When enabled, run_visualcheck(...) and VisualCheck.finalize() automatically open report.html in your default browser.
• CLI already has its own --open-report/--no-open-report option, so env auto-open is ignored there to avoid double-opening tabs.

Outputs:

• Baselines:
  • visual_baseline/<project>/<suite>/figma/<view>/<snapshot>.png
  • visual_baseline/<project>/<suite>/runtime/<view>/<snapshot>.png
• Run artifacts:
  • test_report/<project>/dashboard.html (main dashboard; latest entry point)
  • test_report/<project>/visual_runs/<run_id>/current/...
  • test_report/<project>/visual_runs/<run_id>/diff/...
  • test_report/<project>/visual_runs/<run_id>/report/report.html
  • test_report/<project>/visual_runs/<run_id>/report/report.json
  • test_report/<project>/visual_runs/<run_id>/report/report.junit.xml

Terminal logs now include user-friendly lifecycle messages, for example:

• run started ...
• capture started/completed ...
• no baseline found, creating runtime baseline from current screenshot
• using runtime baseline ... / using cached figma baseline ...
• per-item final status with pixel diff and SSIM
• final report + dashboard paths

Capture only

visualcheck capture --env prod --out current

Diff only

visualcheck diff --baseline visual_baseline/myproj/mysuite/runtime --current current --out report

Sync Figma baselines

export FIGMA_TOKEN="..."
visualcheck figma-sync
# force refresh selected frames:
visualcheck figma-sync --force --only home,pricing
# auto-map page/flow snapshot ids to figma frame names:
visualcheck figma-sync --match-by-name

Compare strictly against Figma baseline

visualcheck figma-diff --env prod

Approve current run as runtime baseline (explicit)

visualcheck approve --env prod --run-id 20260207_235500
# Overwrite existing runtime baselines only with:
visualcheck approve --env prod --run-id 20260207_235500 --force

Validate config

visualcheck validate-config --config visualcheck.yaml

Cleanup old run artifacts

visualcheck cleanup --config visualcheck.yaml --keep-last 20 --dry-run
visualcheck cleanup --config visualcheck.yaml --keep-last 20

Quarantine management

visualcheck quarantine-list --config visualcheck.yaml
visualcheck quarantine-add --config visualcheck.yaml --snapshot search_flow__after_search
visualcheck quarantine-remove --config visualcheck.yaml --snapshot search_flow__after_search

Baseline rules (locked)

For each snapshot + view:

1. If a Figma baseline exists → use it
   • visualcheck will attempt a best-effort figma-sync when figma is listed in baseline.priority so remote Figma baselines are consulted before falling back.
2. Else if a runtime baseline exists → use it
3. Else → capture and create runtime baseline (controlled by baseline.create_if_missing)

Notes on configuration and behavior

• baseline.priority controls the preference order. Example: priority: ["figma","runtime"] (default in examples) — visualcheck will try Figma first.
• By default a Figma baseline is used directly for comparison (reported baseline path will point at .../figma/...).
• By default, Figma does not auto-create runtime baseline copies. Runtime remains website-captured truth (created from current captures when missing).
• If you want legacy behavior (copy Figma into runtime), set:

baseline:
  seed_runtime_from_figma: true

• If you want Figma to become the authoritative runtime baseline (copied into the runtime/ folder and then used), set:

baseline:
  figma_authoritative: true

• This avoids accidental comparisons against an old runtime image when Figma is the source of truth.
• If a figma baseline is missing locally and figma appears in priority, visualcheck will attempt to fetch it via the Figma API (best-effort). If the fetch fails (token/permission), it falls back to runtime per priority.
• If a matching local figma baseline already exists for snapshot+view, visualcheck does not call the Figma API again.
• Optional forced refresh from config:

baseline:
  force_reset_figma_baseline: true

This is equivalent to CLI flag --force-reset-figma-baseline for a run.

Figma metadata in report

• When a snapshot resolves to a Figma frame, report rows include figma.file_key, figma.node_id, and a direct figma.frame_url.

Report upgrades

• HTML report includes status summary cards, client-side status filters, top regressions, and an expanded split layout for faster triage.
• Each result row now provides a larger visual workspace, hover zoom cue, and one-click popup viewer with pan/zoom controls.
• Left-side mismatch region items are clickable and auto-focus the same coordinates on the diff/compare canvas.
• Auto-detected diff hotspots are listed as separate clickable lines with per-hotspot diff% and ssim.
• Mismatch clicks and image popups now prioritize the Diff layer so users see exact issue regions first.
• View mode switching is built in (baseline/current compare, diff, current, baseline) so users can inspect the exact failure context quickly.
• Slider canvas opens in full-image fit by default; users can then zoom and scroll in the same canvas.
• Report now includes a Recent Run History table (status, commit, branch, counts).
• Main dashboard (test_report/<project>/dashboard.html) lists latest runs and links to individual reports.
• Rows include plain-language issue summary and recommended action to make pixel triage faster.
• Report includes flake score, root-cause hint, snapshot owner, quarantine status, and likely-introduced-after commit hint.
• History files are stored under test_report/<project>/history/ for trends, flake scoring, and change correlation.

Use from code (framework integration)

If you already navigate with Playwright/Selenium in your own framework and just want visualcheck to capture + baseline + diff + report, use the code API.

Baselines still go to visual_baseline/<project>/<suite>/... and the HTML report goes to test_report/<project>/visual_runs/<run_id>/report/report.html.

Playwright (sync) example

from playwright.sync_api import sync_playwright

# NOTE: API object names may evolve; refer to the package docs in case of changes.
from visualcheck.api import VisualCheck

vc = VisualCheck(
    project="my_project",
    suite="daily_sanity",
    env="prod",
    base_url="https://example.com",
    view_id="desktop_1440x900",
    baseline_priority=["figma", "runtime"],
    create_if_missing=True,
    ignore_selectors=["#cookie-banner", ".chat-widget"],
)

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page(viewport={"width": 1440, "height": 900})

    page.goto("https://example.com/", wait_until="domcontentloaded")
    vc.check(snapshot_id="home", page=page)

    page.goto("https://example.com/pricing", wait_until="domcontentloaded")
    vc.check(snapshot_id="pricing", page=page)

    vc.finalize()
    print("Report:", vc.report_html)

    browser.close()

Selenium example

from selenium import webdriver
from visualcheck.api import VisualCheck

vc = VisualCheck(
    project="my_project",
    suite="daily_sanity",
    env="prod",
    base_url="https://example.com",
    view_id="desktop_1440x900",
)

driver = webdriver.Chrome()
driver.set_window_size(1440, 900)

driver.get("https://example.com/")
vc.check(snapshot_id="home", selenium_driver=driver)

driver.get("https://example.com/pricing")
vc.check(snapshot_id="pricing", selenium_driver=driver)

vc.finalize()
print("Report:", vc.report_html)

driver.quit()

Notes

• Ignore regions are masked with a solid color before diffing.
• If screenshot sizes differ and compare.resize_on_mismatch=true, current is resized to baseline size and a warning is recorded.

License

MIT

Library & CLI usage (examples)

Programmatic API (Python)

The package exposes a simple programmatic API to integrate with Playwright or Selenium.

Example (Playwright):

from visualcheck.runner import run_visualcheck
from visualcheck.config import load_config

cfg = load_config(Path('examples/wakefit.yaml'))
result = run_visualcheck(cfg, env='prod', headed=False)
print('Report saved at', result.get('report_html'))

CLI usage

• Sync Figma baselines:

export FIGMA_TOKEN="<your_token>"
visualcheck figma-sync --config examples/wakefit.yaml

• Run full check:

visualcheck run --env prod --config examples/wakefit.yaml

Notes on FIGMA token & env handling

• The library reads Figma token from the environment variable name specified in config (defaults to FIGMA_TOKEN).
• Keep secrets out of git; use host-level secrets (e.g. ~/.openclaw/secrets.env) or a CI secret store.