Python SDK for Visual Regression Tracker

Requires Python 3.10 or later. Use 5.x of this SDK with VRT 6.0; image URLs in results need VRT server 5.0.10 or later.

Install

pip install visual-regression-tracker

# or, with the Playwright integration
pip install "visual-regression-tracker[playwright]"
python -m playwright install

Configuration

The server compares images while it answers track(). With VLM checks or a busy comparison queue that can take minutes, so keep httpTimeoutInSeconds above 300.

A Config passed to the constructor is used as it is:

from visual_regression_tracker import VisualRegressionTracker, Config

vrt = VisualRegressionTracker(Config(
    apiUrl='http://localhost:4200',
    apiKey='tXZVHX0EA4YQM1MGDD',
    project='Default project',
    branchName='develop',
    ciBuildId='commit_sha',
))

Without one, the SDK reads vrt.json from the current working directory, then lets environment variables override it. Config.default(path) reads another file.

{
    "apiUrl": "http://localhost:4200",
    "apiKey": "tXZVHX0EA4YQM1MGDD",
    "project": "Default project",
    "branchName": "develop",
    "ciBuildId": "commit_sha",
    "enableSoftAssert": false,
    "httpTimeoutInSeconds": 360,
    "sourceBranch": "main"
}

VRT_APIURL="http://localhost:4200" \
VRT_APIKEY="tXZVHX0EA4YQM1MGDD" \
VRT_PROJECT="Default project" \
VRT_BRANCHNAME="develop" \
VRT_CIBUILDID="commit_sha" \
VRT_ENABLESOFTASSERT=true \
VRT_HTTPTIMEOUTINSECONDS=360 \
VRT_SOURCEBRANCH="main" \
    python -m pytest

Empty variables are ignored. VRT_ENABLESOFTASSERT is true for true or 1.

Usage

from visual_regression_tracker import VisualRegressionTracker, TestRun, IgnoreArea

with VisualRegressionTracker() as vrt:  # start() on enter, stop() on exit
    result = vrt.track(TestRun(
        name='Home page',
        imageBase64=image_base64,
    ))

Without the context manager, call build = vrt.start() (returns Build(id, projectId)) and vrt.stop().

TestRun fields:

Set exactly one of the image fields. Multipart avoids the server's JSON size limit (BODY_PARSER_JSON_LIMIT) and the base64 overhead, so prefer it for large screenshots.

Playwright

from playwright.sync_api import sync_playwright
from visual_regression_tracker.playwright import PlaywrightVisualRegressionTracker, PageTrackOptions

with sync_playwright() as playwright:
    browser_type = playwright.chromium
    page = browser_type.launch().new_page()
    page.goto('https://www.python.org/')

    with PlaywrightVisualRegressionTracker(browser_type) as vrt:
        vrt.trackPage(page, 'Home page')
        vrt.trackElementHandle(page.query_selector('#touchnav-wrapper'), 'Navigation')

The screenshot is sent as multipart. browser is the Playwright browser name and viewport the page's viewport size. The async API has trackPageAsync and trackElementHandleAsync.

The last argument is optional: PageTrackOptions for trackPage, ElementHandleTrackOptions for trackElementHandle. Both take:

• diffTollerancePercent, ignoreAreas, customTags, comment and merge, as in TestRun;
• agent: Agent(os, device, viewport): baseline fields Playwright doesn't know. viewport is used for element handles and for pages without a viewport size;
• screenshotOptions: arguments for Playwright's screenshot(). PageScreenshotOptions takes full_page, clip, omit_background, timeout, type, quality and path. ElementHandleScreenshotOptions takes the same, without full_page and clip.

Result and statuses

track and the Playwright methods return a TestRunResult:

• testRunResponse: id, status, url (the test run in the VRT app), diffPercent, diffTollerancePercent, pixelMisMatchCount, merge;
• imageUrl, diffUrl and baselineUrl: the images, under {apiUrl}/images/. diffUrl and baselineUrl are None when there is no diff or baseline.

With enableSoftAssert, track logs new and unresolved as errors instead of raising.

Errors

All errors derive from VisualRegressionTrackerError.

When the server answers 503 with a Retry-After of at most 60 seconds (the comparison queue is full, or the server is restarting), the SDK waits that long and sends the request again, up to 3 times, logging a warning each time. After that, or without a usable Retry-After, it raises ServerError.

Development

pip install -e '.[playwright]' -r requirements.txt
python -m playwright install chromium
python -m pytest

tox runs the tests on every supported Python.

Releasing

Merging to master releases. semantic-release reads the Conventional Commits since the last tag: fix:, perf: and revert: make a patch, feat: a minor, and ! after the type (feat!:) or a BREAKING CHANGE: footer a major release. Other types (chore:, ci:, docs:, ...) release nothing. A squash merge takes the PR title (or, for a single-commit PR, that commit's message) as its subject, so keep both conventional.

The Release workflow runs the tests, tags the commit (5.1.0, no v), publishes to PyPI and then publishes the GitHub release with the generated notes. The package version comes from the tag (setuptools-scm); nothing is committed back.

If publishing fails, the tag and a draft release already exist: re-run the failed jobs of that run (not the whole run) before anything else is merged.

Links

• Visual Regression Tracker: the server and how to run it
• Python examples
• Other SDKs and agents
• Licence: Apache-2.0