PyVisionAuto

PyVisionAuto is an end-to-end desktop automation toolkit. It is centered on visual image matching and also includes screen recording, mouse automation, and keyboard automation capabilities.

Note: This package is currently developed mainly for internal test automation needs. It is still evolving and not fully polished yet. We appreciate your understanding.

Runtime screenshot

Highlighted match region during runtime:

Scope

• Linux (X11 session) and Windows
• Real physical display required

Install

pip install pyvisionauto

Main APIs

Screen

• Screen.find(image, confidence=None, region=None) → MatchHandle | None
• Screen.wait(image, timeout, confidence=None, poll=None, region=None) → MatchHandle
• Screen.click(image, timeout=10, confidence=None, poll=None, highlight=False, region=None) → MatchHandle
• Screen.wait_vanish(image, timeout, confidence=None, poll=None, region=None, strict=True) → bool
• Screen.click_and_wait_vanish(click_image, vanish_image=None, timeout=10, ...) → bool
• Screen.activate_window(title_substring) → bool
• Screen.region(x, y, w, h) — returns a RegionScreen that scopes all operations to a sub-area
• Screen.check_env(strict=True) — validates platform, display, and required tools

MatchHandle (fluent chaining)

• .highlight(duration=None, color=None, thickness=None, show_center=False) → MatchHandle
• .click(button="left", highlight=False) → MatchHandle
• .double_click(highlight=False) → MatchHandle
• .right_click(highlight=False) → MatchHandle
• .hover(highlight=True, duration=None) → MatchHandle — moves pointer; duration holds it in place (seconds)
• .wait_vanish(timeout, confidence=None, poll=None, strict=True) → bool
• .center — (x, y) absolute screen coordinates of the match center
• .score — normalized match confidence [0.0, 1.0]

Input

• Input.type_text(text, human_like=None, delay_min=None, delay_max=None)
• Input.press(key, human_like=None)
• Input.hotkey(*keys, human_like=None)
• Input.click(x, y, button="left")
• Input.move_to(x, y) / Input.hover(x, y)
• Input.clear_text() — Ctrl+A then Backspace

Recorder

• Recorder.start(output_file, fps=15)
• Recorder.stop()

System dependencies

Linux

• python3-tk — Required for border overlay highlight
• xdotool — Preferred for window activation
• wmctrl — Fallback for window activation
• ffmpeg — Required for screen recording; install via sudo apt install ffmpeg

Windows

• tkinter — Bundled with most Python installations
• ffmpeg — Required for screen recording; download from ffmpeg.org, extract archive, and add the bin folder to system PATH

Verify ffmpeg installation

# Check if ffmpeg is installed and accessible
ffmpeg -version

Note: Screen recording (via Recorder API) requires ffmpeg. On Linux, it uses x11grab codec; on Windows, it uses gdigrab codec. Both are built into ffmpeg by default.

Quick start

Basic usage: Find and click

from pyvisionauto import Screen

screen = Screen()
# Wait for image to appear on screen, highlight it, then click
screen.wait("login_button.png", timeout=10).highlight().click()

# Highlight the match and mark its center with a short cross
screen.wait("login_button.png", timeout=10).highlight(show_center=True)

Advanced example: Record automation with screen capture

This example demonstrates screen recording combined with visual automation:

from pyvisionauto import Screen, Recorder
from pathlib import Path

screen = Screen()
recorder = Recorder()

recorder.start(output_file=Path("automation_demo.mp4"))
try:
    screen.activate_window("Calculator")
    screen.wait("button_1.png", timeout=10).highlight().click()
    screen.click("button_plus.png", timeout=5)
    screen.type_text("5")
    screen.wait("button_equals.png", timeout=5).highlight().click()
    screen.wait("result_7.png", timeout=3).highlight()
finally:
    recorder.stop()

Activate a window before matching

screen.activate_window("Calculator")
screen.click("button.png")

Platform differences

Screen recording requires ffmpeg installed and added to system PATH. Linux uses x11grab, Windows uses gdigrab.

Window focus on Linux (X11)

On X11 systems, mouse clicks alone do not automatically change keyboard focus. The window manager only reassigns focus in response to real hardware events or explicit window activation requests. This means:

• click() moves the cursor to the correct coordinates and clicks, but the keyboard focus stays wherever it was before.
• Any subsequent keyboard action (press(), type_text(), hotkeys) is delivered to whichever window currently has focus — which may not be the window you just clicked.

Rule of thumb: always call activate_window() before any keyboard action, targeting the exact window that should receive it.

Use xdotool to find the precise window name while the application is running:

xdotool search --name "" 2>/dev/null | while read id; do
    printf "ID=%-12s %s\n" "$id" "$(xdotool getwindowname "$id" 2>/dev/null)"
done

Pick the shortest substring that uniquely identifies the target window and use it in activate_window().

Main window vs. dialogs

When a modal dialog is open, activate the dialog directly — do not activate the main window and rely on the WM to forward focus:

from pyvisionauto import Screen

screen = Screen()

# --- Interacting with a dialog ---
# 1. Wait for the dialog image to appear and click it
screen.wait("open_project_dialog.png", timeout=30).click()
# 2. Activate the dialog window so keyboard input goes to it
screen.activate_window("Open Project")   # activate the dialog, not the main window
# 3. Now keyboard actions are reliably delivered to the dialog
screen.input.press("esc")

# --- Interacting with the main window ---
screen.activate_window("My App 2026")
screen.wait("toolbar_button.png", timeout=10).click()

Why not just activate the main window? On GNOME/Mutter, activating the main window does propagate focus to a modal child dialog — but this is WM-specific behaviour. Activating the dialog directly is explicit, portable, and not dependent on WM modal-focus rules.

highlight() and focus

highlight() launches a temporary tkinter overlay window. On some window managers this overlay can briefly steal keyboard focus. To avoid side effects:

• Prefer .click() before .highlight(), not after — the API supports chaining in both directions.
• Do not rely on focus being intact after .highlight() returns; call activate_window() again if keyboard actions follow.
• Pass show_center=True when you want the overlay to draw a short center cross using the same color and thickness as the border. showCenter=True is also accepted as a compatibility alias.

# Safer pattern: click first, highlight after (for visual feedback only)
screen.wait("button.png", timeout=10).click().highlight()

# Risky pattern: highlight steals focus, click lands on wrong window
# screen.wait("button.png", timeout=10).highlight().click()  # avoid

Notes

• Wayland-only and headless environments are not currently supported.
• On Windows with high-DPI scaling, coordinate accuracy may be affected.

Acknowledgments

This project is inspired by Sikulix and built with:

• OpenCV — Computer vision library for template matching
• mss — Fast, efficient screen capture
• PyAutoGUI — Cross-platform mouse and keyboard automation
• ffmpeg — Multimedia framework for screen recording