easy-uiauto

English | 简体中文

easy-uiauto is a UI automation toolkit based on pyautogui and uiautomation.

It provides a comprehensive set of APIs for GUI automation, including mouse control, keyboard input, window management, and control location. It is suitable for automated testing, RPA (Robotic Process Automation), and other desktop automation scenarios.

Features

• Mouse control: click, double-click, right-click, drag and drop
• Keyboard input: text input, key press/release, combination keys
• Window management: activate, maximize, switch windows
• Control location: XPath-based positioning, image recognition
• Visual feedback: real-time control highlighting during recording
• Action recording: record user interactions and generate scripts
• Rich text field support: clipboard-based text input
• Cross-framework support: Win32, Qt, and other UI frameworks

Installation

Install from PyPI:

pip install easy-uiauto

Or install from source:

git clone https://github.com/Poggi-Tang/easyautomation.git
cd easyautomation
pip install -e .

MCP Server

Install the optional MCP dependencies when using easy-uiauto from an MCP client:

pip install "easy-uiauto[mcp]"

Install local OCR and image-template fallback support when needed:

pip install "easy-uiauto[mcp,vision]"

The vision extra provides OpenCV template matching and the Python Tesseract adapter. OCR also requires the system Tesseract executable and relevant language data (for example eng or chi_sim). The MCP tools are find_control_by_image, click_by_image, find_text_on_screen, and click_text_on_screen.

Remote multimodal location does not run a local model or require an AI SDK. Set an OpenAI-compatible vision endpoint and credentials, then use find_control_by_vision or click_by_vision:

EASY_UIAUTO_VISION_API_URL=https://your-api.example/v1/chat/completions
EASY_UIAUTO_VISION_API_KEY=your-api-key
EASY_UIAUTO_VISION_MODEL=your-vision-model

Those tools upload the current screenshot to the configured endpoint only for that request. Use them as a final fallback after UIA, OCR, or image matching.

The MCP server is part of the library and reuses the same automation APIs:

easy_uiauto --help
easy_uiauto --version
easy_uiauto

The standard commands below register, inspect, or remove the global MCP configuration through the client's own CLI. The standard install command does not overwrite an existing entry with the same name.

For a complete Codex deployment, install the package and run full setup:

pip install --upgrade "easy-uiauto[mcp,vision]"
easy_uiauto --full-setup-codex \
  --vision-url https://your-api.example/v1/chat/completions \
  --vision-model your-vision-model

Full setup securely prompts for a missing API key, installs missing Python vision dependencies, installs Tesseract through winget when necessary, replaces only the easy_uiauto Codex MCP entry, and validates UIA, local OCR, and remote AI vision. The OCR and AI checks use generated test images; they do not upload the current desktop. Each result includes its elapsed time, and any failed check makes the command fail.

For a minimal Codex deployment with remote AI vision, use the quick setup command. When the API key is not already present in the Windows user environment, it prompts once through hidden terminal input or a password dialog for non-interactive agents. It persists the three vision variables, replaces only the easy_uiauto Codex MCP entry, and skips OCR installation and UI tests:

easy_uiauto --quick-setup-codex \
  --vision-url https://your-api.example/v1/chat/completions \
  --vision-model your-vision-model

One-sentence instruction for a Codex agent:

Install the latest easy-uiauto[mcp,vision] from PyPI, then run easy_uiauto --full-setup-codex --vision-url URL --vision-model MODEL; do not inspect unrelated projects or search the web, report every validation result and elapsed time, then ask me to restart Codex.

easy_uiauto --install-codex
easy_uiauto --show-codex-config
easy_uiauto --uninstall-codex

easy_uiauto --install-claude-code
easy_uiauto --show-claude-code-config
easy_uiauto --uninstall-claude-code

Codex registration uses its global config.toml. Claude Code registration uses the user scope, so it is available to every local project. Restart the client after installing or removing the server.

The long-running TCP service is also available:

easy_uiauto_service --help
python -m easy_uiauto.mcp.service --port 9876

Application UI Knowledge and CLI

Scan any visible Windows application into an Obsidian-compatible knowledge vault:

easy_uiauto_ui scan "Window title"
easy_uiauto_ui apps
easy_uiauto_ui commands <app-id>
easy_uiauto_ui search <app-id> "search terms"
easy_uiauto_ui run <app-id> <page.region.control.action> --text "optional text"

The default vault is ~/easy_uiauto_vault. Set EASY_UIAUTO_KNOWLEDGE_DIR to use another Obsidian vault. Each application stores Markdown/YAML records, page screenshots, individual control PNG crops, a quarantine directory, and a generated operations/UI-CLI.md catalog. .easy_uiauto/index.json is only a disposable search cache and can be rebuilt with easy_uiauto_ui reindex.

Scanning combines the complete visible UIA tree with remote AI page/region segmentation. All visible controls are saved; actionable controls become UI CLI commands only after LOCATION and control-image validation. When LOCATION is not available, one unique high-confidence image match can validate and execute the control. Ambiguous, stale, or missing controls are quarantined and excluded from execution until a successful rescan repairs them.

The same workflow is available through MCP tools:

• scan_window_knowledge
• list_ui_knowledge_apps
• search_ui_knowledge
• list_ui_commands
• run_ui_command
• rebuild_ui_knowledge_index

--full-setup-codex also installs or updates the bundled easy-uiauto-learning and easy-uiauto-operate Codex skills. Use --install-codex-skills to update only those skills.

For MCP client configuration, start the server with python -m easy_uiauto.mcp.server. Control-vector persistence is optional. To enable it, set EASY_UIAUTO_CONTROL_VECTOR_DB_DIR to a directory containing control_vector_store.py; otherwise capture tools still return records but do not persist them.

Control lookup uses the library's canonical LOCATION object rather than a flat selector. Obtain it from a recorded action or from get_control_at_position, then pass the returned LOCATION object directly to find_control:

{
  "WindowName": "My Application",
  "Name": "Save",
  "ClassName": "ButtonClass",
  "ControlType": "ButtonControl",
  "foundIndex": 1,
  "AutomationId": "saveButton",
  "Xpath": [
    {"ControlType": "WindowControl", "Name": "My Application", "searchDepth": 1},
    {"ControlType": "ButtonControl", "Name": "Save", "foundIndex": 1, "searchDepth": 2}
  ],
  "Img": "",
  "PARAMETERS": {}
}

find_control also accepts a complete recorded action containing LOCATION and the complete result from get_control_at_position. Legacy flat arguments remain supported for compatibility, but full XPath data is more reliable for duplicate or deeply nested controls.

Quick Start

Basic Control Operations

from easy_uiauto.ctrl import Controller

# Left click on a control
Controller.left_click(
    ActionTitle="Click OK Button",
    WindowName="My Application",
    Name="OK",
    ClassName=None,
    ControlType="ButtonControl",
    foundIndex=0,
    AutomationId="",
    Xpath=[],
    Img="",
    PARAMETERS={}
)

# Input text into a field
Controller.input_text(
    ActionTitle="Enter Username",
    WindowName="Login Dialog",
    Name="Username",
    ClassName=None,
    ControlType="EditControl",
    foundIndex=0,
    AutomationId="",
    Xpath=[],
    Img="",
    PARAMETERS={"输入文本": "test_user"}
)

# Keyboard shortcut
Controller.key_group(
    ActionTitle="Save File",
    WindowName="Notepad",
    Name="",
    ClassName=None,
    ControlType="",
    foundIndex=0,
    AutomationId="",
    Xpath=[],
    Img="",
    PARAMETERS={"组合键": "ctrl+s"}
)

Recording User Actions

from easy_uiauto.record import run_record

# Start recording user actions
run_record(write_file=True)
# Press ESC to stop recording
# Generated script will be saved to Record{timestamp}.py

Project Structure

easyautomation
├── .github/
│   └── workflows/
│       ├── ci.yml
│       ├── publish.yml
│       └── release.yml
├── src/
│   └── easy-uiauto/
│       ├── __init__.py
│       ├── ctrl.py          # Core controller (mouse/keyboard actions)
│       ├── draw.py          # Visual feedback (control highlighting)
│       ├── record.py        # Action recording
│       └── utils.py         # Utility functions (control location, caching)
├── tests/
├── CHANGELOG.md
├── LICENSE
├── README.md
├── README.zh-CN.md
└── pyproject.toml

Release Automation

This repository is prepared for a professional Python package workflow:

• CI runs lint and tests on push and pull request.
• Semantic Release updates the version, changelog, tag, and GitHub Release.
• Trusted Publishing publishes to PyPI from GitHub Actions without a PyPI API token.
• Build artifacts include both source distribution and wheel.

Development

pip install -e .[dev]
pytest
ruff check .

Usage Examples

For more examples, please refer to the test files in the demo/ directory or check the docstrings in the source code.

License

MIT License. See LICENSE.

Contact

Scan the QR code to add me on WeChat: