easy-uiauto

English | 简体中文

easy-uiauto is a UI automation toolkit based on pyautogui and uiautomation.

It provides a comprehensive set of APIs for GUI automation, including mouse control, keyboard input, window management, and control location. It is suitable for automated testing, RPA (Robotic Process Automation), and other desktop automation scenarios.

Features

• Mouse control: click, double-click, right-click, drag and drop
• Keyboard input: text input, key press/release, combination keys
• Window management: activate, maximize, switch windows
• Control location: XPath-based positioning, image recognition
• Visual feedback: real-time control highlighting during recording
• Action recording: record user interactions and generate scripts
• Rich text field support: clipboard-based text input
• Cross-framework support: Win32, Qt, and other UI frameworks

Installation

Install from PyPI:

pip install easy-uiauto

Or install from source:

git clone https://github.com/Poggi-Tang/easyautomation.git
cd easyautomation
pip install -e .

MCP Server

Install the optional MCP dependencies when using easy-uiauto from an MCP client:

pip install "easy-uiauto[mcp]"

Install local OCR and image-template fallback support when needed:

pip install "easy-uiauto[mcp,vision]"

The vision extra provides OpenCV template matching and the Python Tesseract adapter. OCR also requires the system Tesseract executable and relevant language data (for example eng or chi_sim). The MCP tools are find_control_by_image, click_by_image, find_text_on_screen, and click_text_on_screen.

Remote multimodal location does not run a local model or require an AI SDK. Set an OpenAI-compatible vision endpoint and credentials, then use find_control_by_vision or click_by_vision:

EASY_UIAUTO_VISION_API_URL=https://your-api.example/v1/chat/completions
EASY_UIAUTO_VISION_API_KEY=your-api-key
EASY_UIAUTO_VISION_MODEL=your-vision-model

Those tools upload the current screenshot to the configured endpoint only for that request. Use them as a final fallback after UIA, OCR, or image matching.

The MCP server is part of the library and reuses the same automation APIs:

easy_uiauto --help
easy_uiauto --version
easy_uiauto

The standard commands below register, inspect, or remove the global MCP configuration through the client's own CLI. The standard install command does not overwrite an existing entry with the same name.

For a complete Codex deployment, install the package and run full setup:

pip install --upgrade "easy-uiauto[mcp,vision]"
easy_uiauto --full-setup-codex \
  --vision-url https://your-api.example/v1/chat/completions \
  --vision-model your-vision-model

Full setup securely prompts for a missing API key, installs missing Python vision dependencies, installs Tesseract through winget when necessary, replaces only the easy_uiauto Codex MCP entry, and validates UIA, local OCR, and remote AI vision. The OCR and AI checks use generated test images; they do not upload the current desktop. Each result includes its elapsed time, and any failed check makes the command fail.

For a minimal Codex deployment with remote AI vision, use the quick setup command. When the API key is not already present in the Windows user environment, it prompts once through hidden terminal input or a password dialog for non-interactive agents. It persists the three vision variables, replaces only the easy_uiauto Codex MCP entry, and skips OCR installation and UI tests:

easy_uiauto --quick-setup-codex \
  --vision-url https://your-api.example/v1/chat/completions \
  --vision-model your-vision-model

One-sentence instruction for a Codex agent:

Install the latest easy-uiauto[mcp,vision] from PyPI, then run easy_uiauto --full-setup-codex --vision-url URL --vision-model MODEL; do not inspect unrelated projects or search the web, report every validation result and elapsed time, then ask me to restart Codex.

easy_uiauto --install-codex
easy_uiauto --show-codex-config
easy_uiauto --uninstall-codex

easy_uiauto --install-claude-code
easy_uiauto --show-claude-code-config
easy_uiauto --uninstall-claude-code

Codex registration uses its global config.toml. Claude Code registration uses the user scope, so it is available to every local project. Restart the client after installing or removing the server.

The long-running TCP service is also available:

easy_uiauto_service --help
python -m easy_uiauto.mcp.service --port 9876

Application UI Knowledge and CLI

Scan any visible Windows application into an Obsidian-compatible knowledge vault:

easy_uiauto_ui scan "Window title"
easy_uiauto_ui apps
easy_uiauto_ui commands <app-id>
easy_uiauto_ui search <app-id> "search terms"
easy_uiauto_ui run <app-id> <page.region.control.action> --text "optional text"
easy_uiauto_ui batch <app-id> '["main.keypad.6.click", "main.keypad.plus.click"]'
easy_uiauto_ui learn-effect <app-id> <command> --recover
easy_uiauto_ui explore <app-id> --policy safe --max-actions 10 --max-depth 3
easy_uiauto_ui interactions <app-id>
easy_uiauto_ui teach <app-id> <control-id> "Meaning" intent "Description"

The default vault is ~/easy_uiauto_vault. Set EASY_UIAUTO_KNOWLEDGE_DIR to use another Obsidian vault. Each application stores Markdown/YAML records, page screenshots, per-control visual-state PNGs, interaction before/after images, a quarantine directory, and a generated operations/UI-CLI.md catalog. .easy_uiauto/index.json is only a disposable search cache and can be rebuilt with easy_uiauto_ui reindex.

The default visual-first scan makes one multimodal request to identify the current page, functional regions, and only task-relevant controls. Pixel targets are mapped back to local UIA controls and scored for stable names, automation IDs, supported actions, and tight bounds. This avoids a complete UIA-tree walk and a second remote semantic pass. Use --strategy full-uia only for diagnostic coverage when visual targeting is unsuitable. Both strategies store user-facing meaning, stable intent, description, aliases, evidence, risk, actions, ambiguity, and confidence. Commands are published only after high-confidence semantics and LOCATION/image validation. The visual request uses SSE streaming when supported and window capture uses virtual-desktop coordinates, including secondary monitors. If vision returns no controls, visual-first reports an empty scan instead of silently invoking a second remote request.

learn-effect captures the target window and local full-desktop state before and after one verified command. It waits for delayed changes to stabilize, reduces pixel changes to local regions, checks only UIA controls in those regions, inventories new or transient top-level windows, captures action-control properties, and asks vision to generalize a success condition. Full desktop images stay local; only target-window images and crops of newly opened related windows are sent to the configured endpoint. explore repeats this for known reversible commands, deduplicates page/command states, presses Escape to recover, and stops after interference or failed recovery. safe permits only safe commands; supervised also permits reversible state-changing commands. External, destructive, or confirmation-required commands are never explored automatically. Scrolling and dragging are intentionally not part of exploration.

At runtime, resolution follows LOCATION, stored multi-state image templates, local OCR, and finally opt-in remote vision. Use --allow-vision-fallback or allow_vision_fallback=true for the last step. Ambiguous, stale, or missing controls are quarantined and excluded from execution until a successful rescan repairs them.

Locator/image verification, semantic verification, and functional execution are tracked separately. Scanning does not click every control: doing so could send, publish, purchase, delete, or otherwise change external state. AI meanings are marked as inferred until directly taught or observed, and execution history never masquerades as proof of the resulting application behavior. Use easy_uiauto_ui teach or the teach_ui_control MCP tool to correct uncertain meanings; manual semantics survive rescans but cannot bypass a failed locator. External or destructive commands require --confirm or confirm=true.

For consecutive operations on one stable page, use easy_uiauto_ui batch or the run_ui_commands MCP tool. A batch loads the command catalog once, finds and captures the window once, verifies every unique control before performing any action, executes the ordered sequence, and rebuilds the knowledge index once. Steps may be command strings or objects such as {"command":"main.form.message.set-text","text":"hello"}. Split a workflow after any command that navigates to another page.

Knowledge created before contextual control semantics were introduced remains readable, but its old commands are intentionally not executable. Rescan each page once to add the required semantic evidence, confidence, risk, and verification fields.

The same workflow is available through MCP tools:

• scan_window_knowledge
• list_ui_knowledge_apps
• search_ui_knowledge
• list_ui_commands
• run_ui_command
• run_ui_commands
• learn_ui_command_effect
• explore_ui_workflows
• list_ui_interactions
• teach_ui_control
• rebuild_ui_knowledge_index

--full-setup-codex also installs or updates the bundled easy-uiauto-learning and easy-uiauto-operate Codex skills. Use --install-codex-skills to update only those skills.

For MCP client configuration, start the server with python -m easy_uiauto.mcp.server. Control-vector persistence is optional. To enable it, set EASY_UIAUTO_CONTROL_VECTOR_DB_DIR to a directory containing control_vector_store.py; otherwise capture tools still return records but do not persist them.

Control lookup uses the library's canonical LOCATION object rather than a flat selector. Obtain it from a recorded action or from get_control_at_position, then pass the returned LOCATION object directly to find_control:

{
  "WindowName": "My Application",
  "Name": "Save",
  "ClassName": "ButtonClass",
  "ControlType": "ButtonControl",
  "foundIndex": 1,
  "AutomationId": "saveButton",
  "Xpath": [
    {"ControlType": "WindowControl", "Name": "My Application", "searchDepth": 1},
    {"ControlType": "ButtonControl", "Name": "Save", "foundIndex": 1, "searchDepth": 2}
  ],
  "Img": "",
  "PARAMETERS": {}
}

find_control also accepts a complete recorded action containing LOCATION and the complete result from get_control_at_position. Legacy flat arguments remain supported for compatibility, but full XPath data is more reliable for duplicate or deeply nested controls.

Quick Start

Basic Control Operations

from easy_uiauto.ctrl import Controller

# Left click on a control
Controller.left_click(
    ActionTitle="Click OK Button",
    WindowName="My Application",
    Name="OK",
    ClassName=None,
    ControlType="ButtonControl",
    foundIndex=0,
    AutomationId="",
    Xpath=[],
    Img="",
    PARAMETERS={}
)

# Input text into a field
Controller.input_text(
    ActionTitle="Enter Username",
    WindowName="Login Dialog",
    Name="Username",
    ClassName=None,
    ControlType="EditControl",
    foundIndex=0,
    AutomationId="",
    Xpath=[],
    Img="",
    PARAMETERS={"输入文本": "test_user"}
)

# Keyboard shortcut
Controller.key_group(
    ActionTitle="Save File",
    WindowName="Notepad",
    Name="",
    ClassName=None,
    ControlType="",
    foundIndex=0,
    AutomationId="",
    Xpath=[],
    Img="",
    PARAMETERS={"组合键": "ctrl+s"}
)

Recording User Actions

from easy_uiauto.record import run_record

# Start recording user actions
run_record(write_file=True)
# Press ESC to stop recording
# Generated script will be saved to Record{timestamp}.py

Project Structure

easyautomation
├── .github/
│   └── workflows/
│       ├── ci.yml
│       ├── publish.yml
│       └── release.yml
├── src/
│   └── easy-uiauto/
│       ├── __init__.py
│       ├── ctrl.py          # Core controller (mouse/keyboard actions)
│       ├── draw.py          # Visual feedback (control highlighting)
│       ├── record.py        # Action recording
│       └── utils.py         # Utility functions (control location, caching)
├── tests/
├── CHANGELOG.md
├── LICENSE
├── README.md
├── README.zh-CN.md
└── pyproject.toml

Release Automation

This repository is prepared for a professional Python package workflow:

• CI runs lint and tests on push and pull request.
• Semantic Release updates the version, changelog, tag, and GitHub Release.
• Trusted Publishing publishes to PyPI from GitHub Actions without a PyPI API token.
• Build artifacts include both source distribution and wheel.

Development

pip install -e .[dev]
pytest
ruff check .

Usage Examples

For more examples, please refer to the test files in the demo/ directory or check the docstrings in the source code.

License

MIT License. See LICENSE.

Contact

Scan the QR code to add me on WeChat: