blindpilot-mcp 0.1.0

MCP server for desktop UI automation

Blindpilot MCP

BlindPilot MCP redefines computer use. Instead of AI wasting time interpreting screenshots, it reads directly on the operating system's UI tree and sends it to the LLM as text, enabling faster, more accurate, and more reliable desktop automation. No screenshot -> AI -> control loop. This project is also compatible with Linux and macOS.

Installation

MCP Configuration

If blindpilot-mcp is published as a Python package, you only need the MCP server name, command, and package executable:

{
  "mcpServers": {
    "blindpilot": {
      "type": "stdio",
      "command": "uvx",
      "args": ["blindpilot-mcp"]
    }
  }
}

Claude Code:

claude mcp add --transport stdio blindpilot -- uvx blindpilot-mcp

MCP UI fields:

Name: blindpilot
Command: uvx
Arguments: blindpilot-mcp

If you publish the package under another name, replace blindpilot-mcp with that package command.

Local Development

Use this while developing from a local clone.

1. Clone the repository:

git clone <repo-url>
cd blindpilot-mcp

2. Copy the absolute path to the project.

On Windows PowerShell:

(Get-Location).Path

On macOS or Linux:

pwd

3. No project install is required for local Claude/uvx usage.

Claude can run the server through uvx, which creates an isolated environment from this project automatically.

For local development, you can still install dependencies with uv:

uv sync

If you prefer pip:

pip install -r requirements.txt

4. Install optional platform dependencies when needed.

Linux accessibility support:

uv sync --extra linux

macOS Appium support:

uv sync --extra macos

Install every optional dependency:

uv sync --all-extras

With pip, use:

pip install ".[linux]"
pip install ".[macos]"

Windows does not need an extra; pywinauto is installed automatically on Windows.

5. Add the local MCP server to Claude Code with stdio and uvx.

Replace <absolute-project-path> with the path copied in step 2:

claude mcp add --transport stdio blindpilot -- uvx --from "<absolute-project-path>" blindpilot-mcp

Example on this machine:

claude mcp add --transport stdio blindpilot -- uvx --from "C:\Users\Noah\Documents\blindpilot-mcp" blindpilot-mcp

To share the MCP configuration with the repository:

claude mcp add --transport stdio --scope project blindpilot -- uvx --from "<absolute-project-path>" blindpilot-mcp

6. For Claude Desktop or another MCP client, use this local stdio server configuration:

{
  "mcpServers": {
    "blindpilot": {
      "type": "stdio",
      "command": "uvx",
      "args": [
        "--from",
        "<absolute-project-path>",
        "blindpilot-mcp"
      ]
    }
  }
}

GitHub Source

You can also run directly from a GitHub repository without cloning first:

claude mcp add --transport stdio blindpilot -- uvx --from "git+https://github.com/<owner>/blindpilot-mcp" blindpilot-mcp

MCP JSON:

{
  "mcpServers": {
    "blindpilot": {
      "type": "stdio",
      "command": "uvx",
      "args": [
        "--from",
        "git+https://github.com/<owner>/blindpilot-mcp",
        "blindpilot-mcp"
      ]
    }
  }
}

Usage

Blindpilot exposes MCP tools for desktop UI automation:

• run multiple tools in order with run_batch
• list open windows
• read accessible elements from a window
• click accessible controls
• type text into controls or the active window
• use keyboard, mouse, and clipboard actions
• terminate processes
• handle system dialogs

Interaction guidance:

• Use run_batch for ordered action chains.
• Do not add wait between normal interactions.
• Use short waits only for loading screens, routine checks, rate limits, or small animation delays.
• Prefer wait_for_element when waiting for a specific UI state.
• If snapshot_window fails, returns empty data, or looks incomplete, call it a second time before giving up.
• Use key_down + key_press + key_up for shortcuts. The unreliable combined hotkey helper is not exposed.

Batch example:

{
  "steps": [
    {"tool": "key_down", "args": {"key": "ctrl"}},
    {"tool": "key_press", "args": {"key": "l"}},
    {"tool": "key_up", "args": {"key": "ctrl"}},
    {"tool": "type_global", "args": {"text": "https://example.com"}},
    {"tool": "key_press", "args": {"key": "enter"}}
  ]
}

Development

Sync dependencies:

uv sync

Run the stdio server manually:

uv run blindpilot-mcp

When launched manually, the process waits for MCP messages on stdin. In normal use, Claude Code, Claude Desktop, or another MCP client starts it automatically.

Compile the modules to validate syntax:

uv run python -m compileall src

Publishing

This project is configured for PyPI Trusted Publishing through GitHub Actions.

In PyPI, create a trusted publisher with:

Workflow name: publish.yml
Environment name: pypi

Then publish by creating a GitHub release, or run the Publish to PyPI workflow manually from the repository's Actions tab.

Project Structure

src/
  main.py                         Compatibility entrypoint
  blindpilot_mcp/
    __init__.py                   Package marker
    server.py                     MCP server factory and stdio runner
    tools/
      __init__.py                 Tool package export
      registry.py                 Tool registration list
      context.py                  Platform detection and shared automation state
      accessibility.py            Window and accessibility-tree tools
      batch.py                    Ordered batch execution tool
      input.py                    Keyboard and mouse tools
      clipboard.py                Clipboard tools
      processes.py                Process termination tools
      sync.py                     Waiting and polling helpers
      values.py                   UI element value readers
      dialogs.py                  System dialog handling

Requirements

• Python 3.14
• uv
• Windows for UIA automation through pywinauto

Linux requires AT-SPI/dogtail support. macOS requires Appium with the mac2 driver.

Troubleshooting

If Claude logs an error like:

error: The system cannot find the file specified. (os error 2)

check the uvx arguments in the MCP configuration.

For the simple published-package setup, use:

"command": "uvx",
"args": ["blindpilot-mcp"]

For local development, the --from value must point to this repository, not to a Claude workspace folder.

Correct:

C:\Users\Noah\Documents\blindpilot-mcp

Incorrect:

C:\Users\Noah\Documents\Claude

The configured directory must contain:

pyproject.toml
src/blindpilot_mcp/server.py