AXPilot

The MCP server for macOS. AXPilot lets any AI agent see and operate the apps on your Mac, through the macOS Accessibility API first and screenshots second, with safety controls that are on by default.

_{A real recording on macOS 26: Claude Code (left) drives TextEdit (right) through AXPilot. Sped up in places; the speed is shown on screen.}

Project status: alpha. The first alpha, 0.1.0a1, is on PyPI; you can also run it from source. It is verified on macOS 26 on Apple silicon; macOS 14 and 15 and Intel Macs are not tested yet. Expect breaking changes until 1.0. See the roadmap.

Why AXPilot

• Sees the UI the way macOS does. A compact text snapshot of the accessibility tree gives every button, field, list and menu a short ref (e1, e2, …), so agents act on elements instead of guessing pixels. Text-only and local models work; vision is optional.
• Acts reliably. Each action tries the most robust strategy first (an Accessibility action), falls back to synthetic mouse and keyboard events, and returns a diff of what changed.
• Safe by default. Scripting is off, destructive actions need your confirmation, password managers and the Privacy & Security settings are off limits, secure fields are never read, and there is no telemetry.
• Works with any MCP client or agent framework. Standard MCP over stdio or Streamable HTTP: Claude Desktop, Claude Code, Cursor, VS Code, Codex CLI, Gemini CLI, OpenAI Agents SDK, LangGraph, Pydantic AI and more.
• Permissions that just work. A tiny frozen AXPilot.app launcher holds the macOS permissions, so you grant Accessibility once, to AXPilot, and every client can use it, across updates.

Features

The feature set planned for 1.0. Every tool below is implemented and tested against the simulated desktop. On a real Mac (macOS 26), the perception and input tools are verified, including dropdowns, menus and dialogs; the app, system, scripting and browser tools, the kill switch and the on-screen indicator are not yet. The tool reference documents each one.

How it works

 MCP clients and agent frameworks
 (Claude, Cursor, VS Code, Codex, OpenAI Agents SDK, LangGraph, …)
        │ spawn `axpilot stdio`                │ Streamable HTTP (`axpilot http`)
        ▼                                      │ 127.0.0.1 + bearer token
 ┌────────────────┐   Unix socket (0600)       │
 │ axpilot bridge │──────────────┐             │
 │ (no perms)     │              ▼             ▼
 └────────────────┘   ┌──────────────────────────────────────────┐
                      │ AXPilot.app: tiny frozen launcher        │
                      │   └─ Python host (child process)         │
                      │       ├ MCP server: tools and schemas    │
                      │       ├ Policy and safety layer          │
                      │       └ Engine: Accessibility, input,    │
                      │         ScreenCaptureKit, Vision OCR     │
                      └──────────────────────────────────────────┘
        macOS permissions (Accessibility, Screen Recording) → AXPilot.app

Spike S1 confirmed this design on macOS 26 (Apple silicon): the permission prompts name AXPilot, and one grant covers every client. Direct mode (axpilot stdio --direct) runs everything in one process instead; macOS then attributes permissions to whichever app started the server (your terminal, Claude, Cursor, …).

Install

Requirements: macOS 14 Sonoma or later, Apple silicon or Intel.

Run from source (today)

You need uv and the Xcode Command Line Tools (xcode-select --install), which build the launcher.

git clone https://github.com/vansh3503/Mac-Mcp.git axpilot
cd axpilot
uv sync
launcher/build.sh                      # builds launcher/build/AXPilot.app
uv run axpilot setup --dev-source .    # installs ~/Applications/AXPilot.app and its runtime
uv run axpilot connect claude-desktop  # or claude-code, cursor, vscode, … (see --list)
uv run axpilot doctor                  # checks the installation

Restart your MCP client, then ask it to use AXPilot ("take a snapshot of Finder"):

1. The first time a client connects, AXPilot asks you whether to allow it. Allow it.
2. The first tool call makes macOS ask whether AXPilot may control your computer. Open System Settings, switch on AXPilot under Accessibility, and ask again; no restart is needed. Screenshots and OCR ask for Screen Recording the same way.

Other ways to run it, without the launcher:

uv run axpilot stdio --engine fake   # the simulated desktop (any OS, no permissions needed)
uv run axpilot stdio --direct        # direct mode: grant Accessibility to the app that starts it

axpilot stdio speaks MCP on stdin/stdout, so it waits for a client. To poke at it by hand, use the MCP Inspector: npx @modelcontextprotocol/inspector uv run axpilot stdio --engine fake.

Install from PyPI (alpha)

uv tool install axpilot     # or: pipx install axpilot
axpilot setup               # checks your Mac and walks you through permissions
axpilot connect claude-desktop

Details, the one-line installer and other channels: docs/install.md. Permissions explained: docs/permissions.md.

Connect a client

Today, from source

In the clone, uv run axpilot connect <client> writes the client's config for you (with a backup), pointing it at the checkout's axpilot by absolute path. Restart the client afterwards. For direct mode, or to write the config by hand, see docs/clients.md.

Installed from PyPI

axpilot connect <client> writes the config for you (with a backup). The snippets it writes:

Claude Desktop, Cursor (~/.cursor/mcp.json) and most other clients:

{ "mcpServers": { "axpilot": { "command": "axpilot", "args": ["stdio"] } } }

(Claude Desktop needs the absolute path of axpilot, e.g. /Users/you/.local/bin/axpilot.)

Claude Code:

claude mcp add axpilot -- axpilot stdio

VS Code (.vscode/mcp.json in a workspace, or your user mcp.json):

{ "servers": { "axpilot": { "type": "stdio", "command": "axpilot", "args": ["stdio"] } } }

More clients (Windsurf, Zed, Codex CLI, Gemini CLI, LM Studio, Goose, Cline, Continue): docs/clients.md.

Use it from an agent framework

Anything that speaks MCP works. Two examples (APIs as of writing; one for each of eleven frameworks is in examples/frameworks/):

OpenAI Agents SDK

from agents import Agent, Runner
from agents.mcp import MCPServerStdio

async with MCPServerStdio(params={"command": "axpilot", "args": ["stdio"]}, cache_tools_list=True) as mac:
    agent = Agent(
        name="Mac operator",
        instructions="Call snapshot before acting, act on element refs, and verify the result.",
        mcp_servers=[mac],
    )
    result = await Runner.run(agent, "Open Notes and create a note titled 'Groceries'")

LangChain / LangGraph

from langchain.agents import create_agent
from langchain_mcp_adapters.client import MultiServerMCPClient

client = MultiServerMCPClient({"axpilot": {"command": "axpilot", "args": ["stdio"], "transport": "stdio"}})
agent = create_agent("<your-model>", await client.get_tools())

More frameworks: docs/frameworks.md.

Tools

The 1.0 tool surface, implemented. Profiles pick a starting set: minimal (10 tools, for small or local models), standard (default), full, and computer (one vision-first computer tool). A windows-mcp naming exposes the same tools under Windows-MCP's names and parameters.

Every tool has an input and output schema, MCP annotations (readOnlyHint, destructiveHint, …) and stable error codes such as STALE_REF, BLOCKED_BY_POLICY or USER_ACTIVE. Full reference, generated from the code: docs/tools.md.

Safety defaults

Screen content is untrusted input for an agent. Read the security model before enabling scripting or browser. To report a vulnerability, see SECURITY.md.

How AXPilot compares

AXPilot is the macOS counterpart of Windows-MCP, with a tool surface (and an optional naming profile) that lets Windows-MCP prompts carry over. Other good projects in this space include MacOS-MCP (Python + PyObjC, the closest peer), Peekaboo (Swift), mcp-server-macos-use (Swift) and cua-driver. AXPilot's focus:

1. Permissions that survive updates: a frozen launcher holds the grant for every client.
2. Safety and privacy as the headline: no telemetry, pairing, kill switch, confirmations, audit.
3. Spec-first MCP: schemas, annotations, structured output, deterministic tool order, stateless handles.
4. Integration kit and evals: framework examples tested in CI, an Agent Skill, published eval results.

Roadmap

About 12 weeks to 1.0: v0.1 alpha (engine and core tools in direct mode; 0.1.0a1 is out), v0.2 beta (launcher, host app, safety), v0.3 public preview (one-line installer, connect for 12 clients, MCP Registry), v0.9 (full toolsets, OCR, frameworks, evals), then v1.0. Details and the open questions are in the project plan; decisions are recorded as ADRs.

Contributing

Contributions are welcome, and most of the code (server, policy, perception) runs and is tested on Linux. Start with CONTRIBUTING.md; AI coding agents should read AGENTS.md. Please follow the Code of Conduct.

License

Apache-2.0.

Acknowledgements

AXPilot is inspired by Windows-MCP by CursorTouch, whose tool design and install experience set the bar for desktop MCP servers. Thanks also to the authors of PyObjC, the MCP Python SDK and the projects listed above.

"Mac" and "macOS" are trademarks of Apple Inc. Claude and Claude Code are trademarks of Anthropic. Other product and framework names mentioned here (such as Cursor, VS Code, LangChain and CrewAI) are trademarks of their respective owners and are used only to describe compatibility. AXPilot is an independent project and is not affiliated with or endorsed by any of them.

The demo video's voiceover was made with ElevenLabs and its music comes from the HeyGen audio library.