autoMate

🤖 Desktop Automation for Apps Without APIs

中文 | 日本語

Give Claude hands and eyes — automate any desktop app, even if it has no API

https://github.com/user-attachments/assets/bf27f8bd-136b-402e-bc7d-994b99bcc368

💡 What is autoMate?

autoMate is an MCP server that gives AI assistants (Claude, GPT, etc.) the ability to control any desktop application — even apps with no API, no plugin system, and no automation support.

What makes it different from filesystem / browser / Windows MCP:

Two modes:

✨ Features

• 🖥️ Automates apps with no API — if it has a GUI, autoMate can drive it
• 📚 Reusable script library — save workflows once, run forever; install community scripts in one command
• ☁️ Cloud Vision — screen parsing via OmniParser + action reasoning via UI-TARS, all in the cloud, zero local GPU
• 🧠 Claude knows when to use it — clear identity prevents autoMate from being bypassed by other MCPs
• 🤖 Zero config for basic use — no API keys, no env vars needed to get started
• 🌍 Cross-platform — Windows, macOS, Linux

🔌 Setup

Prerequisite: pip install uv

Claude Desktop

Open Settings → Developer → Edit Config, then add:

{
  "mcpServers": {
    "automate": {
      "command": "uvx",
      "args": ["automate-mcp@latest"]
    }
  }
}

Restart Claude Desktop — done. @latest keeps autoMate up to date automatically.

OpenClaw

Edit ~/.openclaw/openclaw.json:

{
  "mcpServers": {
    "automate": {
      "command": "uvx",
      "args": ["automate-mcp@latest"]
    }
  }
}

openclaw gateway restart

Cursor / Windsurf / Cline

Settings → MCP Servers → Add:

{
  "automate": {
    "command": "uvx",
    "args": ["automate-mcp@latest"]
  }
}

☁️ Cloud Vision (Optional)

Cloud Vision adds autonomous screen parsing and action reasoning to autoMate — no local GPU required.

It uses two HuggingFace Inference Endpoints:

• OmniParser V2 — detects all UI elements (icons, buttons, text) from a screenshot
• UI-TARS / Qwen-VL — vision-language model that decides what action to take next

Setup

Add these env vars to your MCP config:

{
  "mcpServers": {
    "automate": {
      "command": "uvx",
      "args": ["automate-mcp@latest"],
      "env": {
        "AUTOMATE_HF_TOKEN": "hf_...",
        "AUTOMATE_SCREEN_PARSER_URL": "https://your-omniparser-endpoint.aws.endpoints.huggingface.cloud",
        "AUTOMATE_ACTION_MODEL_URL": "https://your-uitars-endpoint.aws.endpoints.huggingface.cloud",
        "AUTOMATE_ACTION_MODEL_NAME": "ByteDance-Seed/UI-TARS-1.5-7B",
        "AUTOMATE_HF_NAMESPACE": "your-hf-username",
        "AUTOMATE_SCREEN_PARSER_ENDPOINT": "omniparser-v2",
        "AUTOMATE_ACTION_MODEL_ENDPOINT": "ui-tars-1-5-7b"
      }
    }
  }
}

See .env.example in the repo for the full reference.

Cloud Vision workflow

1. warm_endpoints   — wake up scaled-to-zero endpoints (1–5 min)
2. parse_screen     — detect all UI elements via cloud OmniParser
3. reason_action    — ask VLM what to click/type next
   — or —
   smart_act        — full autonomous loop: parse → reason → execute → repeat

🛠️ MCP Tools

Script library — save once, run forever:

Cloud Vision — autonomous UI understanding (requires HF config):

Low-level desktop control — used when building or executing scripts:

📚 Script Library

Scripts are saved as .md files in ~/.automate/scripts/ — human-readable, git-friendly, shareable.

---
name: jianying_export_douyin
description: Export the current 剪映 project as a 9:16 Douyin video
created: 2025-01-01
---

## Steps

1. Open export dialog [key:ctrl+e]
2. Select resolution 1080×1920 [click:coord=320,480]
3. Set format to MP4 [click:coord=320,560]
4. Click export [click:coord=800,650]
5. Wait for export to finish [wait:5]

Inline hint syntax:

Steps without hints are interpreted by the AI vision model at runtime.

📝 FAQ

Q: How is this different from just using Claude's computer-use capability?
autoMate provides persistent, reusable scripts. Once you automate a task, it's saved and runs instantly next time. Cloud Vision mode also lets autoMate do its own screen parsing without relying on Claude's vision.

Q: Why does Claude sometimes use Windows MCP / filesystem MCP instead of autoMate?
Update to v0.4.0+ — the server description now explicitly tells Claude when to use autoMate vs other MCPs.

Q: Do I need a GPU for Cloud Vision?
No — everything runs on HuggingFace Inference Endpoints in the cloud. You only need a HF token and deployed endpoints.

Q: Does it work on macOS / Linux?
Yes — all three platforms. This is the main advantage over Quicker (Windows-only).

🤝 Contributing