---

🎬 See it in action

No API key needed to try it — OmniPilot ships with a fully offline, scripted demo:

pip install omnipilot-agent
omnipilot demo

The agent plans a task, asks for approval before writing files or running shell commands, executes them and reports back — all rendered locally with zero network calls:

Prefer a browser? omnipilot serve gives you a dark chat UI with live streaming tool activity, approval decisions and token/cost accounting:

You can even drive the web UI offline with the bundled streaming demo server:

python examples/demo_streaming_server.py --port 8801
# then open http://127.0.0.1:8801

---

📖 Table of contents

• See it in action
• Why OmniPilot
• Features
• Demos / example tasks
• MCP: use any Model Context Protocol server
• Installation
• Getting an API key
• Quickstart
• Supported LLM providers
• The permission system (allowances)
• Tool catalog
• CLI reference
• Configuration
• Local web/API server
• Using OmniPilot from Python
• Writing your own tools (plugins)
• Architecture
• Safety & security
• Documentation index
• Roadmap
• Contributing
• License & disclaimer

---

💡 Why OmniPilot?

Most agent projects are locked to one model vendor, run on one OS, or execute commands with no safety layer. OmniPilot gives you:

• 🤝 Bring your own model — 13 providers out of the box and any OpenAI-compatible endpoint. Switch mid-conversation with /provider groq or /model ....
• 🖥️ Real computer control — shell, persistent Python, files, apps, mouse, keyboard, screenshots, clipboard, processes, notifications and web requests.
• 🙋 You stay in charge — every risky action is classified (low / medium / high / critical) and either auto-approved, prompted, or blocked. Critical patterns (rm -rf /, drive formatting, fork bombs…) are blocked even in YOLO mode.
• 🪟🍎🐧 Genuinely cross-platform — Windows (cmd + PowerShell), macOS and Linux each get first-class command and application handling.
• 👁️ Vision-based computer use — screenshots are sent to vision-capable models (GPT-4o, Claude, Gemini, vision NIMs…) so the agent can see what it is doing.
• 🔌 Hackable — tiny plugin API, clean provider adapter layer, JSON API server, 100% Python, MIT licensed.

---

✨ Features

Agents

• A mature tool-calling loop (plan → observe → act → verify → summarize)
• Native function-calling adapters — no prompt-parser hacks — for each vendor dialect
• Multiple tool calls per turn, automatic context compaction, token & cost accounting
• Persistent sessions (auto-saved, resumable across restarts) with /save, /load, /sessions, /undo
• Slash-command REPL plus one-shot mode for scripts/CI
• Per-session permission grants ("always allow this tool", "always allow this exact action")
• Built-in local web UI + REST API (omnipilot serve) with SSE streaming
• Live provider/model switching without losing the conversation
• 🔌 MCP client built in — connect any Model Context Protocol server (stdio, streamable-HTTP or SSE) and use its tools as first-class OmniPilot tools, behind the same permission system

Computer control (41 built-in tools)

Domain	Examples
Shell	shell.run_command (cmd / PowerShell / bash / zsh, timeouts, background processes)
Python	python.run_python — persistent interpreter, variables survive between calls
Files	read, write, targeted find/replace edit, list, move, copy, delete, glob search, content grep
Apps	open/close/list applications, open files in default apps (Start Menu, open -a, xdg)
Screen	screenshots → vision models, screen size, image-on-screen lookup
GUI	click, double/right click, move, drag, scroll, type (unicode-safe), keys, hotkeys
Browser/ web	open URLs & searches in your browser, fetch pages, arbitrary HTTP requests, downloads, keyless web search
System	system info, drives, env vars (secret masking), process list/kill, native notifications
Clipboard	read & write

See the full catalog with arguments in docs/tools.md.

Safety

• Dynamic risk classification per command, code snippet, path and process name (60+ patterns, OS-aware — see danger.py)
• Three modes: ask (default), auto (safe reads run unattended), yolo
• Allow / deny rules with globs and command prefixes (shell:git *, fs.write_file:~/Documents/*)
• Filesystem sandbox roots restricting where the agent may write
• Non-interactive / server mode fails closed
• Full audit log of every approved/denied action (~/.omnipilot/logs/audit.log)
• PyAutoGUI failsafe: slam the mouse into a screen corner to abort a GUI runaway

---

🤖 Example tasks

# Productivity
"Open Chrome, go to Gmail, and tell me how many unread emails I have"
"Find every PDF in my Downloads folder, move them into Documents\PDFs by year"
"Open the spreadsheet Q3-sales.xlsx and sum column D"

# Development
"Look at this repo, run the tests, fix whatever is failing, and commit"
"Create a FastAPI todo service in ./todo-api, install deps, run it, and screenshot it"
"Refactor every print() in this project into logging calls, then show me the diff"

# System administration
"Show me what's using all my RAM and close the worst offender"
"Install VLC with winget/brew/apt and pin it to my taskbar/dock"
"Rename all photos in this folder to the pattern YYYY-MM-DD_NNN.jpg"

# Research
"Search the web for the latest NVIDIA driver, download it to Downloads"
"Read the docs at <url> and write a quickstart example using requests"
"Set up a new Python project called inventory-service with pytest and ruff"

One-shot mode is great in scripts:

omnipilot --provider anthropic "compress every PNG in this folder that's over 2 MB"
omnipilot -p ollama -m llama3.1 --mode auto "summarize today's git log"

---

📦 Installation

You need Python 3.10 or newer (python.org, or winget install Python.Python.3.12 / brew install python / apt install python3 python3-pip python3-venv).

# Core install (everything except GUI automation extras)
pip install omnipilot-agent

# Everything — mouse/keyboard, notifications, system stats, web server
pip install "omnipilot-agent[all]"

Install from source (this repository):

git clone https://github.com/xrchris7/OmniPilot.git
cd OmniPilot
pip install -e ".[all]"          # or: make install-all

Optional extras: [gui] (pyautogui/pyperclip), [system] (psutil/plyer), [server] (FastAPI/uvicorn), [dev] (pytest/ruff).

Platform notes:

• Windows 11/10 — nothing extra needed; run PowerShell/cmd as yourself.
• macOS — grant Accessibility and Screen Recording permission to your terminal app (System Settings → Privacy & Security) for GUI/screen tools.
• Linux — screenshots/GUI need an X11/Wayland desktop plus sudo apt install python3-tk scrot (Wayland users may need ydotool/pipewire portal support).
• Headless/SSH servers: shell, Python, files, web and system tools all work; GUI/screen tools need a desktop session.

Detailed walkthrough: docs/installation.md.

---

🔑 Getting an API key

You only need a key for the provider(s) you use. Quick links:

Provider	Get a key
OpenAI	https://platform.openai.com/api-keys
Anthropic Claude	https://console.anthropic.com/settings/keys
Google Gemini (free tier!)	https://aistudio.google.com/app/apikey
NVIDIA NIM (free credits)	https://build.nvidia.com/explore/discover
Groq (free tier, very fast)	https://console.groq.com/keys
OpenRouter	https://openrouter.ai/keys
DeepSeek	https://platform.deepseek.com/api_keys
Mistral	https://console.mistral.ai/api-keys
xAI Grok	https://console.x.ai/
Together AI	https://api.together.xyz/settings/api-keys
Ollama (local, free)	https://ollama.com/download
LM Studio (local, free)	https://lmstudio.ai/

Then either export the variable…

export ANTHROPIC_API_KEY=sk-ant-...        # Windows PowerShell: $env:ANTHROPIC_API_KEY = "..."

…or copy .env.example to .env, fill in keys, and OmniPilot loads it automatically.

Run the built-in doctor at any time:

omnipilot doctor

---

⚡ Quickstart

No API key yet? Try the fully offline demo first (scripted agent, real tools, real approval flow):

omnipilot demo

Then connect a model:

# 1. Create a starter config (~/.omnipilot/config.yaml)
omnipilot config init

# 2. Pick a provider/model for this session and start the REPL
omnipilot --provider anthropic --model claude-sonnet-4-5-20250929
# …or…
omnipilot -p gemini -m gemini-2.0-flash
# …or fully local…
ollama pull llama3.1
omnipilot -p ollama -m llama3.1

You'll see the banner and a prompt. Try:

❯ What can you do on this machine?
❯ Take a screenshot and describe what's on my screen
❯ Create a folder called Reports on the Desktop and put a file in it that says hello
❯ Open Calculator and type 123 * 456
❯ /mode auto
❯ list all python processes
❯ /save
❯ /exit

Run one task and exit (great for automation):

omnipilot "create a 7-day markdown study plan for the AWS SAA exam in ./study-plan.md"

First run tip: keep --mode ask (the default). Watch the approval prompts; answer:

• y — yes, once
• n — no
• a — always allow this tool for the session
• e — always allow this exact action for the session

---

🔌 Supported LLM providers

Key	Provider	Vision	Key variable	Default base URL
openai	OpenAI GPT-4o / 4.1 / o-series	✅	OPENAI_API_KEY	api.openai.com
anthropic	Claude Opus/Sonnet/Haiku	✅	ANTHROPIC_API_KEY	api.anthropic.com
gemini	Google Gemini 2.x / 1.5 (REST)	✅	GEMINI_API_KEY	generativelanguage.googleapis.com
nvidia	NVIDIA NIM catalog (Llama, Qwen, Nemotron, vision models)	✅	NVIDIA_API_KEY	integrate.api.nvidia.com/v1
groq	Groq ultra-fast inference	✅	GROQ_API_KEY	api.groq.com/openai/v1
openrouter	Hundreds of models via one key	✅	OPENROUTER_API_KEY	openrouter.ai/api/v1
deepseek	DeepSeek V3/R1	❌	DEEPSEEK_API_KEY	api.deepseek.com
mistral	Mistral Large/Ministral	❌	MISTRAL_API_KEY	api.mistral.ai/v1
xai	Grok 2 / Grok vision	✅	XAI_API_KEY	api.x.ai/v1
together	Together AI catalog	✅	TOGETHER_API_KEY	api.together.xyz/v1
ollama	Local Ollama models	depends	—	localhost:11434/v1
lmstudio	Local LM Studio server	depends	—	localhost:1234/v1
custom	Any OpenAI-compatible endpoint	configurable	CUSTOM_API_KEY	CUSTOM_BASE_URL

Change provider/model in the REPL at any time:

/provider nvidia
/model meta/llama-3.3-70b-instruct

List models and even fetch a provider's live catalog:

omnipilot models
omnipilot models --provider nvidia --remote

Full per-provider setup: docs/providers.md.

---

🔌 MCP: use any Model Context Protocol server

OmniPilot is an MCP client. Connect external MCP servers and the model can call their tools exactly like the 41 built-ins — each one is named mcp.<server>.<tool> and flows through the same permission engine (risk modes, allow/deny rules, approval prompts, audit log). All three transports are supported: local stdio subprocesses, streamable HTTP, and legacy SSE.

pip install "omnipilot-agent[mcp]"

# ~/.omnipilot/config.yaml
mcp:
  enabled: true
  servers:
    fs:                                                  # -> mcp.fs.*
      command: npx
      args: ["-y", "@modelcontextprotocol/server-filesystem", "/home/me/projects"]
      risk: medium
    fetch:                                               # -> mcp.fetch.*
      url: https://example.com/mcp-server/mcp
      transport: streamable-http
      headers:
        Authorization: "Bearer ${MCP_TOKEN}"

Then inspect and smoke-test from the CLI:

omnipilot mcp list          # show configured servers, status and tool counts
omnipilot mcp test fs       # start the server and list its tools
omnipilot mcp test fs read_file '{"path": "README.md"}'   # call one tool

Remote-server failures never abort startup — the offending server is reported as error while every other server and the built-in tools keep working. Full guide: docs/mcp.md.

---

🛡️ The permission system (allowances)

Modes

Mode	Behavior
ask (default)	Prompts for everything not explicitly allowed; low-risk read-only tools auto-run if auto_approve_low: true
auto	Low-risk actions run automatically; medium/high/critical prompt
yolo	Nothing prompts — except the critical safety list (which still blocks unless you explicitly set allow_critical: true)

omnipilot --mode auto
omnipilot --yolo          # maximum autonomy, maximum responsibility

Rules (in ~/.omnipilot/config.yaml)

permission:
  mode: ask
  allow:
    - fs.read_file              # exact tool name
    - fs.*                      # globs
    - "shell:git status"        # exact-ish command prefix
    - "shell:npm run *"         # wildcard command prefixes
    - "python:print(*"          # python prefix
    - "fs.write_file:~/Documents/*"
  deny:
    - "shell:*format *"
    - "shell:*mkfs*"
  sandbox_paths:                # restrict writes to these roots
    - ~/projects
    - ~/Downloads
  allow_critical: false         # keep false!

• Deny rules win over allow rules; the built-in critical list wins over everything.
• Rules you grant during a session (a / e) are session-scoped and clearly logged.
• The server / --no-input mode cannot answer prompts, so unlisted risky actions are denied automatically — fail closed.
• Every decision is written to ~/.omnipilot/logs/audit.log.

Read more: docs/permissions.md and docs/safety.md.

---

🧰 Tool catalog

41 tools — click to expand the full list

Tool	What it does	Base risk
shell.run_command	Run a shell command (cmd/PowerShell/bash/zsh), sync or background	high
python.run_python	Persistent in-process Python interpreter	high
fs.read_file	Read a text file	low
fs.write_file	Create/overwrite/append files (creates parent dirs)	medium
fs.edit_file	Unique/find-replace edits with ambiguity protection	medium
fs.list_directory	Directory listing with sizes/dates	low
fs.make_directory	mkdir -p	low
fs.move_file	Move/rename	medium
fs.copy_file	Copy files or directories	low
fs.delete_file	Delete files/trees (permanent)	medium
fs.search_files	Recursive glob/name search	low
fs.grep	Search file contents (regex/literal)	low
apps.open_application	Launch apps by friendly name or path	low
apps.close_application	Quit apps (graceful/force)	medium
apps.open_file	Open a file/folder in its default app	low
apps.list_applications	Running apps/processes	low
browser.open_url	Open URLs/web searches in the default browser	low
web.fetch_url	Fetch a page and convert HTML → readable text	low
web.http_request	Any HTTP method with JSON/form bodies	medium
web.download_file	Stream a download to disk	medium
web.search_web	Keyless DuckDuckGo web search	low
screen.screenshot	Capture screen (→ vision model) as PNG	low
screen.get_screen_size	Screen resolution	low
screen.locate_image	Find a PNG template on screen	low
gui.click / gui.double_click / gui.right_click	Mouse clicks at x,y	medium
gui.move_mouse / gui.drag / gui.scroll	Pointer motion	medium
gui.type_text	Type (unicode-safe paste path)	medium
gui.press_key / gui.hotkey	Keys and chords (ctrl+c, alt+tab…)	medium
clipboard.get_clipboard / set_clipboard	Clipboard access	medium / low
processes.list_processes	Process table (CPU/mem)	low
processes.kill_process	Terminate by PID/name	high
system.get_system_info	OS/CPU/memory/disk/battery summary	low
system.get_environment_variable	Env vars with secret masking	medium
system.list_drives	Drives/volumes	low
notify.send_notification	Native toast notifications	low

Every tool also carries a full JSON-Schema argument description — inspect with omnipilot serve → GET /v1/tools, or type /tools in the REPL. Full argument reference: docs/tools.md.

---

🖥️ Command line reference

omnipilot [OPTIONS] [TASK ...]            # REPL, or one-shot when you type a task
omnipilot chat "your task" --provider ... # explicit one-shot command
omnipilot doctor                          # check keys, packages, OS features
omnipilot models [--provider X] [--remote]
omnipilot config init | path
omnipilot sessions list | resume <id>
omnipilot serve [--host 127.0.0.1] [--port 8765] [--mode auto]
omnipilot version

Key flags: -p/--provider, -m/--model, --mode {ask,auto,yolo}, --yolo, --safe, --vision {on,off,auto}, -c/--config, -w/--workdir, --max-iterations, -i/--image, -r/--resume, --system, --no-input.

REPL slash commands: /help /mode /yolo /safe /allow /clear /undo /provider /model /vision /tools /image /screenshot /save /load /sessions /usage /doctor /exit.

Full reference: docs/cli.md.

---

⚙️ Configuration

Resolution order (highest wins): CLI flags → environment variables → --config file → ./omnipilot.yaml → ~/.omnipilot/config.yaml → built-in defaults.

The fully commented reference is config.example.yaml. Highlights:

default_provider: openai
default_model: gpt-4o
vision: auto                 # auto = only send screenshots to vision-capable models

agent:
  max_iterations: 25
  temperature: 0.2
  tool_output_limit: 12000

permission: {mode: ask, sandbox_paths: [], allow_critical: false}

tools:
  enabled: ["*"]
  disabled: ["gui.*"]        # e.g. disable GUI automation entirely

providers:
  nvidia: {model: meta/llama-3.3-70b-instruct}
  ollama: {model: llama3.1, base_url: http://localhost:11434/v1}

Details: docs/configuration.md.

---

🌐 Local web/API server

pip install "omnipilot-agent[server]"
omnipilot serve --host 127.0.0.1 --port 8765

Open http://127.0.0.1:8765 for the built-in chat page, or use the JSON API:

curl -X POST http://127.0.0.1:8765/v1/agent/run \
  -H "Content-Type: application/json" \
  -d '{"message": "create a file hello.txt with content hi", "mode": "yolo"}'

{
  "ok": true,
  "answer": "I created hello.txt …",
  "session_id": "20260912-091500-ab12cd",
  "provider": "openai", "model": "gpt-4o",
  "tool_runs": 3, "input_tokens": 1840, "output_tokens": 212,
  "estimated_cost_usd": 0.0067
}

Prefer streaming? POST /v1/agent/stream returns a text/event-stream (SSE) feed emitting assistant_text, tool_started, tool_finished, tool_denied, info, error and a final done event — the bundled web UI consumes exactly this stream to render live tool activity.

Endpoints: GET /health, GET /v1/tools, POST /v1/agent/run, POST /v1/agent/stream, GET /v1/sessions, GET /v1/sessions/{id}. Interactive OpenAPI docs at /docs.

⚠️ Bind to 0.0.0.0 only on trusted networks; without a terminal the server fails closed — unresolved approvals are denied.

---

🐍 Using OmniPilot from Python

from omnipilot.config.settings import load_settings
from omnipilot.providers.registry import build_provider
from omnipilot.tools.registry import ToolRegistry
from omnipilot.permission.policy import PermissionPolicy
from omnipilot.agent.loop import Agent, NullUI

settings = load_settings()
registry = ToolRegistry.build_default(settings)
provider = build_provider(settings, "openai", "gpt-4o-mini")
policy = PermissionPolicy(settings, non_interactive=True,
                          prompt=lambda **kw: "y")   # auto-confirm in scripts
agent = Agent(
    settings=settings,
    provider=provider,
    provider_name="openai",
    registry=registry,
    policy=policy,
    model=provider.model,
    ui=NullUI(),
)

print(agent.run("Create a file notes.txt containing today's date."))
print("tokens:", agent.usage)

A complete scripted example ships in examples/use_as_library.py.

---

🧩 Writing your own tools (plugins)

A tool is a small class; the entry-point group omnipilot.tools auto-registers plugins.

# my_slack_tool.py
from omnipilot.tools.base import Tool, ToolContext, ToolResult

class SlackPost(Tool):
    name = "slack.post"
    description = "Post a message to a Slack channel via webhook."
    risk = "medium"
    parameters = {
        "channel": {"type": "string"},
        "text": {"type": "string"},
    }
    required = ["channel", "text"]

    def run(self, context: ToolContext, *, channel: str, text: str) -> ToolResult:
        import requests, os
        requests.post(os.environ["SLACK_WEBHOOK"],
                      json={"channel": channel, "text": text})
        return ToolResult.ok(f"Posted to {channel}")

Register it in your package's pyproject.toml:

[project.entry-points."omnipilot.tools"]
slack = "my_slack_tool:SlackPost"

See examples/custom_tool.py and docs/architecture.md.

---

🏛️ Architecture

flowchart TD
    U[User · REPL / CLI / Web API] --> CLI[cli.py · sessions · REPL]
    CLI --> AGENT[Agent loop · agent/loop.py]
    AGENT -->|normalized messages + tool schemas| AD[Provider adapter]
    subgraph Providers
        AD --> OAI[OpenAI-compatible<br/>OpenAI · NVIDIA · Groq · OpenRouter<br/>DeepSeek · Mistral · xAI · Together · Ollama]
        AD --> ANT[Anthropic Claude]
        AD --> GEM[Gemini REST]
    end
    AGENT --> POL{PermissionPolicy<br/>risk assessment · rules · prompts}
    POL -->|approved| TOOLS[ToolRegistry · 41 tools]
    TOOLS --> SHELL[shell / python]
    TOOLS --> FS[files / apps / browser]
    TOOLS --> GUI[screen / mouse / keyboard / clipboard]
    TOOLS --> WEB[web / processes / system / notify]
    TOOLS -->|mcp.&lt;server&gt;.&lt;tool&gt;| MCP[MCP client · stdio / HTTP / SSE]
    TOOLS -->|ToolResult + optional image| AGENT
    POL --> AUDIT[(audit.log)]
    AGENT --> SESS[(sessions/*.json)]

The agent uses a single internal message format; provider adapters translate to each vendor's dialect (OpenAI tools, Anthropic tool_use blocks, Gemini functionDeclarations), including multimodal screenshot messages. Adding a provider = one ~150-line adapter.

---

🧯 Safety & security

• OmniPilot can do anything your user account can do. Treat it like giving someone remote shell access — because it effectively is.
• Start in ask mode. Switch to auto only for routine work; reserve yolo for disposable VMs, containers or throwaway accounts.
• Don't paste long-lived secrets into tasks; env vars containing KEY/TOKEN/SECRET are masked when read by the agent and redacted from logs.
• Review ~/.omnipilot/logs/audit.log. Use sandbox_paths for untrusted tasks.
• Consider running inside a VM/container (Windows Sandbox, macOS VM, Docker with a desktop) for fully autonomous experiments.
• Report vulnerabilities privately — see SECURITY.md.

---

📚 Documentation index

Document	Contents
docs/installation.md	Per-OS install, extras, permissions, virtualenvs
docs/quickstart.md	Guided first session with examples
docs/providers.md	Every provider, key setup, model IDs, local models, custom gateways
docs/configuration.md	All YAML keys and env variables explained
docs/permissions.md	Modes, rule grammar, sandboxes, audit logs
docs/tools.md	Complete tool & argument reference
docs/mcp.md	Connecting MCP servers (stdio / HTTP / SSE), risk, troubleshooting
docs/cli.md	Every command, flag and slash command
docs/architecture.md	Internals, message format, writing adapters & plugins
docs/safety.md	Threat model, safe operating procedures
docs/faq.md	Troubleshooting

---

🗺️ Roadmap

• 13 providers + custom OpenAI-compatible endpoints
• 41 cross-platform computer-use tools incl. vision + GUI
• Risk-aware permission engine, rules, sandboxes, audit log
• Sessions, REPL, CLI, web UI + JSON API + SSE streaming, plugin entry points
• MCP client (stdio / streamable-HTTP / SSE), tools gated by the permission engine
• Offline omnipilot demo for zero-key evaluation
• Optional Docker / Windows-Sandbox execution backends
• MCP (Model Context Protocol) client for external MCP tools
• Voice I/O and accessibility-oriented automation
• Workflow recording (watch → generate a reusable script)
• Multi-agent planning and parallel tool execution
• Windows UI Automation / macOS Accessibility element tree (selector-based, coordinate-free)

Issues/PRs welcome — see CONTRIBUTING.md.

🤝 Contributing

Bug reports, new danger-pattern rules, provider adapters and tools all welcome. Quick local loop: make dev → make test → make lint. Please add tests and run ruff. By contributing you agree your work ships under the MIT license.

---

📄 License & disclaimer

MIT © 2026 xrchris7.

OmniPilot executes model-generated commands on real hardware. It is provided "as is", without warranty. You are responsible for reviewing actions, securing your keys, and for anything the agent does on your machines. It is intended for legitimate automation on systems you own or are authorized to administer.