Klyk

⚠️ What klyk is — and what it isn't

Read this before you install.

klyk gives an AI agent real, OS-level control of your Mac — the same input a human has. It moves the cursor, fires keystrokes, and clicks what's on screen. In its default mode it attempts invisible native input and acts autonomously once the agent decides to. Chromium interactions, command shortcuts and paste, menus and dialogs, and some long presses may activate the app or use visible input; klyk does not guarantee that every native action stays invisible.

This is powerful, and it is dangerous. Be clear-eyed about what that means:

• It can take real, irreversible actions. A click is a click. klyk can press Buy, Send, Confirm Transfer, Delete, or Sign just as you could. Klyk enforces checks such as target-window bounds, ownership, duplicate-label rejection, and its emergency latch, while confirmation guidance and a bounds override remain agent-cooperative and do not establish user approval. There is no sandbox and no spending limit.
• It runs with your full user privileges. Anything you can do on your Mac, klyk can do. It does not isolate itself or drop privileges.
• It is a prompt-injection target. If the agent driving klyk also reads untrusted content — a web page, an email, a document — a malicious instruction hidden there can become real clicks and keystrokes on your machine. "Reads the web" + "controls the Mac" is the high-risk combination. Run klyk only with an agent and a workflow you trust.
• It relies on an undocumented Apple API. Invisible native input uses Apple's private SkyLight framework. Apple does not support or guarantee it; a macOS update can change or break it without notice, and the affected actions may fall back to visible input or require activation.

What klyk is, honestly: an early, experimental, open-source tool built by a solo author — a business student, not a professional developer — working with AI, in good faith. It has not had an independent professional security audit. Core paths are live-tested end-to-end against a real macOS session before every release, but treat it as early software: less-common paths may hold surprises.

No warranty. Use at your own risk. klyk is provided "as is" under the MIT license, with no warranty of any kind. You are responsible for what the agent does on your machine. Don't point it at anything — money, accounts, irreplaceable files — you aren't willing to have an autonomous agent touch.

OS-level macOS app testing for AI agents. Click real buttons, type real keys, see what actually rendered. Native apps, Electron apps, browsers, web pages, system dialogs — anything visible on the screen.

Status: Portfolio project. Showcases product thinking and shipped tooling. Bug reports won't be actively triaged. Well-scoped PRs are welcome — but expect a slow review cadence.

The problem

AI assistants are increasingly asked to test, validate, or operate desktop apps end-to-end. Today they can't. Existing automation tools either require deep app instrumentation (XCUITest, Appium) or simulate user input at a layer too brittle to be trusted (pixel-only click frameworks, headless DOM scrapers). The result: agents that can write apps faster than ever, but can't verify they actually work.

Klyk closes that gap. It gives an AI agent the same input channel a human has — real cursor moves via Apple's CoreGraphics API, real keystrokes posted to the HID event tap, real composited screenshots — and a clean MCP interface to drive it. Native input attempts to stay invisible by default, with the activation and visible-input cases described above. The agent observes, decides, acts, verifies.

What it does

> screenshot the app, then click "Sign in"
[ Klyk takes a real screenshot via CoreGraphics, returns it + the AX tree ]
[ click_element finds "Sign in" via accessibility, falls back to OCR, then to template match ]
[ Real click fires through the HID event tap ]

A flat, MECE tool surface across observation, interaction, evaluation, session management, and system operations. Three-tier click targeting (AX → on-device OCR → pixel template) so a label is reachable regardless of how the app exposes it. Cross-app drag, right-click-then-select, and multilingual OCR are all first-class. Per-call latency + reasoning-gap metrics so the agent can self-pace. Best-effort AX folded into screenshots so most tasks finish in one round-trip.

Install

pipx install klyk        # isolated install — recommended
klyk install

Use pipx (or uv tool install klyk), not bare pip. Klyk accepts NumPy >=1.24; installing it into your global Python can still clash with other packages pinned to different versions. pipx/uv give klyk its own environment while still putting klyk and klyk-call on your PATH — same commands, zero blast radius. Plain pip install klyk works if you want it in the current environment and accept that risk.

klyk install is a turnkey first-run flow:

1. Adds Klyk to ~/.claude.json (so it appears in every Claude Code session).
2. Walks you through granting the two macOS permissions Klyk needs — opens the exact System Settings panes for Accessibility and Screen Recording, waits for you to add your terminal app, then verifies the grant actually came through before continuing.
3. Runs a final klyk doctor pass to confirm every piece is green.
4. Detects every other AI client on your Mac and offers to wire them all in one confirmation — permissions carry over, so it's a single prompt, not a setup pass per client.

For clients that read a natural-language context file (Gemini CLI's GEMINI.md), install also offers — opt-in, defaults to no — to add a short, clearly-marked klyk note there, so the agent can fall back to the klyk-call shell if its own MCP ever fails to surface klyk. It never edits that file without an explicit yes, merges around your existing content, and uninstall removes it.

Restart Claude Code (or whichever MCP client you use) and klyk is live. Try inspect Finder to see it in action. To wire every detected client up front in one shot: klyk install --all.

Troubleshooting: klyk doctor. Run it any time something's off. Reports every dependency, permission, and config grant klyk needs as ✓ / ⚠ / ✗ with the exact next step on anything that's not green. --json gives a structured payload for tooling.

Staying up to date

klyk update

It detects how klyk was installed (pipx, uv tool, or plain pip), runs the matching upgrade, and then restarts the running klyk server automatically — every connected client that shares that installation loads the new version on its next tool call. Clients using pinned, bundled, or separate klyk installations need their own update.

You never have to wonder whether you're behind, either:

• klyk doctor includes a klyk version line — 0.2.0 (latest release) or 0.2.0 → 0.3.0 available with the exact command to run.
• The menu-bar eye shows a one-line ⬆ Update available notice when a newer release exists.
• klyk update --check reports whether an update exists without changing anything.

Behind these sits a single once-a-day HTTPS check of klyk's own PyPI metadata, cached in ~/.klyk/update_check.json and fully offline-safe. The check sends no screen data or captured content, but the normal network request exposes metadata such as your IP address to PyPI and the network path. Set KLYK_UPDATE_CHECK=0 to disable it entirely.

One driver at a time. Any number of MCP clients can have klyk configured and connected simultaneously (Claude Code and OpenCode and Cursor…) — each spawns its own klyk process, and none is ever refused. But only one session holds the control token and actually drives the Mac at a time, so two agents can never interleave clicks and keystrokes into the same app. A new session takes control automatically when the previous driver is gone; taking over from a live driver is an explicit take_control call. Exactly one menu-bar eye is visible: the active driver's.

Use with other MCP clients

klyk install <client> auto-configures any supported client — same turnkey flow as Claude (writes the config, grants permissions, runs a health check):

klyk install opencode    # or: cursor · windsurf · continue · cline · codex · gemini · antigravity (agy) · grok
klyk install --list      # show every supported client and its config path

For OpenCode, setup and verification are two commands:

klyk install opencode
opencode mcp list        # klyk should show connected

Klyk writes OpenCode's global local-MCP entry, so it is available in every workspace and with every provider/model that supports tool use. Existing JSON/JSONC comments, trailing commas, formatting, permissions, providers, and other MCP servers are preserved; re-running the installer is safe, and klyk uninstall opencode removes only klyk. The installer finishes by running OpenCode's own MCP status check, catches launcher-specific macOS permission gaps, and does not declare success until OpenCode reports klyk connected. OpenCode normally reloads config changes automatically.

Any other MCP client works too — klyk speaks MCP natively. Add this entry to its config wherever it lives — use the full path to the Python klyk is installed in as command (run python -c "import sys;print(sys.executable)" in that env; klyk install fills this in automatically). A bare python3 only works if klyk is in your global Python:

{ "mcpServers": { "klyk": { "command": "/path/to/python", "args": ["-m", "klyk.mcp_server"] } } }

Permissions, the singleton lock, and klyk doctor work identically regardless of which client launches klyk.

Choosing a model — the speed vs. intelligence trade-off

klyk runs at the same OS speed no matter what's driving it — the latency and accuracy you feel are the model's, not klyk's. Because klyk is model-agnostic, you pick where you want to sit on a very real trade-off:

• Frontier models (e.g. Claude Opus). Long gaps between actions — sometimes minutes while the model reasons — but each action is usually well-chosen, correctly targeted, and efficient. Fewer wrong clicks, fewer wasted round-trips. Best for high-stakes or irreversible work where a misfire is expensive.
• Fast, smaller models (e.g. Gemini Flash). Often under ~10 seconds between actions — snappy and cheap — but they misfire more: wrong element, wrong order, premature verdict. They lean harder on klyk's observe→act→verify loop to catch and recover from mistakes. Best for fast, iterative, low-stakes tasks where throughput beats precision.

There's no universally "right" model — it's a deliberate choice per task. Match the model to the cost of being wrong: the smarter-but-slower model when an errant click matters, the faster-but-looser model when speed and volume matter and recovery is cheap.

Drive klyk from any AI (no MCP integration required)

MCP support varies a lot between agent harnesses — some gate it, some implement it incompletely, some don't have it. So klyk ships an extra front door that works even when a client's own MCP plumbing doesn't. It reaches the same persistent klyk session.

klyk-call — one shell command, any tool. For any agent that can run a shell command (great for smaller models — no handshake to reason about):

klyk-call --list                              # all tools + their parameter names
klyk-call --schema inspect                    # full JSON schema for one tool
klyk-call --tool inspect --app Finder         # call any tool
klyk-call --tool screenshot --app Finder      # screenshot → saved to disk, path returned
echo '{"tool":"screen_info","args":{}}' | klyk-call --batch   # many calls, one session

Vision over the shell. When a tool returns a screenshot (screenshot, inspect, verdict, image-producing run steps), klyk-call writes the PNG to ~/.klyk/captures/ and returns its saved_path instead of dumping base64 — so the agent views the capture with its own image reader (Claude Code's file read, Gemini's @path, etc.) and keeps the same observe→act→verify loop the native MCP transport has, with no context-flooding payload. The cache keeps the most recent 20 captures.

Quick example

screenshot(app="Google Chrome")
# → returns the image plus the AX element list

run(app="Google Chrome", actions=[
    {"tool": "click", "x": 580, "y": 389},
    {"tool": "fill_field", "x": 580, "y": 389, "text": "This video is incredible!"},
    {"tool": "screenshot"}
])
# → executes the full sequence at OS speed, returns one batched response

verdict(app="Google Chrome", test_description="Posted a comment on a YouTube video")
# → returns final screenshot + logs + grading criteria for the agent to synthesize PASS/FAIL

Why these specific trade-offs

The interesting product decisions weren't tools to build but tools to not build:

• No third-party computer-use libraries. Everything runs through Apple's CoreGraphics, Vision, and Accessibility frameworks via Python's ctypes. Keeps the dependency footprint tiny and the failure surface predictable.
• No visual grounding models. UI-TARS and OmniParser would have closed the "no AX, no text" gap — at the cost of a multi-GB model download and 1–2s per call. Rejected. Template matching covers the same case at 30ms with zero ML dependencies.
• No Chrome DevTools Protocol. It would have helped only Chromium browsers and broken the "like a human" model. Skipped in favor of forcing the renderer-accessibility flag, which gets the entire web AX tree for free.
• macOS only. Cross-platform compromises every primitive. The honest framing: ship one OS well rather than three OSes badly.

Every tool is designed against the same set of failure modes — ambiguity, accidental retries, token bloat, lost reactivity from batching, and so on. The agent-facing contract for each tool lives in its description field in klyk/mcp_server.py.

What's inside

For the full tool reference and behavior contracts, see the tool description fields in klyk/mcp_server.py. For how the internals are shaped and why, see ARCHITECTURE.md.

Security model & trust scope

Klyk is a thin pipe between the agent and the OS. Its trust model is straightforward:

• Local control with an optional metadata check. Klyk does not send screenshots, OCR results, AX labels, or captured screen data in its optional once-daily HTTPS request to PyPI. The request still exposes ordinary network metadata, including your IP address, to PyPI and the network path. Results returned over stdio to the calling agent or client may then be transmitted to that client's model provider under its own settings. Disable the check with KLYK_UPDATE_CHECK=0 and klyk makes no network calls itself.
• macOS permissions are the consent surface. Accessibility and Screen Recording must be granted explicitly via System Settings; klyk doctor shows the current state.
• The agent controls the intent. Klyk doesn't decide what to click or type — it executes what the agent asks, subject to code-enforced ownership, target-window bounds, duplicate-label rejection, and the emergency latch. Run klyk only with agents you trust to act on your behalf.
• Stderr from launched apps is captured for the verdict payload. klyk attempts to scrub common credential patterns (passwords, API keys, JWTs, AWS keys, bearer tokens) on a best-effort basis — it cannot catch every format, so do not rely on it as your only safeguard.
• Private framework usage. Klyk uses Apple's private SkyLight framework for attempted invisible native input delivery (the "autonomous" mode); some browser, shortcut, menu, dialog, and long-press paths may activate or use visible input. This is fine for CLI / PyPI distribution but is the reason klyk can't ship via the Mac App Store.
• Reporting a vulnerability. See SECURITY.md.

License

MIT — see LICENSE.

Disclaimer — No Warranty

klyk is provided "AS IS", without warranty of any kind, express or implied, including merchantability, fitness for a particular purpose, and non-infringement (see LICENSE). To the maximum extent permitted by law, the author is not liable for any damage, data loss, financial loss, account action, privacy exposure, or other harm arising from the use, misuse, or malfunction of klyk — whether caused by the software, the AI agent driving it, or a third-party dependency or macOS framework it relies on. Credential scrubbing is best-effort, and confirmation guidance is agent-cooperative; code-enforced bounds, ownership, duplicate-label checks, and the emergency latch do not guarantee safety or approve consequential actions. You run klyk at your own risk and are solely responsible for what you connect it to and what it does.

Designed and shipped using AI as implementation partner. The product decisions, scope choices, and trade-offs are mine.