planchette 0.1.0

Point at any window and let a coding agent drive it.

planchette

Point at any window, then let a coding agent drive it — click, type, screenshot, and control another app's window. No API key: the agent (Claude Code, Codex, Cursor, …) is the loop, through a small CLI of window-control primitives.

Platforms

• macOS — the polished path: per-window capture, live-overlay picker.
• Windows / Linux (X11) — portable backend (mss + pynput + pywinctl); select windows with pick --name / pick --list instead of the overlay.
• Wayland — unsupported (it blocks synthetic input and capture by design).

Requires Python ≥3.10.

Install

# 1. Put the `planchette` CLI on PATH
uv tool install .
#    (ensure ~/.local/bin is on PATH; `uv tool update-shell` sets this up)

# 2a. Claude Code: add this repo as a plugin (from the repo root)
#     In Claude Code:  /plugin  → add from this directory
# 2b. Any MCP-capable agent (Codex, Cursor, …): register `planchette mcp`
#     — see "MCP mode" below.
# 2c. Any other agent tool: point it at AGENTS.md in this repo —
#     it documents the capture → look → act loop.

MCP mode — any MCP-capable agent

planchette mcp serves the same primitives over MCP (stdio). Screenshots come back inline in the tool result, so the agent needs no file access — this is the recommended door for Codex (no sandbox bypass needed), Cursor, and anything else that speaks MCP.

agent	register with
Claude Code	claude mcp add planchette -- planchette mcp
Codex	codex mcp add planchette -- planchette mcp
Gemini CLI	gemini mcp add planchette planchette mcp
Cursor	add to .cursor/mcp.json: {"mcpServers": {"planchette": {"command": "planchette", "args": ["mcp"]}}}
opencode	add to opencode.json: {"mcp": {"planchette": {"type": "local", "command": ["planchette", "mcp"]}}}

Then ask the agent to pick a window (pick_window, or pick_window name="Telegram") and drive it. macOS permissions (Screen Recording + Accessibility) must be granted to whatever app hosts the agent.

Skill mode — agents that read SKILL.md / AGENTS.md

The skills/control-window/ folder is a standard Agent Skill: the same folder works in Claude Code, Codex, Cursor, and other SKILL.md readers.

agent	install
Claude Code	/plugin → add this repo (also brings the /planchette command)
Codex	cp -r skills/control-window ~/.codex/skills/ (or .codex/skills/ in a project)
other SKILL.md readers	copy skills/control-window/ into the agent's skills directory
anything else	point the agent at AGENTS.md — it documents the whole loop

Required macOS permissions

Grant these to the app you run your agent from (Terminal, iTerm, VS Code…), in System Settings → Privacy & Security:

1. Screen Recording — to capture the target window (else screenshots are black).
2. Accessibility — to move the mouse, click, type, and raise windows.

After granting, fully quit and reopen the terminal app.

Use

In Claude Code:

/planchette open a new tab and search for the weather

…or just ask in chat: "control my Telegram window and message Saved Messages 'hi'." Claude runs the picker (hover + click the window you want), then loops: screenshot → decide → act, until the goal is done.

• Background control (macOS): input is delivered straight to the target app's process — the window needn't be frontmost, your focus and cursor stay put, and you can keep chatting with Claude in the terminal while it drives. (Exceptions: modifier combos briefly activate the target and hand focus back; scroll hops the cursor there and back.) On Windows/Linux input follows real focus — don't fight it while it runs.
• One display in v1.

CLI (what the agent drives)

planchette pick                    # live-overlay picker (macOS) → select a window
planchette pick --list             # print candidate windows
planchette pick --name "Telegram"  # select frontmost match by app/title substring
planchette capture [--out PATH]    # screenshot it → PNG, prints "PATH  WIDTHxHEIGHT"
planchette capture --crop X Y W H  # native-res zoom of a region (for small text)
planchette click X Y [--double] [--right]  # X Y are captured-image pixels
planchette drag X1 Y1 X2 Y2        # select text, sliders, drag & drop
planchette move X Y
planchette type "text"
planchette key "cmd+t" [--repeat N]
planchette scroll X Y up|down [N]
planchette raise
planchette status
planchette agent [--agent claude|gemini]  # macOS: pick a window → floating panel drives it
planchette mcp                     # serve window control over MCP (stdio)
planchette hotkey [--combo C]      # macOS: global hotkey (default ctrl+cmd+p) → agent panel
planchette hotkey --install        # …as a login LaunchAgent; --uninstall removes it

Window bounds are re-read from the live window before every action, so a window that moved after pick still gets clicked in the right place.

Global hotkey → agent panel (macOS)

planchette hotkey --install registers ctrl+cmd+P system-wide (like the ⌘⇧5 screenshot overlay) and keeps it across logins. Pressing it runs planchette agent: pick a window under the overlay, then a small floating panel appears — type a goal ("reply ok to the last message") and a headless agent session (claude or gemini, whichever is installed — force one with planchette agent --agent gemini) drives that window, streaming its actions into the panel. Enter again for follow-ups in the same session; Esc closes it. A dropdown on the input row lists the installed agent CLIs — switch anytime between turns; switching starts a fresh conversation for the new agent.

The spawned session is scoped to window control only (claude: --allowedTools "Bash(planchette:*)" Read; gemini: a generated ~/.planchette/.gemini/settings.json capping its tools the same way). The agent CLI and planchette must be installed; the panel finds them on PATH plus the usual install dirs (~/.local/bin, /opt/homebrew/bin, /usr/local/bin). Gemini also needs auth the spawned process can see, e.g. GEMINI_API_KEY in ~/.gemini/.env.

Note: the LaunchAgent runs outside your terminal, so macOS asks for Accessibility again — grant it to the listed Python in System Settings → Privacy & Security; the daemon retries on its own. Pre-picking for a terminal session still works: pick via the hotkey, then just Esc the panel.