som-window-mcp
Read-only eyes for Windows desktop UI. Let an AI agent see a window — including one covered by other windows — without guessing from full-screen pixels, and without giving it a mouse.
Status: v0.3 — five read-only tools: window list, accessibility tree, single-element inspection, plain capture, and the numbered Set-of-Mark overlay.
One window, captured while it was covered by another. The marks come from the accessibility tree, not from OCR, so every number maps to an exact screen rect.
Why
When you ask an agent to help with a desktop app, it has two bad options today:
- Screenshot the whole screen. Expensive in tokens, and it captures your private windows along with the target one. If the app is behind your terminal, the agent cannot see it at all.
- Install a full desktop clicker. That hands the model a mouse and a
keyboard. Any text on screen can then talk the model into pressing
Win+Rand running something. It also tends to hang on blocked Windows calls.
This server takes a third path: look, never touch. It reads the real Windows
accessibility tree for structure, and PrintWindow for pixels, so a covered
window still captures cleanly and only that window is in the image.
Tools
| Tool | What it returns |
|---|---|
list_windows |
Visible windows: hwnd, process, physical-pixel rect, whether the window is covered (occluded), whether it needs an elevated server (elevated) |
get_ui_tree |
Compact text list of interactive elements — path, type, name, automation id, screen rect — with zoomable max_depth / max_nodes |
inspect_element |
Full content of one element by path: value, bounded text, and a summary of its immediate children |
capture_window |
One window's pixels, even while covered, plus the screen position of image pixel (0,0) |
capture_som |
The same capture with a numbered box on every interactive element, plus a legend that maps each number to a real screen rect |
There is no click, no typing, no mouse movement, and no code execution anywhere in this package.
Set-of-Mark
capture_som draws a numbered box on each interactive control. The boxes come
from the accessibility tree, not from OCR or a detector, so they are pixel-exact
and the legend carries real coordinates:
7 0.4.1 Button "Открыть SDS" @450,320 90x28
A vision model answers "7" instead of estimating a pixel; a clicker uses the rect and never looks at the image at all. Numbering follows reading order (top-to-bottom, left-to-right), and the tree is walked breadth-first so a fixed node budget reaches toolbars and lists before it disappears into one menu.
Three filters keep the overlay readable:
- duplicates — Windows Terminal exposes its caption buttons twice; overlapping controls of similar size are collapsed to the named one
- cells inside rows — Explorer reports four column values per list row; the row is marked, its cells are not
- budget —
max_markscaps the count, and badge placement avoids collisions (verified: zero overlapping badges on a 60-mark Explorer window)
Requirements
- Windows 10/11 (x64)
- Python 3.10+
Install
uvx som-window-mcp
Or install the binary once:
uv tool install som-window-mcp
uv tool install places som-window-mcp in ~/.local/bin; run
uv tool update-shell once if the command is not found afterwards.
With pip:
pip install som-window-mcp
From a clone, for development:
git clone https://github.com/Nikita-Pugach/som-window-mcp
cd som-window-mcp
uv tool install .
Configure
OpenCode
opencode mcp add som-window --global -- som-window-mcp
Or by hand in opencode.jsonc:
{
"$schema": "https://opencode.ai/config.json",
"mcp": {
"servers": {
"som-window": {
"type": "local",
"command": ["som-window-mcp"],
},
},
},
}
Claude Code
claude mcp add som-window -- som-window-mcp
Any other MCP client
{
"mcpServers": {
"som-window": {
"command": "som-window-mcp"
}
}
}
Running from a checkout instead of an installed package:
{
"mcpServers": {
"som-window": {
"command": "python",
"args": ["-m", "som_window_mcp.server"],
"env": { "PYTHONPATH": "C:\\dev\\som-window-mcp\\src" }
}
}
}
Coordinate contract
Everything is in physical screen pixels, and the server sets
PER_MONITOR_AWARE_V2 at startup, so coordinates match the screen on 100%,
125%, 150% and mixed-DPI multi-monitor setups.
capture_window reports origin: the screen coordinate of image pixel (0,0).
screen_x = origin_x + pixel_x
screen_y = origin_y + pixel_y
That is what a clicker needs, and it is the reason the capture is cropped by
DWMWA_EXTENDED_FRAME_BOUNDS: without the crop every image carries a 9 px black
frame from the invisible DWM resize border and every coordinate is off.
Known limitations
Measured, not guessed — see docs/uia-matrix.md for the
raw numbers.
| Case | Behaviour |
|---|---|
| Elevated window, non-elevated server | Windows blocks UIA across integrity levels. The tool says so explicitly instead of returning an empty tree. Run the server elevated to read it. |
| Chrome, VS Code, Slack, Discord (Chromium/Electron) | No UI tree at all unless the app was started with --force-renderer-accessibility (8 nodes vs 173 measured). Screenshots work either way. |
| Qt / Delphi apps | Partial or absent UIA tree. Screenshots still work. |
| Minimised windows, some GPU surfaces | PrintWindow returns a blank frame; this is detected and reported instead of returning a useless image. |
| Dense windows | About 2.5 ms per UIA node. Use max_nodes; a 4000-node walk takes ~10 s. |
Read-only still means "sees your screen": captures are sent to your model provider. Use the window list to target a specific window, and keep password managers and banking apps out of the loop.
How it works
win32.py— DPI awareness, window enumeration (DWM cloaked filter),PrintWindow(PW_RENDERFULLCONTENT),BGRXdecode to dodge the zero-alpha bug.uia_worker.py— UI Automation runs in a separate process; a blocked COM call cannot be cancelled by a thread timeout, so the parent kills the process.worker.py— persistent worker with hard timeout and transparent respawn (cold-importinguiautomationcosts 1–2 s per call, so it stays alive).inspector.py— window resolution and terse formatting. Ambiguous matches are an error, never a silent first hit.overlay.py— Set-of-Mark: de-duplication, containment suppression, reading order, collision-free badge placement.capture.py— blank-frame detection, downscale, smaller of PNG/JPEG.
Roadmap
0.4— optional actions (--enable-actions) in a separate package, so this core stays provably read-only0.5—wait_for_element, for UIs that settle slowly after a click
Development
pip install -e ".[dev]"
pytest
The unit tests cover window resolution, formatting, overlay selection and encoding. They do not touch Win32, so they run in a quarter of a second.
License
MIT
Metadata
Release files for som-window-mcp 0.3.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| som_window_mcp-0.3.1.tar.gz | 127.7 kB | With provenance | Details for som_window_mcp-0.3.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| som_window_mcp-0.3.1-py3-none-any.whl | Python 3 | none | any | Details for som_window_mcp-0.3.1-py3-none-any.whl |
Total release size: 156.0 kB
Release files / som_window_mcp-0.3.1.tar.gz
| Download URL | som_window_mcp-0.3.1.tar.gz |
|---|---|
| Size | 127.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
888deae13f45e7ca5b3c967eca73a63f8adce1d0883a47e776689fd0611b6c62
|
|
BLAKE2b-256 checksum How to use checksums |
5cb5a047b2e2c242f00e73a6fdc6f4d9bdd2ef72cd7288d5a95a535c77f13e2c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.
Transparency logRelease files / som_window_mcp-0.3.1-py3-none-any.whl
| Download URL | som_window_mcp-0.3.1-py3-none-any.whl |
|---|---|
| Size | 28.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
43bd2de3ad95e0df74201e7ad68497d7b3534f79a28c5d956c937ad160a49483
|
|
BLAKE2b-256 checksum How to use checksums |
b793357c1311f44c1b35827e72d34764ffbee2032832c2bc01dba8584d54dee9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.
Transparency logSecurity for som-window-mcp 0.3.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for som_window_mcp-0.3.1.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.