Skip to main content

som-window-mcp

Read-only eyes for Windows desktop UI. Let an AI agent see a window — including one covered by other windows — without guessing from full-screen pixels, and without giving it a mouse.

Status: v0.3 — five read-only tools: window list, accessibility tree, single-element inspection, plain capture, and the numbered Set-of-Mark overlay.

Explorer with every control numbered, captured while covered by other windows

One window, captured while it was covered by another. The marks come from the accessibility tree, not from OCR, so every number maps to an exact screen rect.

Why

When you ask an agent to help with a desktop app, it has two bad options today:

  1. Screenshot the whole screen. Expensive in tokens, and it captures your private windows along with the target one. If the app is behind your terminal, the agent cannot see it at all.
  2. Install a full desktop clicker. That hands the model a mouse and a keyboard. Any text on screen can then talk the model into pressing Win+R and running something. It also tends to hang on blocked Windows calls.

This server takes a third path: look, never touch. It reads the real Windows accessibility tree for structure, and PrintWindow for pixels, so a covered window still captures cleanly and only that window is in the image.

Tools

Tool What it returns
list_windows Visible windows: hwnd, process, physical-pixel rect, whether the window is covered (occluded), whether it needs an elevated server (elevated)
get_ui_tree Compact text list of interactive elements — path, type, name, automation id, screen rect — with zoomable max_depth / max_nodes
inspect_element Full content of one element by path: value, bounded text, and a summary of its immediate children
capture_window One window's pixels, even while covered, plus the screen position of image pixel (0,0)
capture_som The same capture with a numbered box on every interactive element, plus a legend that maps each number to a real screen rect

There is no click, no typing, no mouse movement, and no code execution anywhere in this package.

Set-of-Mark

capture_som draws a numbered box on each interactive control. The boxes come from the accessibility tree, not from OCR or a detector, so they are pixel-exact and the legend carries real coordinates:

 7   0.4.1  Button  "Открыть SDS"  @450,320 90x28

A vision model answers "7" instead of estimating a pixel; a clicker uses the rect and never looks at the image at all. Numbering follows reading order (top-to-bottom, left-to-right), and the tree is walked breadth-first so a fixed node budget reaches toolbars and lists before it disappears into one menu.

Three filters keep the overlay readable:

  • duplicates — Windows Terminal exposes its caption buttons twice; overlapping controls of similar size are collapsed to the named one
  • cells inside rows — Explorer reports four column values per list row; the row is marked, its cells are not
  • budget — max_marks caps the count, and badge placement avoids collisions (verified: zero overlapping badges on a 60-mark Explorer window)

Requirements

  • Windows 10/11 (x64)
  • Python 3.10+

Install

Not on PyPI yet, install from the repository:

git clone https://github.com/Nikita-Pugach/som-window-mcp
cd som-window-mcp
uv tool install .        # or: pip install .

Or straight from git, without cloning:

pip install "git+https://github.com/Nikita-Pugach/som-window-mcp"

Configure

OpenCode

opencode mcp add som-window --global -- som-window-mcp

Or by hand in opencode.jsonc:

{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "servers": {
      "som-window": {
        "type": "local",
        "command": ["som-window-mcp"],
      },
    },
  },
}

Claude Code

claude mcp add som-window -- som-window-mcp

Any other MCP client

{
  "mcpServers": {
    "som-window": {
      "command": "som-window-mcp"
    }
  }
}

Running from a checkout instead of an installed package:

{
  "mcpServers": {
    "som-window": {
      "command": "python",
      "args": ["-m", "som_window_mcp.server"],
      "env": { "PYTHONPATH": "C:\\dev\\som-window-mcp\\src" }
    }
  }
}

Coordinate contract

Everything is in physical screen pixels, and the server sets PER_MONITOR_AWARE_V2 at startup, so coordinates match the screen on 100%, 125%, 150% and mixed-DPI multi-monitor setups.

capture_window reports origin: the screen coordinate of image pixel (0,0).

screen_x = origin_x + pixel_x
screen_y = origin_y + pixel_y

That is what a clicker needs, and it is the reason the capture is cropped by DWMWA_EXTENDED_FRAME_BOUNDS: without the crop every image carries a 9 px black frame from the invisible DWM resize border and every coordinate is off.

Known limitations

Measured, not guessed — see docs/uia-matrix.md for the raw numbers.

Case Behaviour
Elevated window, non-elevated server Windows blocks UIA across integrity levels. The tool says so explicitly instead of returning an empty tree. Run the server elevated to read it.
Chrome, VS Code, Slack, Discord (Chromium/Electron) No UI tree at all unless the app was started with --force-renderer-accessibility (8 nodes vs 173 measured). Screenshots work either way.
Qt / Delphi apps Partial or absent UIA tree. Screenshots still work.
Minimised windows, some GPU surfaces PrintWindow returns a blank frame; this is detected and reported instead of returning a useless image.
Dense windows About 2.5 ms per UIA node. Use max_nodes; a 4000-node walk takes ~10 s.

Read-only still means "sees your screen": captures are sent to your model provider. Use the window list to target a specific window, and keep password managers and banking apps out of the loop.

How it works

  • win32.py — DPI awareness, window enumeration (DWM cloaked filter), PrintWindow(PW_RENDERFULLCONTENT), BGRX decode to dodge the zero-alpha bug.
  • uia_worker.py — UI Automation runs in a separate process; a blocked COM call cannot be cancelled by a thread timeout, so the parent kills the process.
  • worker.py — persistent worker with hard timeout and transparent respawn (cold-importing uiautomation costs 1–2 s per call, so it stays alive).
  • inspector.py — window resolution and terse formatting. Ambiguous matches are an error, never a silent first hit.
  • overlay.py — Set-of-Mark: de-duplication, containment suppression, reading order, collision-free badge placement.
  • capture.py — blank-frame detection, downscale, smaller of PNG/JPEG.

Roadmap

  • 0.4 — optional actions (--enable-actions) in a separate package, so this core stays provably read-only
  • 0.5 — publish to PyPI, make uvx som-window-mcp the install line

Development

pip install -e ".[dev]"
pytest

The unit tests cover window resolution, formatting, overlay selection and encoding. They do not touch Win32, so they run in a quarter of a second.

License

MIT

Metadata

Release files for som-window-mcp 0.3.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for som-window-mcp 0.3.0
File Size Uploaded Provenance Actions
som_window_mcp-0.3.0.tar.gz 125.5 kB Without provenance Details for som_window_mcp-0.3.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for som-window-mcp 0.3.0
File Interpreter ABI Platform Actions
som_window_mcp-0.3.0-py3-none-any.whl Python 3 none any Details for som_window_mcp-0.3.0-py3-none-any.whl

Total release size: 152.9 kB

Release files / som_window_mcp-0.3.0.tar.gz

Download URL som_window_mcp-0.3.0.tar.gz
Size 125.5 kB
Tags Source
SHA-256 checksum
How to use checksums
6b79c4b7d6f2529470b7194e0f9043ba6fa66eeaf183b1d02d118d1c41071852
BLAKE2b-256 checksum
How to use checksums
a837e7a8fb733baf5aab16fa0eac62ff7bd7650163afe023263b52c376be7328
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.9 {"installer":{"name":"uv","version":"0.13.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / som_window_mcp-0.3.0-py3-none-any.whl

Download URL som_window_mcp-0.3.0-py3-none-any.whl
Size 27.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
81356bc85f634cfadf5b77c66e41999784575aabcdd6ad0af8bf923964f7fd6d
BLAKE2b-256 checksum
How to use checksums
2605edb39e626f31a1348b486fa46896d2a4134b429217c3265e868c92adf7c9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.9 {"installer":{"name":"uv","version":"0.13.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for som-window-mcp 0.3.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.3.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page