OmniSight
繁體中文說明:README.zh-TW.md
Computer use for every Windows app window, for any MCP agent: GPU-rendered, covered or in the background.
Agents that drive a desktop through screenshots keep hitting the same walls on Windows:
- GPU-rendered windows (games, emulators, Chromium / Electron, UWP) come back black from classic GDI capture.
- A window that is covered or in the background is not in a full-screen screenshot at all.
- Every step sends a full-screen image to a vision model just to find one button, which is slow and expensive.
- Clicks land in the wrong place on scaled and mixed-DPI monitors.
OmniSight is an MCP server that fixes these at the source. Claude Code, Codex, Cursor, Cline, Claude Desktop and any other MCP client can use it.
How it works
- Structure first, pixels second. By default a snapshot returns the UI Automation element tree: every element has an id, a position and the actions it supports. The whole tree comes back in one cross-process call (a UIA cache request), typically in 30–300 ms, and it is text, not an image.
- Act on elements without the mouse.
click(element="e7")runs the UIA Invoke / Toggle / Select pattern andtype_text(replace=true)sets the value directly. This works on background windows and does not steal focus. - Capture one window, with backends that fall back automatically.
printwindow→wgc(Windows.Graphics.Capture) →bitblt→screen. A black or single-colour frame counts as a failure and the next backend is tried. The backend that worked is remembered per process. - Image and OCR only when the tree is not enough. In
automode a screenshot and OCR text with positions are added when the tree does not describe the content (games, emulators, canvas apps, custom-drawn UIs). - One coordinate space. Element positions, OCR boxes and the image all use
window-relative pixels of the snapshot image, so a position read from any of
them can go straight into
click. The process is per-monitor DPI aware.
Measured on a Windows 10 machine: a covered emulator window captured in about 100 ms, a 153-element Discord tree in about 130 ms, and repeat captures of a covered Discord window in under 200 ms end to end over MCP.
Tools
| Tool | What it does |
|---|---|
list_windows |
Windows front to back: hwnd, process, title, size, and flags such as minimized, admin, capture-blocked |
snapshot |
mode = tree / image / ocr / auto / all |
click |
Click an element id or an x,y; background=true posts messages instead of moving the mouse |
type_text |
Type text in any language, or set a control's value directly |
press_key |
Key combinations such as ctrl+s; scancode=true for games that read hardware scan codes |
scroll / drag |
Scroll and drag |
element_action |
invoke / toggle / expand / select / set_value / focus |
find_element |
Find elements by text |
wait_for_change |
Wait until the window's pixels change (loading, animations) |
screenshot_screen |
Whole-screen capture, for cross-window context |
focus_window |
Bring a window to the front |
The window argument takes an hwnd, or part of the title or process name.
Install
Requires Windows 10 1903 or later and uv.
Nothing to install up front: uvx fetches OmniSight from PyPI the first time
the agent starts it.
The [ocr] extra adds RapidOCR (about 70 MB). It is far more accurate than the
built-in Windows OCR on small UI fonts and mixed Chinese / English text. Without
it, use uvx omnisight and OmniSight falls back to Windows OCR.
Connect an agent
Claude Code
claude mcp add -s user omnisight -- uvx --from "omnisight[ocr]" omnisight
Codex (~/.codex/config.toml)
[mcp_servers.omnisight]
command = "uvx"
args = ["--from", "omnisight[ocr]", "omnisight"]
Cursor, Cline, Claude Desktop and other mcpServers JSON configs
{
"mcpServers": {
"omnisight": {
"command": "uvx",
"args": ["--from", "omnisight[ocr]", "omnisight"]
}
}
}
From source
git clone https://github.com/Alex20090906/omnisight
cd omnisight
uv sync --extra ocr
claude mcp add -s user omnisight -- uv run --directory <path> omnisight
Environment variables
| Variable | Meaning |
|---|---|
OMNISIGHT_OCR_ENGINE |
auto (default: RapidOCR if installed), rapid or windows |
OMNISIGHT_OCR_LANG |
Language for Windows OCR, such as zh-Hant-TW or en-US |
What it deliberately does not do
- It does not bypass
SetWindowDisplayAffinity. Password managers, banking apps and DRM video opt out of capture on purpose. OmniSight says so clearly and suggests the element tree, instead of returning a black image the agent would retry forever. - It does not evade anti-cheat or anti-bot detection, and it does not touch the UAC secure desktop or the lock screen.
- Admin windows: Windows (UIPI) drops input sent to an elevated window from a non-elevated process. OmniSight reports this. To control such a window, start the agent from an elevated terminal.
Known limitations
- On Windows 10, the WGC backend draws a yellow border around the captured
window (Windows 11 can turn it off). That is why
printwindowis tried first and WGC is used only when PrintWindow returns a black frame. - Minimized windows have no pixels.
snapshot(restore_minimized=true)restores the window without activating it. - Background key presses (
background=true) are reliable for single keys only: most apps read the real keyboard state for Ctrl / Alt / Shift. - Background clicks work for classic Win32 and WinForms. Chromium and most
games ignore them; use an element id, or leave
backgroundoff. - Chromium and Electron build their accessibility tree only after a client asks
for it.
snapshotretries once automatically.
Layout
src/omnisight/
winutil.py window enumeration, DPI awareness, elevation, display affinity
capture.py capture backend chain, black-frame detection, WGC session pool
uia.py UI Automation tree (one cache request) and actions
ocr.py RapidOCR / Windows OCR
inputs.py SendInput and background message input
server.py MCP tools
Development
uv sync --extra ocr --group dev
uv run pytest
License
Metadata
Release files for omnisight 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| omnisight-0.1.0.tar.gz | 30.1 kB | With provenance | Details for omnisight-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| omnisight-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for omnisight-0.1.0-py3-none-any.whl |
Total release size: 63.5 kB
Release files / omnisight-0.1.0.tar.gz
| Download URL | omnisight-0.1.0.tar.gz |
|---|---|
| Size | 30.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
400fac3663a425974a0eb4e4e63fc5b8d52c99ad6cb1011678b4d921c46cb9a0
|
|
BLAKE2b-256 checksum How to use checksums |
cf4c23a0dd835516f205debf3315f6bc31b11f151ee75a4be0e4dc1a1cd08382
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.
Transparency logRelease files / omnisight-0.1.0-py3-none-any.whl
| Download URL | omnisight-0.1.0-py3-none-any.whl |
|---|---|
| Size | 33.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
3cf7e65e28f67a60eb41ef5d994256a2cde468d8ddad21676a5d485f5f9669a1
|
|
BLAKE2b-256 checksum How to use checksums |
c60948ea6abcac3d7f64125789858a22015e145907f0e9c5546a4f17d6225174
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.
Transparency logSecurity for omnisight 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for omnisight-0.1.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.