Skip to main content

OmniSight

繁體中文說明:README.zh-TW.md

Computer use for every Windows app window, for any MCP agent: GPU-rendered, covered or in the background.

Agents that drive a desktop through screenshots keep hitting the same walls on Windows:

  • GPU-rendered windows (games, emulators, Chromium / Electron, UWP) come back black from classic GDI capture.
  • A window that is covered or in the background is not in a full-screen screenshot at all.
  • Every step sends a full-screen image to a vision model just to find one button, which is slow and expensive.
  • Clicks land in the wrong place on scaled and mixed-DPI monitors.

OmniSight is an MCP server that fixes these at the source. Claude Code, Codex, Cursor, Cline, Claude Desktop and any other MCP client can use it.

How it works

  1. Structure first, pixels second. By default a snapshot returns the UI Automation element tree: every element has an id, a position and the actions it supports. The whole tree comes back in one cross-process call (a UIA cache request), typically in 30–300 ms, and it is text, not an image.
  2. Act on elements without the mouse. click(element="e7") runs the UIA Invoke / Toggle / Select pattern and type_text(replace=true) sets the value directly. This works on background windows and does not steal focus.
  3. Capture one window, with backends that fall back automatically. printwindow → wgc (Windows.Graphics.Capture) → bitblt → screen. A black or single-colour frame counts as a failure and the next backend is tried. The backend that worked is remembered per process.
  4. Image and OCR only when the tree is not enough. In auto mode a screenshot and OCR text with positions are added when the tree does not describe the content (games, emulators, canvas apps, custom-drawn UIs).
  5. One coordinate space. Element positions, OCR boxes and the image all use window-relative pixels of the snapshot image, so a position read from any of them can go straight into click. The process is per-monitor DPI aware.

Measured on a Windows 10 machine: a covered emulator window captured in about 100 ms, a 153-element Discord tree in about 130 ms, and repeat captures of a covered Discord window in under 200 ms end to end over MCP.

Tools

Tool What it does
list_windows Windows front to back: hwnd, process, title, size, and flags such as minimized, admin, capture-blocked
snapshot mode = tree / image / ocr / auto / all
click Click an element id or an x,y; background=true posts messages instead of moving the mouse
type_text Type text in any language, or set a control's value directly
press_key Key combinations such as ctrl+s; scancode=true for games that read hardware scan codes
scroll / drag Scroll and drag
element_action invoke / toggle / expand / select / set_value / focus
find_element Find elements by text
wait_for_change Wait until the window's pixels change (loading, animations)
screenshot_screen Whole-screen capture, for cross-window context
focus_window Bring a window to the front

The window argument takes an hwnd, or part of the title or process name.

Install

Requires Windows 10 1903 or later and uv. Nothing to install up front: uvx fetches OmniSight from PyPI the first time the agent starts it.

The [ocr] extra adds RapidOCR (about 70 MB). It is far more accurate than the built-in Windows OCR on small UI fonts and mixed Chinese / English text. Without it, use uvx omnisight and OmniSight falls back to Windows OCR.

Connect an agent

Claude Code

claude mcp add -s user omnisight -- uvx --from "omnisight[ocr]" omnisight

Codex (~/.codex/config.toml)

[mcp_servers.omnisight]
command = "uvx"
args = ["--from", "omnisight[ocr]", "omnisight"]

Cursor, Cline, Claude Desktop and other mcpServers JSON configs

{
  "mcpServers": {
    "omnisight": {
      "command": "uvx",
      "args": ["--from", "omnisight[ocr]", "omnisight"]
    }
  }
}

From source

git clone https://github.com/Alex20090906/omnisight
cd omnisight
uv sync --extra ocr
claude mcp add -s user omnisight -- uv run --directory <path> omnisight

Environment variables

Variable Meaning
OMNISIGHT_OCR_ENGINE auto (default: RapidOCR if installed), rapid or windows
OMNISIGHT_OCR_LANG Language for Windows OCR, such as zh-Hant-TW or en-US

What it deliberately does not do

  • It does not bypass SetWindowDisplayAffinity. Password managers, banking apps and DRM video opt out of capture on purpose. OmniSight says so clearly and suggests the element tree, instead of returning a black image the agent would retry forever.
  • It does not evade anti-cheat or anti-bot detection, and it does not touch the UAC secure desktop or the lock screen.
  • Admin windows: Windows (UIPI) drops input sent to an elevated window from a non-elevated process. OmniSight reports this. To control such a window, start the agent from an elevated terminal.

Known limitations

  • On Windows 10, the WGC backend draws a yellow border around the captured window (Windows 11 can turn it off). That is why printwindow is tried first and WGC is used only when PrintWindow returns a black frame.
  • Minimized windows have no pixels. snapshot(restore_minimized=true) restores the window without activating it.
  • Background key presses (background=true) are reliable for single keys only: most apps read the real keyboard state for Ctrl / Alt / Shift.
  • Background clicks work for classic Win32 and WinForms. Chromium and most games ignore them; use an element id, or leave background off.
  • Chromium and Electron build their accessibility tree only after a client asks for it. snapshot retries once automatically.

Layout

src/omnisight/
  winutil.py   window enumeration, DPI awareness, elevation, display affinity
  capture.py   capture backend chain, black-frame detection, WGC session pool
  uia.py       UI Automation tree (one cache request) and actions
  ocr.py       RapidOCR / Windows OCR
  inputs.py    SendInput and background message input
  server.py    MCP tools

Development

uv sync --extra ocr --group dev
uv run pytest

License

Apache-2.0

Metadata

Release files for omnisight 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for omnisight 0.1.0
File Size Uploaded Provenance Actions
omnisight-0.1.0.tar.gz 30.1 kB With provenance Details for omnisight-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for omnisight 0.1.0
File Interpreter ABI Platform Actions
omnisight-0.1.0-py3-none-any.whl Python 3 none any Details for omnisight-0.1.0-py3-none-any.whl

Total release size: 63.5 kB

Release files / omnisight-0.1.0.tar.gz

Download URL omnisight-0.1.0.tar.gz
Size 30.1 kB
Tags Source
SHA-256 checksum
How to use checksums
400fac3663a425974a0eb4e4e63fc5b8d52c99ad6cb1011678b4d921c46cb9a0
BLAKE2b-256 checksum
How to use checksums
cf4c23a0dd835516f205debf3315f6bc31b11f151ee75a4be0e4dc1a1cd08382
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.

Transparency log

Release files / omnisight-0.1.0-py3-none-any.whl

Download URL omnisight-0.1.0-py3-none-any.whl
Size 33.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
3cf7e65e28f67a60eb41ef5d994256a2cde468d8ddad21676a5d485f5f9669a1
BLAKE2b-256 checksum
How to use checksums
c60948ea6abcac3d7f64125789858a22015e145907f0e9c5546a4f17d6225174
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.

Transparency log

Security for omnisight 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for omnisight-0.1.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page