oswright 0.3.1

Playwright-like automation framework for the operating system

OSWright

A Model Context Protocol (MCP) server that provides OS-level desktop automation using OCR and image matching. This server enables LLMs to interact with any desktop application -- click buttons, type text, read screens, and fill forms -- just like Playwright MCP does for browsers.

Key Features

• Cross-platform. Windows (Win32 API), Linux (pynput/X11), macOS (pynput/Quartz).
• Accessibility tree. Find elements deterministically by role and name via Windows UI Automation — 100% accurate, instant, no model needed.
• Fast OCR. Windows OCR (built-in, instant) with EasyOCR fallback for Linux/macOS. Results are cached automatically.
• Image matching. Locates elements by template image via OpenCV.
• Window management. List, focus, minimize, close, and screenshot specific windows.
• Screenshot diffing. Detect when the screen changes with wait_for_change.
• Clipboard access. Read and write system clipboard for data transfer.
• App launcher. Launch applications and wait for them to load.
• Auto-snapshot. Every action returns a screenshot so the agent always sees current state.
• 35+ MCP tools. Screen, OCR, UIA, mouse, keyboard, windows, clipboard, and compound actions.
• Test suite. 22 automated tests covering core functionality.

Requirements

• Python 3.10 or newer
• VS Code, Cursor, Windsurf, Claude Desktop, or any other MCP client

Getting started

First, install the OSWright MCP server with your client.

Standard config works in most tools:

{
  "mcpServers": {
    "oswright": {
      "command": "uvx",
      "args": ["oswright"]
    }
  }
}

Note: If you don't have uvx, you can use pip install oswright and then set "command": "oswright" directly.

Claude Desktop

Follow the MCP install guide, use the standard config above.

Claude Code

claude mcp add oswright uvx oswright

VS Code

Add to your user or workspace settings.json under mcp.servers:

{
  "mcp": {
    "servers": {
      "oswright": {
        "command": "uvx",
        "args": ["oswright"]
      }
    }
  }
}

Or use the VS Code CLI:

code --add-mcp '{"name":"oswright","command":"uvx","args":["oswright"]}'

Cursor

Go to Cursor Settings -> MCP -> Add new MCP Server. Name it oswright, use command type with the command uvx oswright.

Windsurf

Follow Windsurf MCP documentation. Use the standard config above.

Cline

Add to your cline_mcp_settings.json:

{
  "mcpServers": {
    "oswright": {
      "type": "stdio",
      "command": "uvx",
      "args": ["oswright"],
      "disabled": false
    }
  }
}

Goose

Go to Advanced settings -> Extensions -> Add custom extension. Name it oswright, use type STDIO, and set the command to uvx oswright.

Using pip instead of uvx

If you prefer a standard pip install:

pip install oswright

Then use this config:

{
  "mcpServers": {
    "oswright": {
      "command": "oswright"
    }
  }
}

Or run directly:

python -m oswright

Configuration

OSWright MCP server supports the following arguments. They can be provided in the JSON configuration as part of the "args" list:

Option	Description	Env Variable
--port <port>	Port for SSE transport. If omitted, uses stdio (default).	FASTMCP_PORT
--host <host>	Host to bind SSE server to. Default: localhost.	FASTMCP_HOST
--transport <mode>	Transport protocol: stdio, sse, streamable-http. Auto-detected from --port.
--ocr-languages <langs>	OCR languages (default: en). Example: --ocr-languages en es fr	OSWRIGHT_OCR_LANGUAGES
--timeout <seconds>	Default timeout for auto-wait operations (default: 10).	OSWRIGHT_TIMEOUT
--log-level <level>	Logging level: DEBUG, INFO, WARNING, ERROR. Default: INFO.	OSWRIGHT_LOG_LEVEL

Example: Multi-language OCR

{
  "mcpServers": {
    "oswright": {
      "command": "uvx",
      "args": ["oswright", "--ocr-languages", "en", "es", "fr"]
    }
  }
}

Standalone MCP server (SSE)

When running from a remote machine or a worker process, use SSE transport:

uvx oswright --port 8931

Then in your MCP client config:

{
  "mcpServers": {
    "oswright": {
      "url": "http://localhost:8931/mcp"
    }
  }
}

Platform Notes

Platform	Input Backend	OCR Backend	Notes
Windows	Win32 API (SendInput)	Windows OCR (instant) + EasyOCR fallback	No extra deps. Windows OCR is built-in.
Linux	pynput (X11)	EasyOCR	Requires X11 display server. Wayland has limited support.
macOS	pynput (Quartz)	EasyOCR	Grant Accessibility permissions in System Settings > Privacy > Accessibility.

Tools

Screen

• screenshot -- Take a screenshot of the screen or a region. Returns the image as native MCP image content. Optionally saves to a file path.

  • Read-only: true

• get_screen_info -- Get screen dimensions and monitor count.

  • Read-only: true

OCR / Text Finding

• find_text_on_screen -- Find all occurrences of text on screen using OCR. Returns matches with coordinates and confidence.

  • Parameters: text, exact, region bounds, monitor
  • Read-only: true

• read_screen_text -- Read ALL visible text on the screen using OCR. Returns every detected text element with position.

  • Parameters: region bounds, monitor
  • Read-only: true

Image Matching

• find_image_on_screen -- Find all occurrences of a template image on screen using OpenCV template matching.
  • Parameters: template_path, threshold, monitor
  • Read-only: true

Mouse

• mouse_click -- Click the mouse at coordinates or current position. Returns screenshot.

  • Parameters: x, y, button, clicks

• mouse_double_click -- Double-click at coordinates or current position. Returns screenshot.

• mouse_move -- Move the mouse cursor to screen coordinates.

• mouse_scroll -- Scroll the mouse wheel. Returns screenshot.

  • Parameters: amount, x, y

• mouse_drag -- Drag from one point to another. Returns screenshot.

  • Parameters: start_x, start_y, end_x, end_y, button, duration

• get_mouse_position -- Get the current mouse cursor position.

  • Read-only: true

Keyboard

• type_text -- Type text character by character. Returns screenshot.

  • Parameters: text, delay

• press_key -- Press a key or combo like Enter, Ctrl+C, Alt+Tab. Returns screenshot.

  • Parameters: key

Compound Actions

• click_text -- Find text via OCR and click on it. Auto-retries until found or timeout. Returns screenshot.

  • Parameters: text, exact, button, timeout, poll_interval, monitor

• double_click_text -- Find text via OCR and double-click on it. Returns screenshot.

• right_click_text -- Find text via OCR and right-click on it. Returns screenshot.

• hover_text -- Find text via OCR and hover over it. Returns screenshot.

• fill_field -- Find a label, click it, clear, and type a value. Returns screenshot.

  • Parameters: target_text, value, exact, timeout, monitor

• fill_form -- Fill multiple fields in one call. Reduces round-trips.

  • Parameters: fields (list of {label, value}), timeout, monitor

• wait_for_text -- Wait for text to appear on screen. Polls via OCR.

  • Parameters: text, exact, timeout, poll_interval, monitor
  • Read-only: true

• wait_for_text_gone -- Wait for text to disappear from screen.

  • Parameters: text, exact, timeout, poll_interval, monitor
  • Read-only: true

• wait_for_time -- Wait for a specified duration (capped at 30s), then screenshot.

Window Management

• list_windows -- List all visible windows. Optionally filter by title substring.

  • Parameters: title_filter
  • Read-only: true

• focus_window -- Bring a window to the foreground by title. Returns screenshot.

  • Parameters: title

• close_window -- Close a window by title (sends WM_CLOSE). Returns screenshot.

  • Parameters: title

• minimize_window -- Minimize a window by title. Returns screenshot.

  • Parameters: title

• screenshot_window -- Capture a screenshot of just one window.

  • Parameters: title, save_path
  • Read-only: true

Clipboard

• get_clipboard -- Get the current text content of the system clipboard.

  • Read-only: true

• set_clipboard -- Copy text to the system clipboard.

  • Parameters: text

App Management

• launch_app -- Launch an application and optionally wait for it to load.

  • Parameters: command, wait_text, timeout

• get_ocr_info -- Get info about the active OCR backend and available backends.

  • Read-only: true

Accessibility / UI Automation (Windows)

• get_ui_tree -- Get the accessibility tree of the focused window. Returns all interactive elements with names, types, positions. Deterministic and instant.

  • Parameters: window_title, max_depth
  • Read-only: true

• click_ui_element -- Click a UI element using the accessibility tree. More reliable than OCR.

  • Parameters: name, control_type, automation_id, window_title

• fill_ui_element -- Set the value of a UI element (e.g., text box). More reliable than OCR-based fill.

  • Parameters: value, name, automation_id, window_title

Advanced Screen

• get_active_window -- Get info about the currently focused window.

  • Read-only: true

• wait_for_change -- Wait for the screen to visually change. Takes a baseline screenshot, polls until different.

  • Parameters: timeout, poll_interval

Python Library

OSWright also works as a standalone Python library with a Playwright-style API:

from oswright import OSWright

with OSWright() as ow:
    screen = ow.screen()
    screen.click(text="Start")
    screen.type_text("Hello World")
    screen.press("Ctrl+S")
    screen.screenshot("desktop.png")

See the examples/ directory for more.

Architecture

oswright/
  __init__.py          # Package entry point
  core.py              # OSWright class (= Browser)
  screen.py            # Screen class (= Page)
  locator.py           # Locator + Assertions (= Locator + expect)
  capture.py           # Screen capture (mss - cross-platform)
  detect.py            # OCR dispatcher with caching (auto-selects best backend)
  _ocr_windows.py      # Windows OCR backend (instant, built-in)
  accessibility.py     # Windows UI Automation (deterministic element finding)
  cache.py             # Screenshot diffing, image hashing, OCR result cache
  input.py             # Platform dispatcher for input backends
  _input_windows.py    # Windows input backend (Win32 API)
  _input_pynput.py     # Linux/macOS input backend (pynput)
  window.py            # Window management (list, focus, close)
  clipboard.py         # Clipboard read/write (cross-platform)
  mcp_server.py        # MCP server (35+ tools for AI agents)
tests/
  test_core.py         # 22 automated tests

License

MIT