weavgui

English | 中文

Requirements: You need a vision-capable LLM, and the Claw-style platform must support loading images directly from files into context. We recommend weavbot for this workflow.

Turn any vision-capable LLM into a desktop GUI operator.

weavgui provides a minimal CLI toolkit that closes the loop between seeing and acting: take an annotated screenshot, feed it to a vision model, move the mouse, and repeat — until the model clicks exactly where it needs to.

screenshot (with crosshair) → vision model analyzes image → mouse move / click → screenshot → ...

No browser required. No DOM. No accessibility tree. Just pixels and a feedback loop.

npx skills add https://github.com/yankeguo/weavgui/tree/main/skills/weavgui -a openclaw -y

Installation

uv tool install weavgui

To upgrade an existing installation:

uv tool upgrade weavgui

Quick Start

weavgui --version
weavgui screenshot
weavgui mouse move '(0.05,0.05)'
weavgui mouse moveto '(0.5,0.3)'
weavgui mouse click
weavgui pasteboard write hello world
weavgui pasteboard read
weavgui keystroke command+c

Auto-Screenshot Behavior

Every action command automatically captures a screenshot to screenshot.png in the current working directory after a short delay. No flags needed.

The screenshot always includes cursor markers (crosshair + reference boxes). Read screenshot.png as an image after any command to observe the result.

Coordinate System

Mouse and screenshot-related commands use normalized coordinates:

• Values range from 0.0 to 1.0
• Origin at top-left: (0.0, 0.0), bottom-right: (1.0, 1.0)
• x increases to the right (fraction of screen width), y increases downward (fraction of screen height)
• Single-display only (primary monitor)
• Resolution-independent: the same coordinates work regardless of screen size or DPI

Command Reference

Global

screenshot

Capture a screenshot of the primary display and save it as screenshot.png in the current working directory.

weavgui screenshot

The cursor position is always annotated:

• Red crosshair spanning the full screenshot
• Red small box centered on cursor (normalized radius 0.03)
• Green medium box centered on cursor (normalized radius 0.07)
• Blue large box centered on cursor (normalized radius 0.20)

The three boxes serve as positioning references for the next mouse move / mouse moveto command:

weavgui screenshot
# → saves screenshot.png with cursor markers
# → prints cursor position in normalized coordinates to stdout

mouse

mouse move '(dx,dy)'

Move the cursor by a relative delta in normalized coordinates. The argument uses (dx,dy) format. Fails if the target position would leave the valid range [0.0, 1.0). After moving, waits 500 ms and saves a screenshot to screenshot.png.

weavgui mouse move '(0.05,0.05)'
weavgui mouse move '(-0.05,0.03)'

mouse moveto '(x,y)'

Move the cursor to an absolute normalized position. The argument uses (x,y) format. Fails if the position is outside the valid range [0.0, 1.0). After moving, waits 500 ms and saves a screenshot to screenshot.png.

weavgui mouse moveto '(0.5,0.3)'

mouse click

Left click at the current cursor position. Waits 2 s then saves a screenshot to screenshot.png.

weavgui mouse click

mouse doubleclick

Double left click at the current cursor position. Waits 2 s then saves a screenshot to screenshot.png.

weavgui mouse doubleclick

mouse rightclick

Right click at the current cursor position. Waits 2 s then saves a screenshot to screenshot.png.

weavgui mouse rightclick

pasteboard

pasteboard read

Read text from the system pasteboard and print to stdout.

weavgui pasteboard read

pasteboard write <text...>

Write text to the system pasteboard. Multiple arguments are joined by a single space.

weavgui pasteboard write hello world

keystroke

Simulate keyboard input. Supports single keys and combinations joined with +. Waits 1 s then saves a screenshot to screenshot.png.

weavgui keystroke <keys>

Modifier aliases:

On macOS, single-key combos with command are sent via AppleScript for better reliability.

weavgui keystroke c
weavgui keystroke ctrl+c
weavgui keystroke command+c

Development

Clone the repository and install dependencies:

git clone https://github.com/yankeguo/weavgui.git
cd weavgui
uv sync

Run commands from source:

uv run weavgui --version
uv run weavgui screenshot

Notes

• Mouse and keystroke automation requires Accessibility permission.
• If commands fail on macOS, go to System Settings > Privacy & Security > Accessibility and grant access to your terminal application.

License

MIT