weavgui 0.1.6

CLI to orchestrate and automate graphical desktop workflows.

weavgui

English | 中文

Requirements: You need a vision-capable LLM, and the Claw-style platform must support loading images directly from files into context. We recommend weavbot for this workflow.

Turn any vision-capable LLM into a desktop GUI operator.

weavgui provides a minimal CLI toolkit that closes the loop between seeing and acting: take an annotated screenshot, feed it to a vision model, move the mouse, and repeat — until the model clicks exactly where it needs to.

screenshot (with crosshair) → vision model analyzes image → mouse move / click → screenshot → ...

No browser required. No DOM. No accessibility tree. Just pixels and a feedback loop.

npx skills add https://github.com/yankeguo/weavgui/tree/main/skills/weavgui -a openclaw -y

Installation

uv tool install weavgui

To upgrade an existing installation:

uv tool upgrade weavgui

Quick Start

weavgui --version
weavgui screenshot
weavgui mouse move '(0.05,0.05)'
weavgui mouse moveto '(0.5,0.3)'
weavgui mouse click
weavgui pasteboard write hello world
weavgui pasteboard read
weavgui keystroke command+c

Auto-Screenshot Behavior

Every action command automatically captures a screenshot to screenshot.png in the current working directory after a short delay. No flags needed.

Command	Auto-screenshot delay
screenshot	immediate (no delay)
mouse move, mouse moveto	500 ms
mouse click, doubleclick, rightclick	2 s
keystroke	1 s

The screenshot always includes cursor markers (crosshair + reference boxes). Read screenshot.png as an image after any command to observe the result.

Coordinate System

Mouse and screenshot-related commands use normalized coordinates:

• Values range from 0.0 to 1.0
• Origin at top-left: (0.0, 0.0), bottom-right: (1.0, 1.0)
• x increases to the right (fraction of screen width), y increases downward (fraction of screen height)
• Single-display only (primary monitor)
• Resolution-independent: the same coordinates work regardless of screen size or DPI

Command Reference

Global

Command	Description
weavgui --version	Print CLI version
weavgui -h	Show help

screenshot

Capture a screenshot of the primary display and save it as screenshot.png in the current working directory.

weavgui screenshot

The cursor position is always annotated:

• Red crosshair spanning the full screenshot
• Red small box centered on cursor (normalized radius 0.03)
• Green medium box centered on cursor (normalized radius 0.07)
• Blue large box centered on cursor (normalized radius 0.20)

The three boxes serve as positioning references for the next mouse move / mouse moveto command:

Target location	Suggested delta range
Inside the red box	Fine adjustment, ±0.03
Between red and green boxes	Medium adjustment, ±0.03–0.07
Between green and blue boxes	Coarse adjustment, ±0.07–0.20
Outside the blue box	Large move needed — estimate from the full screenshot

weavgui screenshot
# → saves screenshot.png with cursor markers
# → prints cursor position in normalized coordinates to stdout

mouse

mouse move '(dx,dy)'

Move the cursor by a relative delta in normalized coordinates. The argument uses (dx,dy) format. Fails if the target position would leave the valid range [0.0, 1.0). After moving, waits 500 ms and saves a screenshot to screenshot.png.

weavgui mouse move '(0.05,0.05)'
weavgui mouse move '(-0.05,0.03)'

mouse moveto '(x,y)'

Move the cursor to an absolute normalized position. The argument uses (x,y) format. Fails if the position is outside the valid range [0.0, 1.0). After moving, waits 500 ms and saves a screenshot to screenshot.png.

weavgui mouse moveto '(0.5,0.3)'

mouse click

Left click at the current cursor position. Waits 2 s then saves a screenshot to screenshot.png.

weavgui mouse click

mouse doubleclick

Double left click at the current cursor position. Waits 2 s then saves a screenshot to screenshot.png.

weavgui mouse doubleclick

mouse rightclick

Right click at the current cursor position. Waits 2 s then saves a screenshot to screenshot.png.

weavgui mouse rightclick

pasteboard

pasteboard read

Read text from the system pasteboard and print to stdout.

weavgui pasteboard read

pasteboard write <text...>

Write text to the system pasteboard. Multiple arguments are joined by a single space.

weavgui pasteboard write hello world

keystroke

Simulate keyboard input. Supports single keys and combinations joined with +. Waits 1 s then saves a screenshot to screenshot.png.

weavgui keystroke <keys>

Input	Meaning
c	Press c
ctrl+c	Press Ctrl+C
command+c	Press Command+C
shift+a	Press Shift+A
alt+f4	Press Alt+F4

Modifier aliases:

Alias	Resolves to
control	ctrl
cmd	command
option	alt

On macOS, single-key combos with command are sent via AppleScript for better reliability.

weavgui keystroke c
weavgui keystroke ctrl+c
weavgui keystroke command+c

Development

Clone the repository and install dependencies:

git clone https://github.com/yankeguo/weavgui.git
cd weavgui
uv sync

Run commands from source:

uv run weavgui --version
uv run weavgui screenshot

Notes

• Mouse and keystroke automation requires Accessibility permission.
• If commands fail on macOS, go to System Settings > Privacy & Security > Accessibility and grant access to your terminal application.

License

MIT