screen2prompt 0.0.2

Give any LLM computer control (VLA) in three lines — Set-of-Mark screen tagging, Pydantic actions, zero-latency OS control.

Screen2Prompt 🎯

Grant any LLM computer control (VLA) capabilities with just 1 CLI command or 3 lines of code.

Screen2Prompt is an open-source VLA (Vision-Language-Action) toolkit[cite: 1]. It captures the screen, applies a Set-of-Mark (numerical tags) to UI elements, and executes OS actions autonomously based on the Pydantic structured output returned by any LLM (OpenAI, Ollama, LM Studio, etc.)[cite: 1].

---

✨ Key Features

🏷️ Ultimate Set-of-Mark (OpenCV + EasyOCR)

Extracts UI blocks assigning a unique numerical ID to each element[cite: 1].

• OCR Immunity: Text detected by EasyOCR is guaranteed to be tagged, preventing critical web elements (like search bars) from being filtered out.
• DPI Aware: Automatically calibrates coordinates for multi-monitor setups and OS display scaling (e.g., 4K monitors).

⚡ Universal LLM Support (Plug-and-Play)

Not locked into OpenAI! Use Ollama, vLLM, or LM Studio seamlessly. Simply override the base_url to run your local AI agent entirely for free.

🔄 Autonomous Closed-Loop & Token Optimization

• ReAct Feedback Loop: agent.act() automatically captures the new screen state and returns it to the LLM, creating a true autonomous agent.
• Sliding Window: Prevents API cost explosions and context limit errors by keeping only the most recent visual context and converting older screens to text stubs.

🛡️ Human-in-the-Loop & Advanced OS Control

• Safety First: CLI prompts a [Y/N] confirmation before executing physical clicks or typing.
• Rich Action Schema: Supports click, drag, type, wait, press_key (e.g., Enter, Esc), and hotkey (e.g., Win+D). LLMs can actively navigate, wait for loading screens, or recover from errors.

🚀 Hardware Acceleration (GPU / MPS)

By default, Screen2Prompt checks your system environment. If an NVIDIA GPU (CUDA) or Apple Silicon (MPS) is available, it automatically boosts OCR processing speed.

You can also explicitly control hardware acceleration via CLI or Python code:

# Force enable GPU/MPS via CLI
screen2prompt run "Open browser" --gpu

# Enable GPU acceleration in Python
agent = ScreenAgent(gpu=True)

---

🚀 Quick Start

Installation

pip install screen2prompt
# Note: For NVIDIA GPU acceleration, ensure CUDA is installed for EasyOCR.

1. One-Click CLI Agent (The Magic "Click")

You don't need to write Python code. Control your PC straight from the terminal.

# Using OpenAI (requires OPENAI_API_KEY in .env)
screen2prompt run "Empty the recycle bin"

# Using Local LLM (Ollama) - 100% Free & Private
screen2prompt run "Search the weather on Google" --model ollama/llama3-vision --base-url http://localhost:11434/v1

2. Python API (Custom Integration)

from screen2prompt import ScreenAgent, ActionSchema

agent = ScreenAgent()

# 1. Observe the screen
prompt_data, image_path = agent.observe() 

# 2. Execute Action & Get Feedback (Closed-Loop)
success, new_prompt, new_image = agent.act(
    ActionSchema(action_type="double_click", target_id=12)
)

---

📦 API Overview

ActionSchema (Pydantic)

Forces all actions into a strict schema, fully compatible with GPT-4o's Structured Outputs[cite: 1].

ActionSchema(
    action_type="click",       # click | double_click | drag | scroll | type | wait | press_key | hotkey | done
    target_id=3,               # ID from the mapping JSON
    destination_id=None,       # Used for drag
    text=None,                 # Used for type
    scroll_clicks=None,        # Used for scroll (positive=up, negative=down)
    wait_seconds=2.5,          # Used for wait (let the LLM decide loading times)
    key_name="enter",          # Used for press_key
    hotkey_names=["win", "d"]  # Used for hotkey combinations
)

---

🌍 REST API Server

Turn your machine into an accessible AI node.

# Start REST API server (default port 8000)
screen2prompt serve --port 8080 --host 0.0.0.0

Execute Agent via POST:

curl -X POST [http://127.0.0.1:8080/run_agent](http://127.0.0.1:8080/run_agent) \
  -H "Content-Type: application/json" \
  -d '{
        "goal": "Open YouTube and play jazz music",
        "model": "gpt-4o"
      }'

---

⚙️ Platform Permissions

OS	Required Permissions
macOS	System Settings → Privacy & Security → Accessibility (Allow Terminal/Python/IDE)[cite: 1]
Windows	Allow mouse/keyboard control (Run as Administrator if needed)[cite: 1]

---

📁 Project Structure

screen2prompt/
├── screen2prompt/
│   ├── __init__.py
│   ├── llm_client.py  # Universal LLM Client (OpenAI API spec)
│   ├── vision.py      # Set-of-Mark & EasyOCR analysis
│   ├── actions.py     # OS control (PyAutoGUI)
│   ├── agent.py       # ScreenAgent + ActionSchema (Closed-Loop)
│   ├── server.py      # FastAPI REST Server
│   └── cli.py         # Terminal CLI
├── setup.py
├── requirements.txt
└── README.md

---

📄 License

MIT License — Free to use, modify, and distribute[cite: 1].

---

⭐ Star this repo if Screen2Prompt saves your tokens and your sanity!