lookit 0.1.0

LLM-first vision toolkit for GUI grounding, OCR, and more

Lookit

LLM-first vision toolkit for GUI grounding, OCR, and more. Built with LangChain and Qwen3-VL. Outputs minimal plain text optimized for token efficiency.

Installation

1. Install CLI

pip install lookit

2. Configure Environment

Works with any OpenAI-compatible API (Ollama, vLLM, OpenRouter, etc.):

macOS / Linux:

# Add to ~/.zshrc (macOS) or ~/.bashrc (Linux)
cat << 'EOF' >> ~/.zshrc
# Ollama local
export LOOKIT_API_KEY="ollama"
export LOOKIT_MODEL="qwen3-vl"
export LOOKIT_BASE_URL="http://localhost:11434/v1"
EOF

source ~/.zshrc

For better results, use Ollama Cloud with the larger model (get API key):

export LOOKIT_API_KEY="your-ollama-api-key"
export LOOKIT_MODEL="qwen3-vl:235b-cloud"
export LOOKIT_BASE_URL="https://ollama.com/v1"

Skills Setup

Skills follow the Agent Skills specification.

Skill	Description
computer-use	GUI grounding for desktop screenshots
mobile-use	GUI grounding for mobile screenshots
ocr	Text extraction from screenshots

Claude Code

mkdir -p ~/.claude/skills/{computer-use,mobile-use,ocr}

curl -sL https://raw.githubusercontent.com/atom2ueki/lookit/main/skills/computer-use/SKILL.md -o ~/.claude/skills/computer-use/SKILL.md
curl -sL https://raw.githubusercontent.com/atom2ueki/lookit/main/skills/mobile-use/SKILL.md -o ~/.claude/skills/mobile-use/SKILL.md
curl -sL https://raw.githubusercontent.com/atom2ueki/lookit/main/skills/ocr/SKILL.md -o ~/.claude/skills/ocr/SKILL.md

DeepAgents CLI

DeepAgents is an agent framework built on LangChain and LangGraph.

# Install deepagents CLI
pip install deepagents-cli

# Create skill directories
mkdir -p ~/.deepagents/default/skills/{computer-use,mobile-use,ocr}

# Download skills
curl -sL https://raw.githubusercontent.com/atom2ueki/lookit/main/skills/computer-use/SKILL.md -o ~/.deepagents/default/skills/computer-use/SKILL.md
curl -sL https://raw.githubusercontent.com/atom2ueki/lookit/main/skills/mobile-use/SKILL.md -o ~/.deepagents/default/skills/mobile-use/SKILL.md
curl -sL https://raw.githubusercontent.com/atom2ueki/lookit/main/skills/ocr/SKILL.md -o ~/.deepagents/default/skills/ocr/SKILL.md

# Verify skills installed
deepagents skills list

Programmatic Integration

For integrating skills into your own LangChain agents, see deepagents PR #611 (WIP).

from deepagents import create_deep_agent
from deepagents.backends.filesystem import FilesystemBackend
from deepagents.middleware import SkillsMiddleware

# Create backend and skills middleware
backend = FilesystemBackend()
skills_middleware = SkillsMiddleware(
    backend=backend,
    registries=[
        {"path": "/skills/user/", "name": "user"},
        {"path": "/skills/project/", "name": "project"},
    ],
)

# Create agent with skills middleware
agent = create_deep_agent(
    model="openai:gpt-4o",
    middleware=[skills_middleware],
)

# Agent will automatically discover and use lookit skills
result = agent.invoke({
    "messages": [{"role": "user", "content": "Click the submit button in screenshot.png"}]
})

Usage

Same screenshot, different modes and prompts = different results:

OCR Mode	Computer Mode
lookit "extract the transaction history" -s screenshot.png --mode ocr	lookit "click search" -s screenshot.png --mode computer
Max Now Pte. Ltd. Singapore SG 24 Dec 2025 10:07:13 SGD 70.85 140 points Pending ...	left_click 2910,365

Output Format

Action Modes (computer/mobile)

left_click 960,324
type "hello world"
swipe 500,800 to 500,200
key Control+c
scroll -100

OCR Mode

Returns extracted text directly.

Arguments

Argument	Description
query	Natural language instruction
-s, --screenshot	Path to screenshot (required)
--mode	computer, mobile, or ocr (required)
--debug	Debug mode (for humans): print info to stderr, save annotated image

Actions

Computer

left_click, right_click, double_click, type, key, scroll, mouse_move

Mobile

click, long_press, swipe, type, system_button

License

MIT