____        _ _ _____
|  _ \ _   _| | | ____|
| | | | | | | | |  _|
| |_| | |_| | | | |___
|____/ \__,_|_|_|_____|

What is DullE?

DullE is a Windows computer-use agent that understands plain English and executes tasks on your desktop. It uses a CLI-first, vision-fallback approach — shell commands for speed, screenshots + mouse/keyboard only when the task truly needs it.

uv run dulle "open WhatsApp and send Subho a message saying hello"
uv run dulle "create a folder called Projects in Downloads"
uv run dulle "search for lofi music on YouTube"
uv run dulle "open Word, write a cover letter, save it as PDF"

Features

How It Works

You: "open WhatsApp and send Subho a message introducing yourself"
         │
         ▼
  ┌──────────────┐
  │   PLANNER    │  Decomposes into steps (text model, fast + cheap)
  │              │  Step 1: [cli]  discover WhatsApp app ID
  │              │  Step 2: [cli]  launch WhatsApp
  │              │  Step 3: [wait] wait for app to load
  │              │  Step 4: [gui]  find Subho and open chat   ← needs_input?
  │              │  Step 5: [gui]  type and send message
  └──────┬───────┘
         │
         ▼
  ┌──────────────┐
  │   EXECUTOR   │  Drives the screen (vision model)
  │              │  • Runs CLI commands directly via cmd.exe
  │              │  • Takes screenshots, decides next action
  │              │  • Batches predictable sequences (click+type+send)
  │              │  • Asks you if it hits genuine ambiguity
  │              │  • Handles popups, errors, loading states
  └──────┬───────┘
         │
         ▼
  ┌──────────────┐
  │   VERIFIER   │  Confirms each step worked (reliable model)
  │              │  Retries if page is still loading
  └──────────────┘

Quick Start

1 — Install

git clone https://github.com/yourname/DullE.git
cd DullE
uv sync

2 — Configure

copy .env.example .env
# Edit .env — set your API key, base URL, and model names

Minimum required in .env:

DULLE_API_KEY=your-api-key
DULLE_BASE_URL=https://your-endpoint/v1
DULLE_PLANNER_MODEL=your-text-model
DULLE_EXECUTOR_MODEL=your-vision-model
DULLE_VERIFIER_MODEL=your-reliable-model

3 — Run

# Single command
uv run dulle "open notepad and type hello world"

# Interactive mode
uv run dulle

# Auto mode — no confirmation prompts
uv run dulle --auto
uv run dulle -y "create a folder called test in Downloads"

Usage

Interactive mode

DullE> open chrome and go to youtube
DullE> search for lofi music
DullE> open word and write a short poem about rain
DullE> quit

Single command

uv run dulle "open calculator"
uv run dulle "take a screenshot and save it to Desktop"
uv run dulle "open WhatsApp and message Mom saying I'll be late"

Auto mode (skip confirmation prompts)

uv run dulle --auto
uv run dulle -y "create a folder called Projects in Downloads"

Eval suite

# Run all tasks
uv run dulle --eval

# Run specific tasks
uv run dulle --eval --tasks notepad_open,calculator_open

# Compare two runs
uv run dulle --eval-compare eval/results/20261012_120000 eval/results/20261012_130000

Configuration

All settings in .env use the DULLE_ prefix.

# ── API ─────────────────────────────────────────────────────────
DULLE_API_KEY=                        # Your API key
DULLE_BASE_URL=                       # OpenAI-compatible endpoint

# ── Models ──────────────────────────────────────────────────────
DULLE_PLANNER_MODEL=                  # Text model — plans steps (cheap)
DULLE_EXECUTOR_MODEL=                 # Vision model — drives the screen
DULLE_VERIFIER_MODEL=                 # Reliable model — confirms results
DULLE_EXECUTOR_MODEL_HEAVY=           # Fallback when executor gets stuck

# ── Agent behavior ──────────────────────────────────────────────
DULLE_MAX_ITERATIONS=40               # Max actions per step
DULLE_MAX_STEPS=20                    # Max steps per task
DULLE_STEP_TIMEOUT_S=120.0            # Seconds before a step is abandoned
DULLE_EXECUTION_MODE=hybrid           # hybrid | vision | cli

# ── Shell safety ────────────────────────────────────────────────
DULLE_REQUIRE_SHELL_CONFIRMATION=true # Ask before write commands
DULLE_REQUIRE_CONFIRMATION=true       # Master confirmation toggle
DULLE_ENABLE_SHELL_BLOCKLIST=true     # Block dangerous commands

# ── Screenshot ──────────────────────────────────────────────────
DULLE_SCREENSHOT_FORMAT=png           # png | jpeg
DULLE_MAX_LONG_EDGE=1568              # Resize before sending to model

# ── Knowledge base ──────────────────────────────────────────────
DULLE_KNOWLEDGE_FILE=knowledge.yaml   # Your personal facts file

# ── Audit trail ─────────────────────────────────────────────────
DULLE_ENABLE_AUDIT_TRAIL=true
DULLE_AUDIT_TRAIL_DIR=audit_trail

Recommended model combos

Safety

Three layers protect against accidents:

🚫 Blocklist — never runs, no matter what

format C:, del /s, reg delete, shutdown, diskpart, bcdedit, rmdir /s, and more. These patterns are blocked before any command executes.

✅ Allowlist — runs freely

Read-only commands like dir, type, start, tasklist, ipconfig, echo — safe by definition, no prompt.

⚠️ Confirmation gate — asks before writing

  Confirmation needed:
  Command: copy "report.docx" "C:\Users\subha\Downloads\"
  Reason:  This command modifies the system.
  Allow? [y/N]

Skip this for a session with --auto / -y, or permanently with DULLE_REQUIRE_CONFIRMATION=false in .env.

Knowledge Base

Edit knowledge.yaml to teach DullE facts about your setup:

- fact: "I have YouTube Premium — no ads, no skip buttons needed"
  tags: [youtube, video, ads]

- fact: "My work Chrome profile is Profile 2"
  tags: [chrome, browser, profile, work]

- fact: "Subho's WhatsApp name is Subho Chattopadhyay"
  tags: [whatsapp, contact, subho]

- fact: "Downloads folder is at C:\Users\subha\Downloads"
  tags: [downloads, folder, path]

Relevant entries are automatically retrieved and injected into the planner and executor context at runtime.

Project Structure

DullE/
├── .env.example              # Copy to .env and configure
├── pyproject.toml            # Dependencies
├── knowledge.yaml            # Your personal facts
├── eval_tasks.yaml           # Benchmark task definitions
│
├── src/dulle/
│   ├── config.py             # Settings (pydantic, DULLE_ prefix)
│   ├── models.py             # Shared data classes
│   ├── main.py               # CLI entry point + interactive REPL
│   │
│   ├── agent/                # Core orchestration
│   │   ├── core.py           # DullEAgent — plan → execute → verify loop
│   │   ├── steps.py          # Step routing (CLI / GUI / verify / wait)
│   │   ├── batch.py          # Single-action and batch-action execution
│   │   ├── loop_detect.py    # Stuck-loop detection and intervention
│   │   ├── context.py        # Conversation pruning, message builders
│   │   └── audit.py          # On-disk audit trail (screenshots + actions)
│   │
│   ├── llm/                  # LLM client and prompts
│   │   ├── client.py         # LLMClient — retry, token tracking, escalation
│   │   └── prompts.py        # System prompts for planner / executor / verifier
│   │
│   ├── execution/            # What actually runs on the machine
│   │   ├── actions.py        # ActionExecutor — all GUI + CLI action handlers
│   │   ├── shell.py          # Safe shell execution, blocklist/allowlist
│   │   └── screen.py         # Screenshot capture, DPI scaling
│   │
│   ├── planning/             # Task decomposition
│   │   ├── planner.py        # plan_task(), describe_screen()
│   │   └── knowledge.py      # KnowledgeBase — retrieval from knowledge.yaml
│   │
│   └── eval/                 # Evaluation suite
│       ├── runner.py         # Runs tasks, records results
│       ├── checks.py         # Post-task verification (process, file, screen)
│       └── report.py         # Compare runs, show diffs
│
├── eval/results/             # Saved eval run results (JSON)
└── audit_trail/              # Per-task screenshots and action logs

Requirements

• Python 3.11+
• Windows 10 or 11
• An OpenAI-compatible API endpoint with at least one vision-capable model
• uv for dependency management (install)

How the Agent Handles Surprises

DullE is designed to recover, not crash: