screenmind 0.2.5

Captures your screen → Analyzes with Gemma 4 → Builds a searchable AI memory
100% local. 100% private. Zero cloud dependencies.

Website · Features · Comparison · Gemma 4 Deep Dive · Quick Start · Architecture · Agent Platform · MCP · API

Agents

💬 Chat in Action — Ask anything about your screen history

Microsoft showed the world wants screen-aware AI with Recall. But Recall stores data in plaintext, sends telemetry, and was met with massive privacy backlash. ScreenMind is the open-source, privacy-first alternative — every screenshot analyzed, every insight generated, every search result — all computed locally using Gemma 4's multimodal capabilities.

It's not just a screen recorder. It's an AI memory you can talk to, search through, and build automations on top of.

✨ Features

🧠 Core Intelligence

• 📸 Smart Capture — Content-change detection, not a fixed timer. Captures when your screen actually changes.
• 🔬 Gemma 4 Vision Analysis — Every screenshot analyzed: app detection, activity categorization, mood, scene description, spatial layout regions.
• 🔍 Hybrid Search — Semantic embeddings (MiniLM) + FTS5 keyword search. Find anything by meaning, not just keywords.
• 💬 Chat with Memory — Conversational RAG with follow-up support. Ask "what did Alex say on Discord?" → get the actual message.
• 🧠 Model Hub — In-app model download with variant selection (Q4_0, Q8_0, BF16, etc.). Supports Gemma 4 E2B, E4B, and 12B. Chat and Summary are locked with witty brain animations until the model is ready — then auto-unlock. No terminal needed.
• 🎙️ Voice Memos — Hold Ctrl+Shift+V → Gemma 4's native audio encoder transcribes. Screenshot captured alongside, full transcript searchable. Audio always saved, even if transcription fails — re-transcribe any time.
• 🎤 Meeting Transcription — Auto-detects Zoom, Teams, Webex, Slack and Discord apps, plus Google Meet, Teams, Zoom and Webex in a browser tab. Records only after voice is detected, transcribes, and generates structured summaries. Off by default (Settings → Audio).
• 📊 Analytics Dashboard — Category breakdown, top apps, hourly heatmap, meeting stats, focus metrics.
• ⏪ Day Rewind — Timelapse playback of your entire day with play/pause/scrub/speed controls.
• 🚀 Desktop Launcher — Splash screen with auto-open dashboard. Desktop shortcut auto-created on first run.
• 🔄 Run at Startup — Cross-platform auto-start (Windows Registry, macOS LaunchAgent, Linux XDG). Toggle from Settings or CLI.

⚡ Performance

• Three Analysis Modes — Accurate (~76s, deep thinking + layout), Balanced (~40s, thinking), or Fast (~12s, no thinking). You choose.
• Per-App pHash Cache — 3-tier caching with app-aware staleness. Communication apps refresh faster than IDEs. Significantly fewer inference calls.
• Chat-First GPU Priority — Chat cancels in-flight analysis instantly. GPU freed in <1s.
• Parallel Pipeline — Dev context detection and embedding generation run concurrently (~2-3s savings per frame).
• Auto-Pause Heavy Apps — Games, video editors, 3D software detected → capture pauses automatically.

🔒 Privacy & Security

• 100% Local — All data stays on your machine. No telemetry. Ever. The network is only used to download llama-server and models, and for integrations you turn on.
• Sensitive Data Filter — Auto-redacts credit cards, SSNs, API keys, passwords before storage.
• Encryption at Rest — AES encryption for screenshots (Fernet, key in the OS keyring).
• Dashboard PIN Lock — Salted PBKDF2 PIN, session-based auth with configurable auto-lock timeout.
• Incognito Mode — One click: no captures, no bookmarks, no meeting recording.
• Data Retention — Screenshots, recordings and history older than N days (default 7) are deleted automatically.

⌨️ System-Wide Hotkeys

All hotkeys customizable from Settings. On macOS, global hotkeys aren't supported yet — use the dashboard. On Linux they need root or membership in the input group.

📊 How ScreenMind Compares

TL;DR: ScreenMind is the only option that's fully MIT open-source, runs on any hardware (including a $150 GPU), handles vision + audio + reasoning with a single local model, and lets you actually chat with your screen memory.

🧠 How Gemma 4 Is Used

Gemma 4 is not a bolt-on — it's architecturally load-bearing. ScreenMind uses all three modalities (E2B shown as default; E4B and 12B also supported):

1. Vision — Screenshot Analysis

Every screenshot is sent to Gemma 4 with OCR context. It returns structured JSON:

• App name, activity category, summary, detailed context
• Mood classification, confidence score
• Rich scene description (every visible element inventoried)
• Layout regions (sidebar, chat area, toolbar boundaries)

Three modes (benchmarked on GTX 1650 4GB — scales dramatically with better GPUs):

• Accurate — single call with thinking (~76s). Best layout detection.
• Balanced — thinking enabled, analysis-only (~40s). Richer descriptions than Fast.
• Fast — no-thinking prefill trick (~12s). Layout via OCR clustering instead.

2. Audio — Voice Memos & Meeting Transcription

Gemma 4's encoder-free architecture handles audio natively across all model sizes. ScreenMind uses it for:

• Voice memo transcription (hold hotkey → speak → release)
• Meeting transcription (15s chunks, map-reduce summarization for long meetings)

No Whisper dependency. One model handles everything.

3. Reasoning — Summaries, Chat, Agents

• Daily summaries with deep reasoning (think=True)
• Chat answers grounded in actual screen data (text-first RAG with vision fallback)
• Agent execution — Gemma processes markdown agent prompts with injected screen data

Supported Models

E2B is the recommended default — it checks all the boxes:

If you have more VRAM, E4B offers richer analysis. The 12B model gives the best text reasoning for complex prompts.

E2B and E4B come from ggml-org (official llama.cpp-compatible GGUFs), 12B from bartowski. Models are stored in ~/.screenmind/models/.

🚀 Quick Start

Requirements: Python 3.10+ · GPU recommended (4GB+ VRAM) · ~2GB disk for model (E2B Q4_0)

1️⃣ Install

pip install screenmind

2️⃣ Run

screenmind              # Normal start (foreground)
screenmind --launch     # Splash screen + auto-open dashboard
screenmind --background # Run silently without console window

3️⃣ Open → http://127.0.0.1:7777

On first run, ScreenMind will:

• Prompt to install AI packages (~2.5GB one-time download)
• Auto-detect your GPU and download llama-server if not found (CUDA/CPU auto-selected)
• Open the Model Hub — pick your model + quantization variant, download with progress tracking right in the UI
• Chat and Summary stay locked (🧠💤 "I need my brain to think!") until the model is ready, then auto-unlock
• Start llama-server in background (or adopt an already-running external server)
• Show the welcome screen to set up an optional PIN
• Create ~/.screenmind/ for data storage, ~/.screenmind/models/ for GGUF files
• Auto-create a desktop shortcut

screenmind --install-startup    # Register to start at system login
screenmind --uninstall-startup  # Remove from system startup
screenmind --install-shortcut   # Create desktop shortcut
screenmind --version            # Show version
screenmind --help               # Show all options

git clone https://github.com/ayushh0110/ScreenMind.git
cd ScreenMind

python -m venv venv
venv\Scripts\activate        # Windows
# source venv/bin/activate   # macOS/Linux

pip install -e ".[ai,dev]"

cp .env.example .env
# Edit capture interval, blocked apps, hotkeys, etc.

🏗️ Architecture

For a full deep-dive into threading, caching, search internals, and the privacy pipeline, see architecture.md.

┌─────────────────────────────────────────────────────────────────────┐
│                          ScreenMind                                  │
│                                                                     │
│  ┌────────────┐    ┌──────────────┐    ┌─────────────────────────┐ │
│  │  Capture   │───▶│  Async Queue │───▶│    Analysis Worker      │ │
│  │  Worker    │    │  (max: 100)  │    │                         │ │
│  │            │    └──────────────┘    │  ┌───────────────────┐  │ │
│  │ • Screen   │                        │  │  Per-App pHash    │  │ │
│  │ • Window   │                        │  │  Cache (3-tier)   │  │ │
│  │ • Dedup    │                        │  └───────────────────┘  │ │
│  │ • A11y     │                        │           │             │ │
│  │ • Privacy  │                        │           ▼             │ │
│  └────────────┘                        │  ┌───────────────────┐  │ │
│                                        │  │   EasyOCR         │  │ │
│  ┌────────────┐                        │  │   (text extract)  │  │ │
│  │   Audio    │                        │  └───────────────────┘  │ │
│  │   Worker   │                        │           │             │ │
│  │            │                        │           ▼             │ │
│  │ • Meeting  │                        │  ┌───────────────────┐  │ │
│  │   detect   │                        │  │   Gemma 4 E2B     │  │ │
│  │ • Record   │                        │  │   (via llama.cpp) │  │ │
│  │ • Transcr. │                        │  │   Vision + Audio  │  │ │
│  └────────────┘                        │  └───────────────────┘  │ │
│                                        │           │             │ │
│  ┌────────────┐                        │           ▼             │ │
│  │   Agent    │                        │  ┌───────────────────┐  │ │
│  │  Scheduler │                        │  │  Layout Analyzer  │  │ │
│  │            │                        │  │  (spatial OCR)    │  │ │
│  │ • .md AI   │                        │  └───────────────────┘  │ │
│  │ • .py code │                        │           │             │ │
│  └────────────┘                        │           ▼             │ │
│                                        │  ┌───────────────────┐  │ │
│                                        │  │  MiniLM-L6-v2     │  │ │
│                                        │  │  (embeddings)     │  │ │
│                                        │  └───────────────────┘  │ │
│                                        └─────────────────────────┘ │
│                                                    │               │
│                                                    ▼               │
│                                        ┌───────────────────┐       │
│                                        │   SQLite (WAL)    │       │
│                                        │   + FTS5 index    │       │
│                                        └─────────┬─────────┘       │
│                                                  │                 │
│  ┌───────────────────────────────────────────────┘                 │
│  │                                                                 │
│  ▼                                                                 │
│  ┌───────────────────────────────────────────────────────────────┐ │
│  │                    FastAPI REST Server                         │ │
│  │  /timeline · /search · /chat · /stats · /agents · /mcp       │ │
│  │                                                               │ │
│  │  ┌───────────────────────────────────────────────────────┐   │ │
│  │  │           Web Dashboard (Vanilla JS SPA)               │   │ │
│  │  │  Timeline · Chat · Search · Analytics · Agents · Settings │ │
│  │  └───────────────────────────────────────────────────────┘   │ │
│  └───────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────┘

Multi-Model AI Pipeline

Screenshot → EasyOCR (text) → Gemma 4 E2B (understanding) → MiniLM (embeddings) → SQLite + FTS5
                                     ↑
                              OCR text fed as context
                              (Gemma sees image + reads text)

Four AI models working in concert, with Gemma 4 as the brain:

1. EasyOCR — extracts raw screen text
2. Gemma 4 E2B — understands what you're doing (vision + reasoning)
3. MiniLM-L6-v2 — generates semantic vectors for natural language search
4. FTS5 — indexes text for instant keyword search

🤖 Agent Platform

ScreenMind includes a full agent/plugin system. Build any automation on top of your screen data.

Two Modes

Markdown Agent Example

---
name: Daily Focus Report
schedule: every 6h
data: timeline, apps, mood
output: local, obsidian
---

Analyze my screen activity and generate a focus report:
- How many hours of deep work vs shallow work?
- What were my main distractions?
- Give me a focus score out of 10.

Drop this file in ~/.screenmind/agents/ and turn on Settings → Automation → Agent System — it then runs on its schedule.

Python Plugin SDK

"""
name: URL Tracker
schedule: every 1h
"""
from screenmind.screenmind_sdk import get_activities, get_urls_visited, load_state, save_state, ask_gemma

def run(context):
    # Today's activities filtered by app
    activities = get_activities(app="Chrome", limit=20)

    # Persistent state across runs
    last_count = load_state("url_count", 0)
    urls = get_urls_visited()
    save_state("url_count", len(urls))

    # Ask Gemma (GPU-safe — waits for the model to be idle)
    insight = ask_gemma(f"Summarize these URLs: {[u['url'] for u in urls[:10]]}")
    return f"{len(urls) - last_count} new URLs.\n\n{insight}"

Python plugins must be approved once per session (✅ Approve & Run) unless auto-run is enabled. See the full guide: Build Your Own Agent.

Data Selectors (Frontmatter)

Markdown agents declare what data they need:

Agents receive today's data, sized to your Context Window setting.

4 Agents Ship Built-In

• daily-journal.md — First-person journal entry from your day
• focus-report.md — Focus score, deep work hours, distractions
• meeting-actions.md — Action items and decisions from today's recorded meetings (off by default)
• code-changelog.md — Summarize coding activity (commits, files, repos)

🔌 MCP Server (Claude / Cursor / VS Code)

ScreenMind exposes your screen history to any MCP-compatible AI tool:

python -m screenmind.mcp_server  # stdio transport

Install the MCP extra (pip install "screenmind[mcp]"), then add ScreenMind to your client. Claude Desktop (%APPDATA%\Claude\claude_desktop_config.json on Windows, ~/Library/Application Support/Claude/claude_desktop_config.json on macOS):

{
  "mcpServers": {
    "screenmind": {
      "command": "python",
      "args": ["-m", "screenmind.mcp_server"]
    }
  }
}

Claude Code, Cursor and VS Code setup: see MCP_SETUP.md.

Tools Available

📡 API Reference

Full Swagger docs at http://127.0.0.1:7777/docs

Key Endpoints

🔧 Tech Stack

🐧 Wayland Support

ScreenMind auto-detects Wayland sessions and uses compositor-native capture:

Install grim (recommended for wlroots compositors):

# Arch
sudo pacman -S grim

# Ubuntu / Debian (if available)
sudo apt install grim

# Fedora
sudo dnf install grim

GNOME / KDE Wayland: Best-effort only. Screenshots use the XDG Desktop Portal, which prompts for permission on each capture — not viable for continuous background recording. For full functionality, use an X11 session or a wlroots-based compositor with grim.

Optional (for portal fallback): python3-gi / python-gobject system package.

ScreenMind/
├── screenmind/                   # The Python package
│   ├── main.py                   # Entry point — starts all services, CLI flags
│   ├── config.py                 # Settings (.env + settings.json overrides)
│   ├── launcher.py               # Splash screen launcher (tkinter)
│   ├── startup.py                # Run-at-login registration
│   ├── setup_llama.py            # Detect / download llama-server
│   ├── screenmind_sdk.py         # SDK for Python plugin agents
│   ├── mcp_server.py             # MCP server for Claude / Cursor / VS Code
│   │
│   ├── capture/                  # Screenshot capture layer
│   │   ├── screen.py             # Capture facade (mss / Wayland backend)
│   │   ├── wayland.py            # Wayland backend (grim / XDG Portal)
│   │   ├── window.py             # Active window detection
│   │   ├── dedup.py              # Perceptual hash deduplication
│   │   ├── hotkey.py             # Global hotkeys (bookmark, pause, voice)
│   │   └── voice_recorder.py     # Mic recording for voice memos
│   │
│   ├── engine/                   # AI & intelligence layer
│   │   ├── analyzer.py           # Gemma 4 screenshot analysis (3 modes)
│   │   ├── llm_client.py         # llama-server client (chat, vision, audio)
│   │   ├── model_manager.py      # Server lifecycle, model download/switch
│   │   ├── embedder.py           # MiniLM semantic embeddings
│   │   ├── ocr.py                # EasyOCR text extraction
│   │   ├── layout_analyzer.py    # Spatial OCR organization
│   │   ├── dev_context.py        # Git repo/branch/diff detection
│   │   ├── a11y_extractor.py     # Accessibility text extraction
│   │   ├── agent_runner.py       # Agent scheduling & execution
│   │   ├── voice_memo.py         # Voice memo transcription + storage
│   │   └── browsers.py           # Browser identification per platform
│   │
│   ├── workers/                  # Background processing
│   │   ├── capture_worker.py     # Smart capture loop, privacy, incognito
│   │   ├── analysis_worker.py    # OCR → Gemma → Layout → Embed → Store
│   │   └── audio_worker.py       # Meeting detection & transcription
│   │
│   ├── storage/                  # Data persistence
│   │   ├── database.py           # SQLite + FTS5 + migrations
│   │   ├── files.py              # Screenshot / recording files and cleanup
│   │   └── models.py             # Pydantic data models
│   │
│   ├── privacy/                  # Privacy & security
│   │   ├── encryption.py         # Fernet encryption at rest
│   │   └── data_filter.py        # Sensitive data redaction
│   │
│   ├── platform_support/         # Cross-platform abstraction
│   │   ├── base.py               # PlatformAdapter interface
│   │   ├── windows.py            # Win32 + UI Automation
│   │   ├── macos.py              # Quartz + AXUIElement
│   │   └── linux.py              # xdotool / Wayland IPC + AT-SPI
│   │
│   ├── integrations/             # External connections
│   │   ├── obsidian.py           # Vault markdown export
│   │   ├── notion.py             # Notion API export
│   │   ├── webhooks.py           # HTTP webhooks (named hooks, HMAC, retry)
│   │   └── smart_notify.py       # Distraction / break / focus notifications
│   │
│   ├── api/                      # REST API + dashboard
│   │   ├── server.py             # FastAPI app + PIN middleware
│   │   ├── dependencies.py       # Shared state for routes
│   │   ├── serializers.py        # Response shaping for the dashboard
│   │   ├── routes/               # 16 route modules
│   │   └── static/               # Web dashboard (HTML + CSS + JS)
│   │
│   ├── ui/overlay.py             # On-screen overlay notifications
│   ├── default_agents/           # 4 built-in agents (.md)
│   └── assets/                   # Logo, favicon
│
├── tests/                        # pytest suite
├── docs/                         # BUILD_YOUR_OWN_AGENT.md, screenshots
├── architecture.md               # System design deep-dive
├── MCP_SETUP.md                  # MCP client setup
├── requirements.txt              # Full pinned dependencies
└── requirements-test.txt         # Lightweight CI dependencies (no PyTorch)

🛡️ Error Handling & Resilience

🎨 Dashboard

The web dashboard at http://127.0.0.1:7777 features:

• Timeline — Browse activities by date with thumbnails, AI summaries, category badges
• Chat — Conversational AI with screen memory. Ask anything about your history. Locked with 🧠💤 brain animation until model is ready.
• Search — Semantic + keyword hybrid search with OCR highlighting on screenshots
• Analytics — Category and top-app charts, tracked hours, meeting stats and analysis status for a day, week or month
• Rewind — Timelapse player with play/pause/scrub/speed controls
• Memos — Voice memos with audio player, transcript and re-transcribe
• Agents — Create, edit, run, and monitor agents
• Bookmarks — Everything you bookmarked (hotkey, menu, or auto-bookmark keywords)
• Meetings — Recorded meetings with structured summaries and full transcripts
• Summary — AI daily summary and standup notes for any past day
• Settings — Model Hub (download/switch models with progress), Shortcuts, Capture, AI, Audio, Privacy, Automation, Integrations, Storage

Dark glassmorphism UI. No build step. Instant load.

🧪 Development

Run the test suite:

# Fast (lightweight deps — same as CI, no PyTorch)
pip install -r requirements-test.txt
pip install -e .
pytest --cov=screenmind --cov-report=term-missing -q

# Full (includes the ML stack — sentence-transformers, easyocr)
pip install -e ".[ai,dev]"
pytest --cov=screenmind --cov-report=term-missing -q

CI runs on every push/PR (Windows, Python 3.10) with the lightweight deps, plus a JavaScript syntax check and a no-print() lint. See CONTRIBUTING.md.

🤝 Contributing

Contributions welcome! See CONTRIBUTING.md for guidelines. High-impact areas:

• 🍎 macOS/Linux testing — platform adapters exist, need real hardware testing
• 🐳 Docker container — one-command setup
• 🧩 Community agent registry — share agents between users
• 🌐 Browser extension — richer URL/tab context
• 📤 Export formats — Markdown, CSV, JSON

⭐ Show Your Support

If you find ScreenMind useful, please consider:

• ⭐ Star this repo — it helps others discover the project
• 🍴 Fork it — build your own agents and features
• 🐛 Report issues — help us improve
• 📣 Share it — tell others about privacy-first AI

📝 License

MIT License — see LICENSE for details.