olly-desktop 1.2.9

Local-first AI assistant with agent mode, RAG, and MCP support

✦ AI Assistant

A cross-platform desktop AI assistant that lives in your system tray and works on whatever you are already doing — selected text, screenshots, clipboard images, and your own documents. It runs through Ollama so models stay local by default, with optional support for remote or cloud Ollama endpoints when you choose.

---

What makes this different

Most AI tools today are browser tabs, IDE plugins, or single-platform utilities tied to one vendor’s cloud. This project is built around a different idea: bring a capable assistant to the OS layer, without replacing your apps or sending everything to a SaaS backend.

	Typical cloud assistants (ChatGPT, Copilot, Gemini)	Ollama WebUI / chat apps	This project
Where it runs	Vendor cloud	Local server in a browser tab	Native desktop app (Windows, macOS, Linux)
How you invoke it	Switch app, paste, type	Open browser, paste	Selection action bar at the cursor, global hotkey, tray
Context from your work	Manual copy-paste	Manual copy-paste	Captures selection, target window, screenshots
Your files	Upload per chat / enterprise connectors	Manual upload or plugins	Folder RAG — index a directory, ask from the action bar
Model choice	Vendor models only	Any Ollama model	Any Ollama model + quality presets + vision model picker
Privacy posture	Data leaves device by default	Stays local if Ollama is local	Local-first; you control URL, capture, and offline mode
Insert back into apps	Copy manually	Copy manually	Insert last reply hotkey into the foreground app

Novel behaviors this project incorporates

1. Selection action bar (not a radial menu, not a sidebar)
After you select text, a compact toolbar appears near the cursor with one-click intents: Explain, Summarize, Translate, Ask, Screen, and Ask my files. Other tools usually make you open a separate window and paste — here the intent is chosen in context.

2. Foreground-aware capture
Before the assistant takes focus, it remembers which window and cursor position you were using. Screen capture targets that window — not the assistant’s own popup. That avoids the common “screenshot captured my chat window” failure mode.

3. System-wide workflow, not app-specific
Works in browsers, editors, PDF readers, terminals, and more via OS-level selection and hotkeys — not only inside one host application.

4. Local RAG on a folder you own
Point at a watch folder (Documents, a project directory, etc.). Files are chunked and embedded with Ollama; Ask my files pulls relevant passages into the prompt. No per-file upload dance each session.

5. Vision from the desktop
Paste images, capture a window, or use the Screen action for LeetCode-style problems, UI mockups, or diagrams — with configurable vision timeouts and image sizing for slow or cloud vision models.

6. Native UI per platform
PyQt6 with dedicated styling for Windows (Segoe), macOS (translucent / SF Pro), and Linux (GNOME-inspired) — not a generic Electron shell.

7. Power-user controls others often hide
Quality presets, custom model names, thinking-mode toggle for Qwen3/cloud models, remote Ollama URL detection with adjusted timeouts, chat export, recent chats, tone chips (Shorter / Simpler / Formal), and insert-reply hotkey.

What others do that this project does not (by design)

• No bundled proprietary model — you install and choose models via Ollama.
• No multi-user cloud sync — chats live in your app data directory on your machine.
• No IDE-only scope — it is a general desktop assistant, not a code-editor extension.
• No always-on cloud — when Ollama runs locally, inference stays on your PC; cloud use is opt-in via your Ollama URL and model choice.

---

Features

• Selection action bar — Explain, Summarize, Translate, Ask, Screen, Ask my files
• Global hotkey — open assistant with current selection (Alt+S on Windows/Linux, ⌥S on macOS)
• Insert reply — paste the last AI response into any app (Ctrl+Shift+V / ⌘⇧V)
• Vision — paste images, screenshot button, window/screen capture from the action bar
• RAG — index a configurable folder with Chroma + Ollama embeddings
• Chat — streaming, recent chats, export, tone chips, safe markdown code rendering
• Settings UI — tabbed: general, AI & models, hotkeys, files, advanced
• First-run wizard — Ollama setup and model download with progress
• Tray integration — launch at login, new chat, settings, paste screenshot, quit
• Platform UI — Windows, macOS (liquid glass), Linux (GNOME-inspired)

Agent mode (beta)

Opt in via Settings → AI & models → Enable tools (beta). When enabled and your model supports Ollama tool calling (e.g. qwen3, llama3.2), the assistant can run a short read-only tool loop before answering:

• Search indexed files — RAG over your watch folder
• List / read files — only inside the configured watch folder (path-scoped; no arbitrary disk access)
• Read clipboard — current text clipboard
• Capture screen — OCR text from the foreground window (respects the screen-capture setting)

Read-only tools use the same local-first rules as chat (including offline-only mode). Requires a tool-capable Ollama model; without one, chat falls back to plain streaming.

Desktop actions (beta)

With Enable tools and Allow desktop actions both on in Settings, the AI can — each behind an Allow / Deny dialog — write text files inside the watch folder only (.txt, .md, .csv, .json, .log), paste text where you click after a 3-second countdown, open http/https links in your browser, and open documents from the watch folder.

Constraints: text insertion is unavailable on Wayland; on macOS it needs the same Accessibility permission as hotkeys. The assistant never runs programs, presses arbitrary keys, or moves the mouse.

MCP servers (beta)

Configure stdio MCP servers in Settings → Advanced → MCP servers. When agent mode and MCP are enabled, tools from connected servers are advertised to the model automatically. Anything the server does not mark with a read-only hint triggers an Allow / Deny dialog that shows the exact arguments before execution.

• Disabled in offline-only mode — MCP servers are third-party programs that may use the network.
• Trust on first connect — you must explicitly trust a server before it is started.
• Requires the server's runtime — e.g. Node.js for npx @modelcontextprotocol/server-filesystem ….

Example: add a filesystem server scoped to a notes folder (npx -y @modelcontextprotocol/server-filesystem ~/Notes). Ask the agent to find action items from meeting notes; it can search and read files in that folder, then summarize. If it needs to write todo.md, a non-read-only MCP tool shows the confirmation dialog first.

SSE/HTTP MCP transport and image tool results are not in this release.

---

Privacy & data

Default	Your choice
Ollama at 127.0.0.1	Point to a remote or cloud Ollama URL in Settings
Chats stored under app data on your PC	Export or delete via the chat menu
Screen capture can be disabled	Toggle in Settings → Advanced
Images stripped from saved chat JSON	Raw prompts with base64 are not persisted

When Ollama runs locally, prompts and model output stay on your machine. If you use a cloud model (for example qwen3-vl:235b-cloud), inference goes to that endpoint — configure it explicitly in Settings → AI & models.

---

Installation

Requires Python 3.11+ (python.org on Windows, Homebrew or python.org on macOS, your distro's package manager on Linux).

pip install olly-desktop
olly

olly detaches from the terminal — safe to close the terminal after launch. Check the system tray (^ by the clock). On first launch, the setup wizard installs Ollama and downloads a model.

Updating

pip install --upgrade olly-desktop

Uninstalling

pip uninstall olly-desktop

Your chats and settings remain in the app data directory unless you delete them manually.

Platform notes

OS	Permissions / limits
Windows	Global hotkeys use the native RegisterHotKey API (no keyboard hook); selection capture may still prompt some AV products once
macOS	Grant Accessibility for hotkeys and text capture
Linux X11	Best support for global hotkeys and selection capture
Linux Wayland	Global hotkeys, selection capture, and agent text insertion may be unavailable — use the tray menu

---

From source

git clone https://github.com/tp-0604/ai-assistant.git
cd ai-assistant
python -m venv venv
source venv/bin/activate        # Windows: venv\Scripts\activate
pip install -r requirements.txt
# Windows only:
pip install -r requirements-windows.txt

Install Ollama, then:

# macOS / Linux
./launch.sh
# or
python main.py

# Windows
launch.bat

---

Usage

Action	How
Selection bar	Select text (drag, or double-click a word if enabled in Settings)
Open assistant	Alt+S (Windows/Linux) · ⌥S (macOS)
Paste image	Ctrl+V / ⌘V in chat input
Insert last reply	Ctrl+Shift+V / ⌘⇧V
Settings	Tray → Settings, or ⋮ in popup
Ask my files	Selection bar → Files (enable RAG and pick a folder in Settings)
Screenshot	Tray → Paste screenshot, or Screen in the action bar

---

Settings

Stored in the app data directory — editable via Settings:

OS	Location
Windows	%APPDATA%\AIAssistant\
macOS	~/Library/Application Support/AIAssistant/
Linux	~/.local/share/AIAssistant/

Option	Description
Quality preset	Speed / Balanced / Quality models
AI & models	LLM, vision model, thinking mode, timeouts, Ollama URL
Hotkeys	Global open and insert-reply shortcuts
Ask my files	RAG folder, enable/disable indexing
Theme	Follow system / Dark / Light
Advanced	Screen capture, image limits, system prompt, offline-only

---

Release

1. Bump the version in pyproject.toml and update CHANGELOG.md.
2. Push a version tag:

git tag v1.2.0
git push origin v1.2.0

3. CI (.github/workflows/build.yml) runs tests and creates a GitHub Release.
4. Publish the release on GitHub → publish.yml uploads olly-desktop to PyPI.

---

Project structure

ai-assistant/
├── bootstrap.py            # Console entry (detach from terminal)
├── main.py                 # App entry, tray, services
├── core/                   # Ollama client, settings, RAG, capture, platform
├── ui/                     # Popup, action bar, settings, onboarding, markdown
├── ui/styles/              # windows.py, macos.py, linux.py
├── utils/                  # Global hotkeys
└── tests/

---

Optional: OCR

Install Tesseract for text extraction from images when no vision model is available.

---

License

MIT