✦ AI Assistant

A cross-platform desktop AI assistant that lives in your system tray and works on whatever you are already doing — selected text, screenshots, clipboard images, and your own documents. It runs through Ollama so models stay local by default, with optional support for remote or cloud Ollama endpoints when you choose.

What makes this different

Most AI tools today are browser tabs, IDE plugins, or single-platform utilities tied to one vendor’s cloud. This project is built around a different idea: bring a capable assistant to the OS layer, without replacing your apps or sending everything to a SaaS backend.

Novel behaviors this project incorporates

1. Selection action bar (not a radial menu, not a sidebar)
After you select text, a compact toolbar appears near the cursor with one-click intents: Explain, Summarize, Translate, Ask, Screen, and Ask my files. Other tools usually make you open a separate window and paste — here the intent is chosen in context.

2. Foreground-aware capture
Before the assistant takes focus, it remembers which window and cursor position you were using. Screen capture targets that window — not the assistant’s own popup. That avoids the common “screenshot captured my chat window” failure mode.

3. System-wide workflow, not app-specific
Works in browsers, editors, PDF readers, terminals, and more via OS-level selection and hotkeys — not only inside one host application.

4. Local RAG on a folder you own
Point at a watch folder (Documents, a project directory, etc.). Files are chunked and embedded with Ollama; Ask my files pulls relevant passages into the prompt. No per-file upload dance each session.

5. Vision from the desktop
Paste images, capture a window, or use the Screen action for LeetCode-style problems, UI mockups, or diagrams — with configurable vision timeouts and image sizing for slow or cloud vision models.

6. Native UI per platform
PyQt6 with dedicated styling for Windows (Segoe), macOS (translucent / SF Pro), and Linux (GNOME-inspired) — not a generic Electron shell.

7. Power-user controls others often hide
Quality presets, custom model names, thinking-mode toggle for Qwen3/cloud models, remote Ollama URL detection with adjusted timeouts, chat export, recent chats, tone chips (Shorter / Simpler / Formal), and insert-reply hotkey.

What others do that this project does not (by design)

• No bundled proprietary model — you install and choose models via Ollama.
• No multi-user cloud sync — chats live in your app data directory on your machine.
• No IDE-only scope — it is a general desktop assistant, not a code-editor extension.
• No always-on cloud — when Ollama runs locally, inference stays on your PC; cloud use is opt-in via your Ollama URL and model choice.

Features

• Selection action bar — Explain, Summarize, Translate, Ask, Screen, Ask my files
• Global hotkey — open assistant with current selection (Alt+S on Windows/Linux, ⌥S on macOS)
• Insert reply — paste the last AI response into any app (Ctrl+Shift+V / ⌘⇧V)
• Vision — paste images, screenshot button, window/screen capture from the action bar
• RAG — index a configurable folder with Chroma + Ollama embeddings
• Chat — streaming, recent chats, export, tone chips, safe markdown code rendering
• Settings UI — tabbed: general, AI & models, hotkeys, files, advanced
• First-run wizard — Ollama setup and model download with progress
• Tray integration — launch at login, new chat, settings, paste screenshot, quit
• Platform UI — Windows, macOS (liquid glass), Linux (GNOME-inspired)

Agent mode (beta)

Opt in via Settings → AI & models → Enable tools (beta). When enabled and your model supports Ollama tool calling (e.g. qwen3, llama3.2), the assistant can run a short read-only tool loop before answering:

• Search indexed files — RAG over your watch folder
• List / read files — only inside the configured watch folder (path-scoped; no arbitrary disk access)
• Read clipboard — current text clipboard
• Capture screen — OCR text from the foreground window (respects the screen-capture setting)

Read-only tools use the same local-first rules as chat (including offline-only mode). Requires a tool-capable Ollama model; without one, chat falls back to plain streaming.

Desktop actions (beta)

With Enable tools and Allow desktop actions both on in Settings, the AI can — each behind an Allow / Deny dialog — write text files inside the watch folder only (.txt, .md, .csv, .json, .log), paste text where you click after a 3-second countdown, open http/https links in your browser, and open documents from the watch folder.

Constraints: text insertion is unavailable on Wayland; on macOS it needs the same Accessibility permission as hotkeys. The assistant never runs programs, presses arbitrary keys, or moves the mouse.

MCP servers (beta)

Configure stdio MCP servers in Settings → Advanced → MCP servers. When agent mode and MCP are enabled, tools from connected servers are advertised to the model automatically. Anything the server does not mark with a read-only hint triggers an Allow / Deny dialog that shows the exact arguments before execution.

• Disabled in offline-only mode — MCP servers are third-party programs that may use the network.
• Trust on first connect — you must explicitly trust a server before it is started.
• Requires the server's runtime — e.g. Node.js for npx @modelcontextprotocol/server-filesystem ….

Example: add a filesystem server scoped to a notes folder (npx -y @modelcontextprotocol/server-filesystem ~/Notes). Ask the agent to find action items from meeting notes; it can search and read files in that folder, then summarize. If it needs to write todo.md, a non-read-only MCP tool shows the confirmation dialog first.

SSE/HTTP MCP transport and image tool results are not in this release.

Privacy & data

When Ollama runs locally, prompts and model output stay on your machine. If you use a cloud model (for example qwen3-vl:235b-cloud), inference goes to that endpoint — configure it explicitly in Settings → AI & models.

Installation

Requires Python 3.11+ (python.org on Windows, Homebrew or python.org on macOS, your distro's package manager on Linux).

pip install olly-desktop
olly

olly detaches from the terminal — safe to close the terminal after launch. Check the system tray (^ by the clock). On first launch, the setup wizard installs Ollama and downloads a model.

Updating

pip install --upgrade olly-desktop

Uninstalling

pip uninstall olly-desktop

Your chats and settings remain in the app data directory unless you delete them manually.

Platform notes

From source

git clone https://github.com/tp-0604/ai-assistant.git
cd ai-assistant
python -m venv venv
source venv/bin/activate        # Windows: venv\Scripts\activate
pip install -r requirements.txt
# Windows only:
pip install -r requirements-windows.txt

Install Ollama, then:

# macOS / Linux
./launch.sh
# or
python main.py

# Windows
launch.bat

Usage

Settings

Stored in the app data directory — editable via Settings:

Release

1. Bump the version in pyproject.toml and update CHANGELOG.md.
2. Push a version tag:

git tag v1.2.0
git push origin v1.2.0

3. CI (.github/workflows/build.yml) runs tests and creates a GitHub Release.
4. Publish the release on GitHub → publish.yml uploads olly-desktop to PyPI.

Project structure

ai-assistant/
├── bootstrap.py            # Console entry (detach from terminal)
├── main.py                 # App entry, tray, services
├── core/                   # Ollama client, settings, RAG, capture, platform
├── ui/                     # Popup, action bar, settings, onboarding, markdown
├── ui/styles/              # windows.py, macos.py, linux.py
├── utils/                  # Global hotkeys
└── tests/

Optional: OCR

Install Tesseract for text extraction from images when no vision model is available.

License

MIT