Pengy 🐧

A local-first AI agent with tools. Desktop GUI, web UI, and command-line — all backed by the same agent core, talking to any OpenAI-compatible API.

What is Pengy?

Pengy is an LLM agent that runs on your own machine. It defaults to a local server — Ollama's OpenAI-compatible port — and also speaks to llama.cpp, vLLM, LM Studio, or any hosted OpenAI-compatible API (OpenAI, Groq, OpenRouter). It gives the model 16 built-in tools to operate on your filesystem, inspect images, run code, search the web, and more — all with your approval.

Three interfaces, one agent:

All three share the same core, tools, chat history, and config. Use whichever fits your flow.

Quick Start

Install

# Recommended — uv installs Pengy with a compatible Python automatically
curl -LsSf https://astral.sh/uv/install.sh | sh
uv tool install pengy

# Or with pip (Python 3.10+)
pip install pengy

That one command gives you the complete headless experience — agent core, terminal CLI and browser Web UI, no extra flags and no Qt download:

Desktop GUI (optional)

The Qt desktop app is the only piece that is not installed by default — it is a ~80 MB download and needs a display:

pip install "pengy[gui]"                     # or: uv tool install --force "pengy[gui]"
pengy

Prefer a native desktop app with no Python at all? The Rust and C++ editions ship AppImage, .deb, .dmg and Windows .zip builds:

https://github.com/patw/PengyR/releases

pengy[all] and pengy[desktop] are aliases that add the GUI. pengy[cli] and pengy[web] still work, but their dependencies are now part of the default install — asking for them adds nothing.

Add Pengy to your application menu (Linux)

A pip/uv-tool install gives you commands, not a menu icon. Ask the program to add one — user-level only, no sudo, nothing installed system-wide:

pengy --install-launcher      # add a menu entry + icon
pengy --uninstall-launcher    # remove it again

It writes ~/.local/share/applications/pengy.desktop plus a 256×256 icon under ~/.local/share/icons/hicolor/, and refreshes the desktop/icon caches. The entry launches this environment's interpreter, so it never picks up a different Pengy edition that happens to be earlier on your PATH.

If a pengy.desktop already exists that Pengy did not write — for example a launcher for the native AppImage build — the command refuses to replace it and shows you the existing Exec= line. Add --force if you really mean to overwrite it.

(On Linux, sudo dpkg -i pengy_*.deb already does all of this system-wide — the launcher command is for pip-style installs.)

Windows: use pengy-gui for shortcuts

pengy is a console program, so a shortcut to it shows a black console window behind the GUI. Point shortcuts at pengy-gui instead — it is packaged as a console-less launcher. pengy keeps its console so pengy --version still prints in a terminal.

CLI (interactive or single-shot)

First, make sure there is a model to talk to. The default endpoint is a local server, so all it takes is Ollama itself — and no API key, ever:

ollama serve                  # if it is not already running
ollama pull llama3.2          # any model you like
pengy-cli /models             # list what the endpoint offers
pengy-cli /model llama3.2     # select one

There is deliberately no default model: a local server ships none of its own, so naming one would simply fail on your first message. Until you pick one, Pengy says so and tells you how — it never sends an empty model name to the endpoint.

Using a different server (llama.cpp, vLLM, LM Studio) or a hosted API? Point Pengy at it once, and it is remembered:

pengy-cli /baseurl http://127.0.0.1:8080/v1     # llama.cpp
pengy-cli /baseurl https://api.openai.com/v1    # or a hosted API…
pengy-cli /apikey sk-...                        # …which needs a key

The same settings live in Settings in the GUI and Web UI.

pengy-cli
pengy-cli "What is the capital of France?"

Web UI

pengy-web

The web UI is for single-user personal use. For remote access, put it behind nginx with SSL; use --trusted-host to set the public hostname when reverse-proxying.

Features

• Local-first — Defaults to a local Ollama endpoint (no API key, no account). Also works with llama.cpp, vLLM, LM Studio, OpenRouter, Groq, OpenAI, or any OpenAI-compatible endpoint
• 16 built-in tools — Read files and inspect images; write and edit files transactionally; run bash (with sudo support) and Python; search the web and fetch URLs; explore directories, glob files, and search code; track multi-step ops with structured to-do lists; ask clarifying questions when instructions are vague
• Agentic workflow — The LLM chains multiple tool calls per turn, piping results from one into the next
• Tool confirmation — Three modes: auto-approve everything, auto-approve read-only tools only, or confirm every call
• Tabbed chat — Multiple concurrent chat sessions, each with its own worker thread
• Theme system — System/light/dark modes plus 8 accent colours; fonts scale with the UI
• Tasks — Reusable prompt templates with %placeholder% tokens for workflows you run on repeat
• Model discovery — Fetch available models from your endpoint with one click or /models
• File attachments — GUI: attach from the input bar; CLI: /attach or @path syntax
• Templated system message — Auto-fills {date}, {username}, {hostname}, {osinfo} at send time
• Persistent config — Settings, task templates, and chat history in ~/.config/pengy/, shared between all interfaces and across all editions (Python, Rust, C++)

Screenshots

Main chat UI	Settings / theme controls	Tasks templates

Configuration

Desktop: Click ⚙ Settings in the sidebar.
CLI: Run /config to view, /model <name> to switch models.
Web: Click ⚙ in the top-right navbar.

First run? Configure before you chat. Pengy keeps its credentials in its own settings file (~/.config/pengy/settings.json, shared by the CLI, Web UI and GUI). Environment variables like OPENAI_API_KEY are not read. From the CLI: /apikey <key>, /baseurl <url>, /model <name>; or open pengy-web → Settings.

If credentials are missing or wrong, Pengy tells you exactly that (and prints the commands above) instead of relaying the API's own env-var advice — and exits 2 so scripts can tell configuration failures apart from other errors (1; interactive mode always exits 0).

Tasks

Tasks are reusable prompt templates for workflows you repeat often — summarizing a YouTube video, drafting a release note, or running a code-review checklist. Open Tasks from the desktop sidebar to create, edit, delete, or play templates.

Use %placeholder% tokens anywhere in the template to ask for values when the task is played:

Summarize this YouTube video: %Youtube Video URL%
Always use the youtube transcription skill.

When you hit ▶ Play, Pengy collects each placeholder once, renders the full prompt, and sends it through the normal chat pipeline — tools, skills, history, and confirmation settings all work exactly like a hand-typed prompt. Tasks live in ~/.config/pengy/tasks.json, shared across all interfaces and editions.

Tools

Pengy gives the LLM these 16 tools to operate on your machine:

Skills

The 16 built-in tools cover the basics, but Pengy is designed to be extended with skills — local instruction files with optional helper scripts.

A local skill normally has a skillname/skillname_skill.md instruction file, optionally backed by a bash or Python helper. Put installed skills under ~/skills/ and list them in skill_index.md; Pengy's system instructions tell it to consult that index and read the selected skill before acting. For reusable packages, BotSkills lets you inspect a skill, download its ZIP, and review its manifest and helper scripts before installing it.

This means your Pengy can do whatever you need it to:

• Fetch weather from an API
• Control devices on your home network
• Query your local databases
• Generate reports from your own data
• Run system administration tasks
• Send notifications, emails, or messages
• Map repository structure and run test suites
• Anything you can describe in a prompt and a script

Skills are also self-authoring — ask Pengy to create one for you, and it writes the markdown, writes the script, and updates the index, all in one conversation.

📖 Start with BotSkills or read the full local guide: skills/README.md — covers the philosophy, how skills work, 4 complete examples, and how to make your own.

API Compatibility

Development

Project structure

pengy/
├── main.py              # Desktop GUI entry point
├── cli/                 # CLI entry point
├── core/                # Config, chat manager, tools, LLM client
├── ui/                  # Chat view, input, workers, settings, theme
└── web/                 # Flask app, routes, SSE, templates

Install from source

git clone https://github.com/patw/pengy.git
cd pengy
# CLI + Web UI (default install — no Qt needed)
uv sync

# Add the Qt desktop GUI
uv sync --extra gui

Or with pip:

```bash
pip install -e .            # CLI + Web UI
pip install -e ".[gui]"     # + Qt desktop GUI

Running tests

python -m pytest tests/ -v

Dependencies

The GUI depends on PySide6-Essentials, not the PySide6 meta-package: the GUI imports only QtCore/QtGui/QtWidgets/QtSvg, so PySide6-Addons (~175 MB Linux, ~332 MB macOS) is never downloaded.

Also Available

Pengy (Python) is the reference implementation. Two high-performance ports share the same ~/.config/pengy/ data directory:

All three offer the same 16 tools, durable image attachments, desktop theme controls, reusable task templates, three interfaces (GUI/CLI/Web), and full chat/task interop. PengyR and PengyCPP ship pre-built AppImage, .deb, .dmg, and .zip releases; Pengy (this one) installs from PyPI with pip install pengy, which includes the CLI and Web UI, plus the Qt GUI via pengy[gui].

Documentation

• Configuration reference — all settings.json fields explained
• Reverse proxy setup — nginx, Caddy, SSH tunnels, Docker
• Skills deep-dive — skill patterns, ~/.secrets, uv dependencies
• API compatibility — provider support, model discovery, local endpoints
• FAQ — common questions and troubleshooting
• Building from source — platform-specific build instructions
• Changelog — version history

License

MIT