Features

• Background server — start/stop/restart llama-server as a daemon
• Multi-model routing — uses llama.cpp's own preset file, so nothing llb writes is llb-specific
• Interactive downloads — search HuggingFace, pick a quant, download with progress and resume
• Rich terminal UI — tables, panels, interactive selectors, and live search
• GGUF inspector — view model metadata, architecture, and sampling parameters
• Server props — inspect active sampling parameters on loaded models
• Sampling sync — automatically applies GGUF-recommended sampling params to your preset
• Per-model settings — context size, GPU layers, flash attention, and more
• Multi-token prediction — models that ship an MTP head get speculative decoding switched on by themselves. A head shipped as a separate file is downloaded with the model and only wired in once the installed llama.cpp is confirmed to load it; the check runs again after every llama.cpp upgrade
• Idle sleeping — llama-server releases the weights of a model after a configurable idle timeout
• VRAM tracking — automatically parses server logs to show memory usage per model
• Live changes — downloads, removals and settings changes reload the running router's model list instead of restarting it

Screenshots

Installation

pipx install llama-buddy

Or with uv:

uv tool install llama-buddy

This installs the llb command into an isolated environment and adds it to your PATH.

Prerequisites

• Python 3.10+
• llama.cpp installed and llama-server on your PATH

Quick start

# Download a model (interactive search)
llb download

# Or specify directly
llb download mistralai/Ministral-3-3B-Instruct-2512-GGUF:Q4_K_M

# Start the server
llb start

# List all models
llb models

# Chat with a model (uses llama-cli)
llb chat

# Inspect model metadata
llb info

# Show active sampling params for a loaded model
llb props

# Apply GGUF-recommended sampling params to all models
llb info --apply-sampling

# Configure settings (interactive TUI)
llb settings

# Open the web UI in your browser
llb open

# Stop the server
llb stop

Commands

Configuration

llb keeps no configuration format of its own. Settings live in llama.cpp's preset file, which llama-server --models-preset reads, and the port lives in llama.cpp's config.ini. Anything llb writes works the same without llb.

The router lists every cached model by itself, so the preset only has a section for a model whose settings differ from the defaults. [*] holds the settings for all models; a model's own section overrides them. Defaults are llama-server's own: context from the model (shrunk to fit memory), no idle sleep. llb edits the file line by line, so comments and hand-written keys survive.

Using your own preset file

If you already point llama-server at a preset, llb uses that one instead. It looks at LLAMA_ARG_MODELS_PRESET, then models-preset in ~/.config/llama.cpp/config.ini. Sections can give a model any name and point it at an HF repo or a local file:

[*]
sleep-idle-seconds = 300

[coder]
hf-repo = unsloth/Qwen3.8-27B-GGUF:Q8_0
temp = 0.6

[laguna]
model = /data/models/laguna-s-2.1-Q4_K_M.gguf

Per-model settings

Run llb settings and select Model Settings to configure per-model overrides:

• Context size, GPU layers, flash attention
• Custom aliases
• Any llama-server parameter

Other files in ~/.config/llama/ are llb's own state: server.pid, server.log, models.effective.ini (the copy the router reads, with context clamped to each model's maximum), and caches for VRAM usage, MTP detection and llama-server's flags.

Development

# Clone and install
git clone https://github.com/thilomichael/llama-buddy.git
cd llama-buddy
uv sync

# Run
uv run llb <command>

# Test
uv run pytest

# Lint
uv run ruff check src/ tests/

License

MIT