smol-llm-proxy

A small API proxy for self-hosted llama.cpp setups. Routes across multiple llama-server instances, per-user API keys, token usage tracking. In-memory rate limiting (RPM/TPM) with async SQLite flush. ~0.13 ms is proxy logic (per-request), the 2–4 ms end-to-end figure includes the FastAPI/uvicorn/httpx baseline and asyncio event-loop contention at high concurrency. <=1100 code lines, ~53 MB RAM.

Built for the case where you run multiple llama-server instances (different models, different GPUs) and want to share them across users with token tracking.

What it is not

smol-llm-proxy is built for one specific case: multiple llama-server instances, multiple users, per-user token accounting.

• LiteLLM — much broader scope: 100+ cloud providers, virtual keys, budgets, admin UI, fallbacks. Requires Postgres + Redis for full features. Use it if you need a production gateway across cloud LLMs.
• llama-swap — solves a different problem: hot-swapping models on one llama.cpp instance. No users, no accounting. Use it if you run many models on one machine and want them loaded on demand.
• llama.cpp router mode — built into llama-server itself. Same scope as llama-swap, no auth layer.

If you self-host several llama-server instances on one or more machines and want to share them with a small group while tracking usage, smol-llm-proxy is the smallest thing that does that. Otherwise, one of the above is probably a better fit.

Architecture

[users] ──HTTP──> [proxy :port] ──HTTP──> [llama-server 1 :port]
                       │                  [llama-server 2 :port]
                       │                  [llama-server N :port]
                       │
                       ├── in-memory cache (keys, routes) — TTL 30s
                       ├── validate API key + resolve routing (SQLite on first call, then cache)
                       ├── rate limiter: read-only DB check + in-memory reservation + reconcile on response
                       ├── async batch flush to SQLite (1s interval)
                       ├── forward request via connection-pooled httpx client
                       ├── async usage logger (batch queue, 50 items / 1s timeout)
                       └── retention cleanup (90-day purge, daily)

TLS termination is handled externally — Cloudflare, Caddy, nginx, or any reverse proxy in front of the proxy.

Features

• Per-user API keys (create / delete / toggle active)
• Rate limiting — per-key RPM/TPM, sliding 60-second window, 429 + Retry-After on exceed
• Multi-server routing by model name with in-memory cache (TTL 30s)
• Model aliases (alias -> model-name.gguf)
• Token usage logging — prompt/completion tokens, timings, 90-day retention with daily purge
• Streaming and non-streaming proxy support for chat, completions, and embeddings
• Connection-pooled httpx client (keepalive connections to backends)
• SQLite backend (zero external DB dependencies)

Quick Start

Docker Compose (recommended)

Clone the repo, then:

cp .env.example .env                # set ADMIN_KEY
cp config.example.yaml config.yaml  # fill in your servers
docker compose up -d --build

The proxy listens on 0.0.0.0:8000 by default.

Plain Docker

docker build -t smol-llm-proxy .
docker run -p 8000:8000 \
  -e ADMIN_KEY=secret \
  -v db-data:/app/data \
  -v $(pwd)/config.yaml:/app/config.yaml:ro \
  smol-llm-proxy

Pip install

pip install smol-llm-proxy

Example configs ship with the package. Copy them:

python -c "import smol_llm_proxy, shutil, os; d=os.path.dirname(smol_llm_proxy.__file__); shutil.copy2(f'{d}/config.example.yaml','config.yaml'); shutil.copy2(f'{d}/.env.example','.env')"
# Edit .env and set ADMIN_KEY, then uncomment and fill in config.yaml with your servers
ADMIN_KEY=secret smol-llm-proxy
# or: ADMIN_KEY=secret python -m smol_llm_proxy

Or download from GitHub:

curl -sO https://raw.githubusercontent.com/robolamp/smol-llm-proxy/main/.env.example && cp .env.example .env
curl -sO https://raw.githubusercontent.com/robolamp/smol-llm-proxy/main/config.example.yaml && cp config.example.yaml config.yaml

From source

git clone https://github.com/robolamp/smol-llm-proxy.git
cd smol-llm-proxy
pip install -e .

Copy example configs and run:

cp .env.example .env                # set ADMIN_KEY
cp config.example.yaml config.yaml  # fill in your servers
ADMIN_KEY=secret python -m smol_llm_proxy

Quick usage

1. Create a user key:

curl -X POST http://localhost:8000/admin/keys \
  -H "Authorization: Bearer $ADMIN_KEY" \
  -d '{"name": "my-user"}'

The response contains a JSON object with a key field — that's the user's Bearer token. Save it; you'll need it for proxy requests.

2. Send a chat completion with the user key:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-<full-key-from-step-1>" \
  -d '{
    "model": "Qwen3.5-2B",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": true
  }'

Configuration

The proxy reads two files: config.yaml for routing and .env for runtime settings.

config.yaml — servers, models, aliases

Loaded into SQLite at startup, persisted across restarts:

servers:
  - name: my-server
    url: http://host:port
    api_key: ""              # optional, if llama-server requires auth
    models:
      - model-name.gguf

aliases:
  alias: model-name.gguf     # short name -> real model name

Config sync semantics

On startup, config.yaml is merged into the database:

• Servers/aliases present in config.yaml are upserted into the database (new ones created, existing ones updated).
• Servers/aliases present in the database but not in config.yaml are kept — YAML is additive, not destructive.
• Models listed under a server in YAML are synced; extra models on that server are removed.
• API keys, usage logs, and rate limits are not affected by config sync.
• The in-memory route cache is cleared after sync.

Servers and aliases managed via the Admin API persist across restarts. To remove them, use the Admin API (DELETE /admin/servers/{id}, DELETE /admin/aliases/{name}).

Docker volumes

The Compose setup mounts two volumes:

• ${DATA_DIR:-./data}:/app/data — SQLite database (DB_PATH=/app/data/proxy.db), persists across container restarts. Override with DATA_DIR in .env.
• ./config.yaml:/config/config.yaml:ro — config file, read-only. Set CONFIG_PATH=/config/config.yaml in Compose (hardcoded, not from .env).

curl -X PUT http://localhost:8000/admin/keys/1/limits \
  -H "Authorization: Bearer $ADMIN_KEY" \
  -d '{"rpm_limit": 200, "tpm_limit": 100000}'

curl -X POST http://localhost:8000/admin/servers/2/models \
  -H "Authorization: Bearer $ADMIN_KEY" \
  -d '{"model_name": "qwen3-30b-a3b-instruct.gguf"}'
# 409: Model 'qwen3-30b-a3b-instruct.gguf' already assigned to server 'server-a' (id=1). Use ?reassign=true to move.

curl -X POST "http://localhost:8000/admin/servers/2/models?reassign=true" \
  -H "Authorization: Bearer $ADMIN_KEY" \
  -d '{"model_name": "qwen3-30b-a3b-instruct.gguf"}'

curl -X PATCH http://localhost:8000/admin/aliases/fast \
  -H "Authorization: Bearer $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{"real_model_name": "new-model.gguf"}'

curl "http://localhost:8000/admin/usage?key_id=1&start_date=2024-01-01T00:00:00&end_date=2024-01-31T23:59:59" \
  -H "Authorization: Bearer $ADMIN_KEY"

curl "http://localhost:8000/admin/usage/summary/real?key_id=1&start_date=2024-01-01T00:00:00&end_date=2024-01-31T23:59:59" \
  -H "Authorization: Bearer $ADMIN_KEY"

Each request logs: user, server, model name, prompt/completion tokens, timings (ms), and total tokens. No conversation content is stored.

ADMIN_KEY=secret
PROXY_PORT=8000