distillery

Capture your LLM traffic through a drop-in proxy, curate it into a clean dataset, and distill a small model that replaces the frontier model for that workload.

app ──▶ distillery proxy ──▶ OpenAI / Anthropic
              │
              ▼ (captures)
        SQLite store ──▶ scrub ──▶ curate ──▶ dataset.jsonl ──▶ train ──▶ eval report

• One env var to adopt. Point OPENAI_BASE_URL or ANTHROPIC_BASE_URL at the proxy. Streaming, tool calls, and vision inputs pass through byte for byte.
• Safe by default. PII scrubbing is on, credentials are never written to disk, and nothing leaves your machine unless you choose a hosted training backend.
• Every stage is a command. JSONL in, JSONL out. Use the proxy alone, the scrubber alone, or the whole pipeline.
• Backend-agnostic training. Train locally with LoRA on one GPU, or submit the same dataset to Volition with a cost estimate up front.

60-second quickstart

uv tool install llm-distillery        # or: pip install llm-distillery
distillery init                   # pick an upstream, paste your keys once

distillery serve                                              # terminal 1
OPENAI_BASE_URL=http://localhost:8000/v1 python your_app.py   # terminal 2, unmodified app
distillery stats

Your app keeps working exactly as before. Each request is stored with its response, latency, token usage, and any tags you attach.

init writes your choices to ~/.distillery/distillery.toml and stores keys in ~/.distillery/credentials.toml with mode 600. You can also add or replace one key at a time:

distillery login openai          # prompts, hides input, verifies against the provider
distillery login volition --api-key sk-...
distillery login upstream        # generic key for OpenRouter, vLLM, or any OpenAI-compatible host
distillery logout anthropic
distillery config set proxy.upstream_openai https://openrouter.ai/api
distillery config show

Environment variables (OPENAI_API_KEY, ANTHROPIC_API_KEY, VOLITION_API_KEY, MODEL_API_KEY) and a .env file still work and take precedence over stored credentials.

From traffic to a small model

distillery export --scrub --out data/conversations.jsonl
distillery curate --in data/conversations.jsonl --out data/ --val-fraction 0.1
distillery train --data data/ --backend volition --base-model qwen3-4b
distillery eval --run runs/<name> --judge gpt-4.1-mini

export turns captures into vendor-neutral conversations with PII replaced by typed placeholders. curate filters errors, truncations, refusals, and duplicates, then writes train.jsonl, val.jsonl, a DATASET_CARD.md, and a curation_report.json. train fine-tunes a LoRA adapter. eval runs the held-out set through the fine-tuned model and the untouched base model, computes metrics for both against the reference, optionally asks a judge model to compare outputs pairwise, and writes eval_report.md. The base-model column is what tells you whether fine-tuning did anything.

A real run

examples/json_extraction captures receipt-extraction traffic and distills it into a 4B model. This is the report from 300 captured requests to gpt-4o-mini (via OpenRouter), curated into 270 train / 30 val rows, fine-tuned on Volition (qwen3-4b, LoRA, one $2.00 job), and evaluated on the 30 held-out prompts with gpt-4.1-mini as the pairwise judge. The baseline column is the same qwen3-4b before fine-tuning:

Candidate matches the reference on 73% of held-out prompts, is schema-conformant on 100%, wins or ties 97% of pairwise judgments (win rate 52%, 95% CI 47%-58%). Before fine-tuning the base model matched on 63%, was schema-conformant on 100%, won or tied 87% of judgments.

Training loss went from 1.13 to 0.055 over 340 steps and validation loss fell monotonically from 0.088 to 0.068, so the adapter learned the task rather than memorising it. The gain over the untuned model is real but modest here: the base model already produced valid, schema-conformant JSON, and 270 examples are only enough to nudge conventions such as whether payment_method keeps the card's last four digits and how DD/MM/YYYY dates are read. Thirty held-out cases cannot separate 63% from 73% with confidence; treat it as directional and rerun with more traffic.

The proxy

Request headers the proxy understands and strips before forwarding:

Every captured response carries an X-Distillery-Capture-Id header. Post feedback against it:

curl -X POST localhost:8000/distillery/feedback \
  -d '{"capture_id": "01J...", "score": 1.0, "label": "accepted"}'

Feedback drives the feedback filter and the preference dataset format for DPO.

Train on Volition

Volition is a hosted fine-tuning service with an OpenAI-compatible API. distillery talks to it directly.

export VOLITION_API_KEY=sk-...        # or put VOLITION_KEY=sk-... in a .env file
distillery train --data data/ --backend volition --base-model qwen3-4b --wait

The backend uploads your dataset, waits for validation, shows the estimated cost and your balance, asks for confirmation (--yes to skip), submits the job with an idempotency key, and streams training events until it finishes. The resulting ft: model id is written to runs/<name>/run.json.

distillery eval --run runs/<name> --judge gpt-4.1-mini

For a Volition run, eval resolves the model id, warms up serving, and calls https://api.volition.network/v1/chat/completions for you. The same model works with the OpenAI SDK by swapping base_url.

Useful commands: distillery volition models, distillery volition jobs, distillery volition job <id>, distillery volition cancel <id>.

Train locally

pip install 'llm-distillery[train]'
distillery train --data data/ --backend local --base-model Qwen/Qwen2.5-1.5B-Instruct
distillery train --data data/ --dry-run        # sequence-length stats, no training

Defaults target a single 24 GB GPU with a 1–3B base model: LoRA r=16, bf16, gradient checkpointing, cosine schedule. Presets are chosen from dataset size (tiny under 500 rows, small under 5k, medium above) and can be overridden with --preset or individual flags. Serve the adapter with vLLM, Ollama, or llama.cpp and pass --candidate-url to eval.

Privacy

• Only the two chat endpoints are captured, and only their JSON bodies plus metadata.
• Authorization, x-api-key, organization headers, and cookies are never persisted. A test greps the database file for the key to prove it.
• Scrubbing covers emails, phones, Luhn-valid cards, SSNs, IPs, IBANs, API keys, and JWTs, inside tool-call arguments and system prompts too. deny_if_found drops whole conversations instead of masking.
• Optional NER for names and locations: pip install 'llm-distillery[presidio]'.

Read docs/privacy.md for exactly what is and is not caught.

Configuration

Settings load from defaults, then distillery.toml in the working directory or ~/.distillery/, then DISTILLERY_* environment variables (nested with __, for example DISTILLERY_PROXY__PORT=9000), then CLI flags. A .env file in the working directory is loaded too. See distillery.example.toml.

Upstream credentials come from the client's own Authorization or x-api-key header. When the client sends none, the proxy falls back to OPENAI_API_KEY / ANTHROPIC_API_KEY, then to a generic MODEL_API_KEY. The same lookup serves eval when it calls the reference or judge model.

Using OpenRouter or another OpenAI-compatible provider

distillery serve --upstream-openai https://openrouter.ai/api
distillery eval --run runs/<name> --judge openai/gpt-4.1-mini   # judge goes through the same upstream

Development

uv sync
uv run pytest
uv run ruff check . && uv run mypy

Integration tests spin up the proxy against a fake upstream and drive it with the official openai and anthropic SDKs, including streaming and tool-call round trips. See CONTRIBUTING.md.

License

Apache-2.0