distill-anything 0.2.0

The open-source lifecycle framework for distilling any model: generate data from teachers, distill students, evaluate, and benchmark.

  ██████╗ ██╗███████╗████████╗██╗██╗     ██╗
  ██╔══██╗██║██╔════╝╚══██╔══╝██║██║     ██║
  ██║  ██║██║███████╗   ██║   ██║██║     ██║
  ██║  ██║██║╚════██║   ██║   ██║██║     ██║
  ██████╔╝██║███████║   ██║   ██║███████╗███████╗
  ╚═════╝ ╚═╝╚══════╝   ╚═╝   ╚═╝╚══════╝╚══════╝
              A N Y T H I N G
     Distill any model into one you own

generate → distill → judge → benchmark · white-box KD + black-box seqKD · any teacher (Claude / GPT / HF / Ollama) · LLM-as-judge report card · runs on a MacBook

Install · Proof · Report card · Examples · Teachers · Compared to · Roadmap

---

Big models know things. Small models ship. Distill Anything covers the whole distillation lifecycle — a teacher generates your dataset, a student trains on its logits or its words, a judge scores the result blind, and a benchmark prices it — with one YAML recipe schema that scales from a 16GB MacBook to a GPU cluster.

_{Live: distill smoke — full logit-KD pipeline, offline, <60s on a MacBook. loss 2.24 → 2.08, PASS.}

_{The full picture. What's shipped vs planned: status table.}

What it does

• Any teacher — claude, openai:gpt-4o-mini, ollama:llama3.2, or any local hf: model, selected by one spec string
• Synthetic data — distill generate turns seed prompts into a deduped instruction dataset, with self-instruct-style prompt expansion (--expand 5)
• Quality gating — an LLM judge scores every record 1–10 before training (--judge claude --min-score 7)
• White-box logit KD — forward KL (Hinton), reverse KL (MiniLLM-style), generalized JSD, temperature scaling, and top-k truncation so 50k-vocab KD fits in laptop memory
• Black-box seqKD — no logits needed; fine-tune on what an API teacher wrote
• LLM-as-judge eval — blind A/B, position-swapped to kill judge bias → win/tie/lose and one headline number: quality retention
• Report card — distill report writes a shareable REPORT.md: quality vs the teacher, p50/p95 latency, tokens/s, memory, $ per 1K tokens
• LoRA / QLoRA students — freeze the base, train adapters: 1–3B students on a 16GB laptop ([lora] extra); 4-bit QLoRA on CUDA ([qlora])
• Hardware-aware — CUDA (bf16) → Apple Silicon MPS → CPU, detected automatically; teachers load in half precision on GPU/MPS

How it works (30 seconds)

One DistillConfig YAML describes the whole run; the same recipe format works for a 135M student on a MacBook and a 7B student on an A100.

Get started (60 seconds)

# 1 — Install
pip install distill-anything                    # core (torch, transformers)
pip install "distill-anything[anthropic]"       # optional: Claude as teacher/judge
pip install "distill-anything[openai]"          # optional: OpenAI / vLLM / Ollama teachers

# 2 — Verify your machine (tiny random models, zero downloads, <1 min)
distill smoke

# 3 — First real distillation (SmolLM2-360M → 135M, ~1GB download, runs on MPS)
git clone https://github.com/AIAnytime/distillanything && cd distillanything   # example seeds + recipes
distill generate examples/data/seed_prompts.txt \
  --teacher hf:HuggingFaceTB/SmolLM2-360M-Instruct --out data/train.jsonl --expand 5
distill train recipes/mac-small.yaml
distill report runs/mac-small --dataset data/train.jsonl \
  --teacher hf:HuggingFaceTB/SmolLM2-360M-Instruct --judge hf:HuggingFaceTB/SmolLM2-360M-Instruct

Prefer Claude as the teacher? export ANTHROPIC_API_KEY=... then swap --teacher claude and distill train recipes/claude-blackbox.yaml.

Proof

We publish only what we've measured. Every number below reproduces on a stock Apple-Silicon MacBook with the commands shown — no cluster required.

End-to-end pipeline, offline, under a minute (distill smoke):

Training mode=logit device=mps steps=30 loss=forward_kl(T=2.0, alpha=0.5)
  step 10/30  loss 2.2371   →   step 30/30  loss 2.0767
PASS: loss 2.237 -> 2.077 over 30 steps

Benchmark output (distill benchmark sshleifer/tiny-gpt2 --n-runs 3 --cost-per-hour 1.20, measured on an M-series MacBook):

Metric	Value
latency_p50_s	0.054
latency_p95_s	0.079
tokens_per_s	262.4
memory_mb	0.4
cost_per_1k_tokens_usd	0.001271

A real distillation, end to end on one MacBook — the exact Quickstart above (SmolLM2-360M teaches 135M on 60 records it generated itself, ~15 min total, all local):

	Student (135M, distilled)	Teacher (360M)
Parameters	134.5M	361.8M
Tokens / s (MPS)	70.5	56.7
Latency p50	0.40s	0.76s
Disk	538MB	1447MB
$ / 1K tokens @ $1.20/hr	$0.0047	$0.0059

• Quality retention 87.5% on 24 held-out prompts (1 win / 20 ties / 3 losses), judged blind with position swap by the teacher itself. A 360M judge is coarse — most pairs tie; point --judge claude at it for sharper discrimination.
• Token-level teacher agreement 81%, perplexity 2.17 (3-example eval split — demo scale).
• The full unedited artifact: examples/sample-report.md, generated by distill report.

This is the quickstart, not a leaderboard claim — 60 records and 8 optimizer steps prove the loop runs on a laptop, not the ceiling of the method.

Honest-by-construction evals: the judge sees answers blind and judges each pair twice with positions swapped — a judge that always prefers "Answer A" nets out to a tie (that exact adversarial case is in the test suite). Disagreements and unparseable verdicts count as ties, never wins. 36 tests run fully offline against tiny random models.

The report card

The question that matters is "did the student keep the teacher's quality at a fraction of the cost?" — so every run can end with a one-page answer:

distill report runs/mac-small \
  --dataset data/train.jsonl \
  --teacher hf:HuggingFaceTB/SmolLM2-360M-Instruct \
  --judge claude --n 32 --cost-per-hour 1.20

REPORT.md leads with quality retention (how often the student matches or beats the reference), then a side-by-side efficiency table (params, tokens/s, p50/p95, memory, $/1K tokens — "3.0x smaller and 3.0x faster"), sample outputs, and the training metrics. report.json sits next to it for machines.

Examples, notebook, and tests-as-docs

Three ways to learn the framework, all installing from PyPI (pip install distill-anything or uv pip install distill-anything):

📓 The notebook — examples/sample_notebook.ipynb walks the entire lifecycle with commentary: install → offline self-test → teacher-generated dataset → logit KD → rendered report card → benchmark → swapping in Claude/GPT teachers. Section 1 runs instantly offline; the full tour is ~15 min on a MacBook.

🐍 The scripts — the same flow as three standalone, heavily commented files you can copy into your own project:

Script	What it teaches
examples/01_generate_dataset.py	Teacher specs, seed prompts, self-instruct expansion, dataset JSONL format
examples/02_train_student.py	Student.learn(), white-box vs black-box mode selection, the KD loss knobs
examples/03_evaluate_and_report.py	Blind position-swapped judging, quality retention, the REPORT.md artifact

🧪 The tests as living docs — the suite runs offline in seconds and doubles as executable documentation:

Test file	Read it to learn
tests/test_losses.py	The KD math contracts: non-negativity, masking, top-k truncation, backprop
tests/test_judge.py	How to fake a Teacher for your own tests; why position-swap kills judge bias
tests/test_trainer.py	Training end-to-end with tiny random models (the pattern for fast iteration)
tests/test_report.py	The report data shape and a full build-report integration run

git clone https://github.com/AIAnytime/distillanything && cd distillanything
pip install "distill-anything[dev]" && pytest    # green in seconds, no GPU/keys/network

Teachers

One string selects any knowledge source — as teacher or as judge:

Spec	Backend	KD mode
hf:HuggingFaceTB/SmolLM2-360M-Instruct	local Hugging Face model	white-box (logits)
claude / claude:claude-opus-4-8	Anthropic API	black-box (seqKD)
openai:gpt-4o-mini	OpenAI API (or any compatible endpoint)	black-box
ollama:llama3.2	local Ollama server	black-box

One rule to know: logit KD requires student and teacher to share a tokenizer (same model family). Across tokenizers, use mode: seqkd — cross-tokenizer logit alignment (ULD) is on the roadmap.

Recipes

Everything is a YAML recipe (distill init writes a starter):

mode: logit                # or seqkd
teacher: { spec: hf:HuggingFaceTB/SmolLM2-360M-Instruct }
student: { model: HuggingFaceTB/SmolLM2-135M-Instruct }
data:    { path: data/train.jsonl, max_seq_len: 512 }
loss:    { kind: forward_kl, temperature: 2.0, alpha: 0.5, top_k: 256 }
train:   { output_dir: runs/out, lr: 1.0e-4, epochs: 2, batch_size: 2, grad_accum: 8 }

Or skip YAML entirely:

from distillanything import Student

student = Student("HuggingFaceTB/SmolLM2-135M-Instruct")
student.learn(teacher="claude", dataset="data/prompts_only.jsonl")   # generates missing responses, then trains
print(student.generate("Explain what a database index is."))
print(student.benchmark())

LoRA & QLoRA — big students on small hardware

Full fine-tuning caps laptop students around ~360M (AdamW keeps two moments per weight). LoRA freezes the base and trains ~0.5% of the params, so a 1.5B student learns from a 3B teacher on 16GB of RAM — see recipes/mac-lora.yaml:

student = Student("Qwen/Qwen2.5-1.5B-Instruct", lora={"r": 16, "alpha": 32})
student.learn(teacher="hf:Qwen/Qwen2.5-3B-Instruct", dataset="data/train.jsonl",
              gradient_checkpointing=True, batch_size=1, grad_accum=16)

Adapters merge into the base on save by default, so the output stays a plain HF checkpoint (set merge_lora: false for tiny adapter-only saves — every distill command loads those transparently). On CUDA, add "qlora": True to load the frozen base in 4-bit (recipes/gpu-qlora.yaml); QLoRA is CUDA-only because bitsandbytes has no Apple Silicon backend. Install: pip install "distill-anything[lora]".

When to use · When to skip

Great fit if you…

• want a specialized model that's 10–100x smaller than the API model you're calling today
• have (or can seed) a few hundred domain prompts and an API key or a local teacher model
• work on a laptop — everything here is sized to run and iterate on 16GB of RAM

Skip it if you…

• need RAG or prompt engineering, not a trained model — distillation is for when the task is stable and volume is high
• need cross-tokenizer logit KD, multimodal, or distributed training today (roadmap, not shipped)
• expect a hosted platform — this is a framework; you bring the compute

What's real today vs the vision

The architecture diagram is the north star, not the changelog. Every box, mapped honestly:

Legend: ✅ shipped  ·  🚧 partial  ·  🗺️ planned, not yet built

Diagram section	Box	Status	Notes
Seed Prompts	Text / Docs, Custom	✅	.txt (one prompt/line) and .jsonl (prompt/messages/text)
Seed Prompts	Code	🚧	No special handling — treated as plain text, works but untuned
Teacher	Claude API, GPT API, Hugging Face (Local), Ollama (Local)	✅	hf: / claude / openai: / ollama: teacher specs
Synthetic Dataset Pipeline	Generate, Deduplication, Curated Dataset	✅	distill generate, normalized-content dedup
Synthetic Dataset Pipeline	Filtering	🚧	Empty/too-short response filter only — no content or toxicity filters yet
Synthetic Dataset Pipeline	Quality Scoring	✅	LLM-judge 1-10 scoring: distill generate --judge claude --min-score 7
Distillation Engine	Forward KL, Reverse KL, JSD, Top-k/Temp	✅	White-box logit KD, all three divergences + top-k truncation
Distillation Engine	Response Supervision, Instruction Tuning	✅	Black-box seqKD (fine-tune on teacher-generated text)
Distillation Engine	Preference (Optional)	🗺️	No DPO/preference-based distillation yet
Evaluation	Perplexity	✅
Evaluation	Teacher Agreement	✅	Win/tie/lose via blind, position-swapped LLM judge (distill report --judge) + token-level agreement
Evaluation	Accuracy/EM, BLEU/ROUGE/BERTScore, Safety/Bias Checks	🗺️	No task-benchmark or safety-eval harness yet
Benchmark	Tokens/s, Memory, Model Size	✅	distill benchmark
Benchmark	Latency	✅	p50/p95 over repeated runs (--n-runs)
Benchmark	Cost / 1K Tokens	✅	From measured throughput × your --cost-per-hour
Outputs	Model Weights, Tokenizer & Config, Training Logs, Benchmarks	✅	Saved to output_dir on every run
Outputs	Evaluation Report	✅	distill report writes a shareable REPORT.md + report.json per run
Core Capabilities	Reproducible Pipelines	✅	Seeded runs + full config snapshot saved alongside the checkpoint
Core Capabilities	Multi-Teacher Support, Multi-Modal, Distributed Training, Quantization/Export, Experiment Tracking	🗺️	One teacher/one device per run today; text-only; no W&B/MLflow hooks
Integrations	Hugging Face	✅	Models, tokenizers, chat templates
Integrations	Weights & Biases, MLflow, S3/GCS/Azure Blob, Docker/Kubernetes	🗺️	Not integrated yet
How you interact	Python SDK, CLI	✅	Student().learn(...) and distill ...
How you interact	REST API, Web UI	🗺️	CLI/SDK only for now

What's inside

• teachers/ — one Teacher abstraction; local HF models expose logits (white-box), API teachers expose text (black-box). Registry resolves spec strings.
• losses/kd.py — masked, temperature-scaled (T² gradient correction) forward KL / reverse KL / generalized JSD, with top-k support truncation and automatic vocab-padding alignment.
• data/ — JSONL/txt loading, chat-template rendering, prompt-masked tokenization, normalized dedup, teacher-driven generation with prompt expansion.
• train/trainer.py — hand-rolled loop (no HF Trainer): grad accumulation, cosine LR + warmup, grad clipping, KD+CE mixing, MPS-safe autocast policy.
• eval/judge.py — pairwise blind judging with position-swap debiasing; absolute 1–10 scoring for dataset gating.
• eval/benchmark.py — p50/p95 latency, throughput, peak memory, $/1K tokens.
• eval/report.py — REPORT.md + report.json builder.
• testing.py — tiny random Llama models + in-memory char tokenizer, so the whole suite runs offline.

Compared to

Excellent distillation trainers exist — the gap this project fills is the lifecycle around the training loop:

	Primary focus	Data generation from API teachers	Blind LLM-judge → report card	Sized for laptops
Distill Anything	Full lifecycle: generate → distill → judge → benchmark	✅	✅	✅
TRL (GKD)	RL/KD training methods in the HF ecosystem	—	—	GPU-oriented
DistillKit	KD training techniques	—	—	GPU-oriented
EasyDistill	KD training + data synthesis toolkit	✅	—	GPU-oriented

If you already have a curated dataset and a GPU and just want a training loop, TRL's GKD trainer is great — and our loss implementations follow the same literature (Hinton KD, MiniLLM, DistiLLM).

Roadmap

Beyond closing the 🗺️ gaps in the status table:

• LoRA/QLoRA students — distill into 1–3B students on 16GB of RAM (shipped in v0.2)
• Hidden-state / feature KD with learned projectors
• Cross-tokenizer logit distillation (ULD)
• Multi-teacher voting and ensembling
• VLM, embedding, and reranker distillation
• Eval harness integration (lm-eval-harness) and regression tracking

Everything in this repo is and stays Apache-2.0. A hosted layer (GPU orchestration, dashboards, model registry) may come later — the framework is the product, not the funnel.

Development

uv venv && uv pip install -e ".[dev]"
pytest            # 36 tests, fully offline (tiny random models, fake judges)
distill smoke     # end-to-end pipeline check on your hardware

PRs welcome — the test suite needs no GPU, no API keys, and no downloads.

License

Apache 2.0 — see LICENSE.