⚗ Distillarium

Replace narrow LLM pipeline steps with tiny CPU-runnable Spirits. Distill a teacher API into a deployable 20–50M-param model. Same outputs, $0 at inference.

distillarium is a Python toolkit for distilling a teacher LLM (Gemini, Claude, GPT) into a tiny, task-specific model — a Spirit — that runs on CPU, edge, or browser with zero API dependency at inference.

v0.1.1 — alpha. The reference Spirit (Needle, tool calling) is published; the toolkit itself is single-task at the moment with more recipes coming. Pipeline-replacement claims in the docs are based on Needle's measured numbers (78% tool-name accuracy on held-out cuts). See Status below.

Pick this if: you're doing the same narrow LLM call thousands of times a day (intent, routing, NER, classification, tool calling) and want to stop paying frontier-API prices for it. Don't pick this if: you need open-ended generation, multi-step reasoning, or a generalist assistant — keep the frontier model for that.

Why distillation, not fine-tuning

Install

pip install distillarium[gemini]

Hello, Spirit

Three commands, one Spirit:

# 1. Distill a teacher into a Spirit using a recipe
distillery distill recipes/needle.tool-calling-v1.yaml

# 2. Taste the Spirit — held-out eval against the teacher
distillery taste spirits/needle.pt

# 3. Bottle the Spirit — export to ONNX / GGUF for deployment
distillery bottle spirits/needle.pt --format onnx

A real example — Needle (tool calling, 20.7M params)

Reproduces the reference Spirit at 78° proof on tool-name accuracy (held-out cuts of 100 examples):

# recipes/needle.tool-calling-v1.yaml
name: needle.tool-calling
version: 1

teacher:
  provider: gemini
  model: gemini-2.5-flash
  temperature: 0.9

mash:
  total_examples: 1000
  examples_per_call: 10
  tools_per_call: { min: 3, max: 6 }

student:
  arch: attention-only-glu
  d_model: 384
  n_heads: 6
  n_layers: 8
  max_seq_len: 256
  tokenizer: wordpiece-4096

cuts: { train: 0.9, eval: 0.1 }

still:
  epochs: 8
  batch_size: 16
  lr: 3.0e-4

tasting:
  metrics: [tool_name_accuracy, arg_key_f1, exact_call_accuracy]
  held_out: 100

Run it:

distillery distill recipes/needle.tool-calling-v1.yaml --out spirits/

Expected result on a single RTX 5090: ~30 minutes, ~$0.30 in Gemini Flash API, 78% tool-name accuracy on held-out, 0.73 arg-key F1, and 3% exact-call accuracy (the value-prediction weak spot we're working on — see Status).

The Distillation Vocabulary

Python API

from distillarium import distill, taste, bottle, Recipe

# Load a recipe
recipe = Recipe.from_file("recipes/needle.tool-calling-v1.yaml")

# Distill
spirit = distill(recipe)

# Taste (eval against held-out cuts)
notes = taste(spirit, held_out=100)
print(notes.metrics)
# {'tool_name_accuracy': 0.78, 'arg_key_f1': 0.73, 'exact_call_accuracy': 0.03, ...}

# Bottle (export)
bottle(spirit, format="onnx", out="spirits/needle.onnx")

What's in the box

• distillarium.engine — attention-only transformer architecture + tokenizer + trainer + inference
• distillarium.teacher — pluggable teacher backends (Gemini, more coming)
• distillarium.tasting — held-out evaluation + Tasting Notes generation
• distillarium.bottling — exporters (ONNX in v0.1, GGUF in v0.2)
• distillarium.cli — distillery distill | taste | bottle commands

How it compares

This sits in a specific gap in the existing distillation ecosystem:

Honest framing: for breadth and SOTA function-calling scores, TinyAgent is the right pick. For replacing one narrow LLM step in a production pipeline with a CPU-runnable artifact you can audit, fork, and ship — that's what Distillarium is for.

We plan to publish BFCL (Berkeley Function Calling Leaderboard) numbers for Needle in v0.2 so the comparison is apples-to-apples.

Status

Current release: v0.1.1 (alpha) — single-task: tool calling via Gemini Flash. The reference Needle Spirit is published; other Spirits listed on the site are roadmap items.

What's NOT solved in v0.1.1 (be honest)

• Argument-value accuracy. Exact-call sits at 3% on Needle. The WordPiece tokenizer splits JSON values awkwardly. v0.2's byte-level BPE is the fix.
• Only Gemini teacher is wired up. Claude and OpenAI providers are stubs.
• No BFCL score yet. v0.2 will publish.
• Tasting Notes are statistical. They don't catch semantic failures like predicting "Twitter" instead of "Instagram." LLM-as-judge eval is on the roadmap.

License

MIT.

Built on top of the Research Radar — Automate Capture's autonomous research-to-product pipeline. The reference Needle Spirit was distilled from a Show HN paper the Radar surfaced in May 2026.