AutoDistiller

Find the best way to deploy your LLM — automatically.

The problem

You want to run a model on your GPU. Should you quantize it? To INT8, INT4, FP8? Which one keeps enough quality? Will it even fit in VRAM? How much faster is it actually?

Answering that by hand means compressing several versions, evaluating each, starting a server, benchmarking, and tracking what you measured. Hours of work, repeated whenever anything changes.

What AutoDistiller does

One command:

uv run autodistiller optimize --model Qwen/Qwen3-0.6B --max-vram 8GB --min-quality 95

It compresses the realistic options, checks quality against your baseline, benchmarks the survivors in a real vLLM or llama.cpp server, and tells you which to deploy — and what you give up by choosing it.

 Option                 Candidate   Wins on                      Gives up
 fastest (throughput)   int4-awq    peak throughput 641 tok/s    hellaswag/acc_norm 0.5625 against a
                                                                 best of 0.5781 (within noise -- not
                                                                 a measurable difference)
 best quality           int8-w-o    hellaswag/acc_norm 0.5781    peak throughput 463 against 641 tok/s

It reports error bars and says when a difference is too small to be real, because a recommendation built on noise is worse than no recommendation.

It does not implement quantization itself. It drives the tools that do — LLM Compressor, llama.cpp — measures the results honestly, and picks.

Install

pip install autodistiller

Check what it found on your machine:

uv run autodistiller env

CPU-only install: UV_TORCH_BACKEND=cpu uv sync

Try it

See the options before spending anything on them. Free — it is arithmetic, no GPU needed.

uv run autodistiller candidates --model Qwen/Qwen3-0.6B --max-vram 8GB

Measure your model as it is, so later numbers mean something.

uv run autodistiller evaluate --model Qwen/Qwen3-0.6B --task wikitext2

Compress one version and check what it cost.

uv run autodistiller compress --model Qwen/Qwen3-0.6B --method fp8

Or let it decide, and export the winner ready to serve.

uv run autodistiller optimize --model Qwen/Qwen3-0.6B --max-vram 8GB \
  --launch-preset wsl-vllm --export ./my-model

Nothing is measured twice — repeat any of these and it reuses the earlier result.

What it measured

Real output from the tool, on an RTX 5070 Laptop (8 GiB), vLLM 0.27, plugged in.

Qwen3-4B — where quantizing is not optional

bf16 Qwen3-4B needs ~8 GB of weights, so it does not fit. Every option below is one you would actually have to choose between. Accuracy is acc_norm on 128 questions each from hellaswag and arc_easy:

Read the error bars. The accuracy spread is 0.016; the standard error is 0.044 — nearly three times larger. These models are not measurably different in accuracy, and int4-awq is 44% faster and 40% smaller. The tool says so rather than leaving you to notice:

 Option                 Candidate   Wins on                      Gives up
 fastest (throughput)   int4-awq    peak throughput 641 tok/s    hellaswag/acc_norm 0.5625 against a
                                                                 best of 0.5781 (within noise -- not
                                                                 a measurable difference)

That run took 20.4 min — most of it evaluation, and only that fast because the cache reused four compressions and four benchmarks from an earlier search.

Qwen3-0.6B — where it is optional

Small enough that bf16 fits, so here you get quality retention against a real baseline:

Note the reversal: fp8 wins at 0.6B, int4-awq wins at 4B. At 4B on an 8 GiB card the model is memory-bandwidth-bound, so halving the weights beats cheaper arithmetic. You would not guess that; you have to measure it, which is the point.

Your numbers will differ. The same model on the same GPU measured 3x apart during development: once because the laptop was on battery (GPU capped to 34 W), once because another process held 5.9 GiB of the card. That is why every run records the hardware and software it ran on, and why a cached result is never reused across a change in either.

Which models work

Any Hugging Face causal LM. There is no supported-model list in the code and no per-architecture handling — dimensions come from config.json and loading goes through AutoModelForCausalLM. Every one of these was checked on an 8 GiB card:

Vision-language models are read by their language tower — Gemma 3 keeps the decoder in a nested text_config, and that is the half that gets quantized and whose KV cache dominates memory. The vision encoder is not counted.

Gated models (Llama, Gemma) need Hugging Face access: run hf auth login, then request access on the model page and wait for approval.

Not supported: encoder-decoder models (T5, BART), because loading goes through AutoModelForCausalLM. Non-NVIDIA accelerators are also out of scope — the capability rules are keyed on CUDA compute capability.

Compression methods

Methods marked yes need --calibration wikitext2 (or your own corpus) and fail immediately with a clear message otherwise. autodistiller methods prints this for your actual GPU, marking what it can and cannot run.

Commands

--help on any of them. autodistiller methods shows what your GPU and backend support.

Status

Phases 1–9 are done, which is the whole v1.0 scope.

Both backends are verified end to end against real binaries: vLLM on GPU, and llama.cpp built from source and actually run — CPU-only so far, so GGUF throughput has not been measured on a GPU.

On Windows, llama.cpp's binaries are Linux executables, so build it in WSL and add --llama-cpp-wsl:

uv run autodistiller compress --model Qwen/Qwen3-0.6B --method gguf-q4-k-m \
  --llama-cpp ~/llama.cpp --llama-cpp-wsl

v1.0 targets Hugging Face models on NVIDIA GPUs, vLLM first and llama.cpp next, with INT4/INT8/AWQ/GPTQ and FP8 through existing backends. Phase 10 is deliberately post-v1: knowledge distillation, pruning, student-model search, and Bayesian optimization — the last only if the discrete search space proves limiting.

More

• Design notes — why it works this way, and the failures that shaped it
• vLLM on WSL — four undocumented blockers, and their fixes
• Contributing · Changelog

AutoTrainer is a separate project: it trains and fine-tunes, AutoDistiller takes a trained model and makes it deployable. They stay interoperable rather than merged.

License

Apache 2.0. See LICENSE and NOTICE.