AutoDistiller

Find the best way to deploy your model — automatically.

The problem

You want to run a model on your GPU. Should you quantize it? To INT8, INT4, FP8? Which one keeps enough quality? Will it even fit in VRAM? How much faster is it actually?

Answering that by hand means compressing several versions, evaluating each, starting a server, benchmarking, and tracking what you measured. Hours of work, repeated whenever anything changes.

What AutoDistiller does

One command:

uv run autodistiller optimize --model Qwen/Qwen3-0.6B --max-vram 8GB --min-quality 95

It compresses the realistic options, checks quality against your baseline, benchmarks the survivors in a real vLLM or llama.cpp server, and tells you which to deploy — and what you give up by choosing it.

 Option                 Candidate   Wins on                      Gives up
 fastest (throughput)   int4-awq    peak throughput 641 tok/s    hellaswag/acc_norm 0.5625 against a
                                                                 best of 0.5781 (within noise -- not
                                                                 a measurable difference)
 best quality           int8-w-o    hellaswag/acc_norm 0.5781    peak throughput 463 against 641 tok/s

It reports error bars and says when a difference is too small to be real, because a recommendation built on noise is worse than no recommendation.

It does not implement quantization itself. It drives the tools that do — LLM Compressor, llama.cpp — measures the results honestly, and picks.

Install

pip install autodistiller

Check what it found on your machine:

uv run autodistiller env

CPU-only install: UV_TORCH_BACKEND=cpu uv sync

Try it

See the options before spending anything on them. Free — it is arithmetic, no GPU needed.

uv run autodistiller candidates --model Qwen/Qwen3-0.6B --max-vram 8GB

Measure your model as it is, so later numbers mean something.

uv run autodistiller evaluate --model Qwen/Qwen3-0.6B --task wikitext2

Compress one version and check what it cost.

uv run autodistiller compress --model Qwen/Qwen3-0.6B --method fp8

Or let it decide, and export the winner ready to serve.

uv run autodistiller optimize --model Qwen/Qwen3-0.6B --max-vram 8GB \
  --launch-preset wsl-vllm --export ./my-model

Nothing is measured twice — repeat any of these and it reuses the earlier result.

What it measured

Real output from the tool, on an RTX 5070 Laptop (8 GiB), vLLM 0.27, plugged in.

A complete run: Qwen3-4B on an 8 GiB card

bf16 Qwen3-4B is 7.49 GiB of weights, so it does not fit. Quantizing is not an optimisation here, it is the only way to run the model at all.

uv run autodistiller optimize --model Qwen/Qwen3-4B --max-vram 8GiB   --task hellaswag --task arc_easy   --method fp8 --method int4-awq --method int8-weight-only   --calibration wikitext2 --launch-preset wsl-vllm   --no-stop-early --objective throughput

It compresses each method, evaluates accuracy on both tasks, starts a real vLLM server for each survivor and benchmarks it, then ranks:

| Candidate                | Stage       | Quality               |     Size | Throughput | TTFT |
|--------------------------|-------------|-----------------------|----------|------------|------|
| int4-awq-ctx2048         | benchmarked | hellaswag 0.5625 ±.044| 2.48 GiB |  641 tok/s | 60ms |
|                          |             | arc_easy  0.7422 ±.039|          |            |      |
| int8-weight-only-ctx2048 | benchmarked | hellaswag 0.5781 ±.044| 4.17 GiB |  463 tok/s | 68ms |
|                          |             | arc_easy  0.7812 ±.037|          |            |      |
| fp8-ctx2048              | benchmarked | hellaswag 0.5703 ±.044| 4.12 GiB |  444 tok/s | 67ms |
|                          |             | arc_easy  0.7734 ±.037|          |            |      |

Search took 20.4 min -- 13.0 compressing, 25.1 evaluating

Then the trade-offs, and what each option costs:

Pareto frontier - hellaswag/acc_norm vs Peak VRAM vs TTFT p50 vs Peak throughput
| Candidate                | hellaswag/acc_norm | Peak VRAM | TTFT p50 | Peak throughput | Verdict        |
|--------------------------|--------------------|-----------|----------|-----------------|----------------|
| int4-awq-ctx2048         |             0.5625 |  7.75 GiB |     60ms |       641 tok/s | Pareto-optimal |
| fp8-ctx2048              |             0.5703 |  7.82 GiB |     67ms |       444 tok/s | Pareto-optimal |
| int8-weight-only-ctx2048 |             0.5781 |  7.82 GiB |     68ms |       463 tok/s | Pareto-optimal |

Recommendations
| Option               | Candidate        | Wins on                   | Gives up                          |
|----------------------|------------------|---------------------------|-----------------------------------|
| fastest (throughput) | int4-awq-ctx2048 | peak throughput 641 tok/s | hellaswag/acc_norm 0.5625 against |
|                      |                  |                           | a best of 0.5781 (within noise -- |
|                      |                  |                           | not a measurable difference)      |
| smallest             | int4-awq-ctx2048 | artifact size 2.48 GiB    | (same)                            |

What that tells you

Read the error bars. The accuracy spread across all three is 0.016 while the standard error on each is 0.044 — nearly three times larger. These models are not measurably different in accuracy, so int4-awq is 67% smaller and 44% faster for no cost anyone can demonstrate. The tool says "within noise" rather than letting you read 0.5625 vs 0.5781 as a real loss.

That 20.4 minutes is fast only because the cache reused four compressions and four benchmarks from an earlier search. A cold run of the same command is closer to an hour, most of it evaluation.

Qwen3-0.6B — where it is optional

Small enough that bf16 fits, so here you get quality retention against a real baseline:

Note the reversal: fp8 wins at 0.6B, int4-awq wins at 4B. At 4B on an 8 GiB card the model is memory-bandwidth-bound, so halving the weights beats cheaper arithmetic. You would not guess that; you have to measure it, which is the point.

Your numbers will differ. The same model on the same GPU measured 3x apart during development: once because the laptop was on battery (GPU capped to 34 W), once because another process held 5.9 GiB of the card. That is why every run records the hardware and software it ran on, and why a cached result is never reused across a change in either.

Which models work

Any Hugging Face causal LM. There is no supported-model list in the code and no per-architecture handling — dimensions come from config.json and loading goes through AutoModelForCausalLM. Every one of these was checked on an 8 GiB card:

Vision-language models are read by their language tower — Gemma 3 keeps the decoder in a nested text_config, and that is the half that gets quantized and whose KV cache dominates memory. The vision encoder is not counted.

Gated models (Llama, Gemma) need Hugging Face access: run hf auth login, then request access on the model page and wait for approval.

Not supported: encoder-decoder models (T5, BART), because loading goes through AutoModelForCausalLM. Non-NVIDIA accelerators are also out of scope — the capability rules are keyed on CUDA compute capability.

Compression methods

Methods marked yes need --calibration wikitext2 (or your own corpus) and fail immediately with a clear message otherwise. autodistiller methods prints this for your actual GPU, marking what it can and cannot run.

Commands

--help on any of them. autodistiller methods shows what your GPU and backend support.

What it supports

Decoder-only LLMs throughout — Llama, Qwen, Mistral, Phi and friends, plus the language half of a VLM. Encoder / embedding models (BERT, BGE, E5, GTE, MiniLM, RoBERTa) work end to end as well, with a smaller method list:

int4-awq is decoder-only: AWQ smooths activations through per-architecture mappings and llmcompressor registers none for encoders. GGUF on an encoder is untested, so it is not offered. MoE and encoder-decoder models (Whisper) are refused rather than mis-estimated.

Measured on an RTX 5070, bge-small-en-v1.5 against a real vLLM pooling server:

| Candidate               | Quality retention | Latency p50 | Throughput      |
| baseline batch 1        |           100.00% |        57ms |   112.6 texts/s |
| baseline batch 8        |           100.00% |        59ms |  1126.6 texts/s |
| int8-weight-only batch 8|           100.33% |        59ms |   995.8 texts/s |

Ten times the throughput for two milliseconds of latency — from batch size, not from quantization, which bought about one percent. And the retrieval task earns its place: int4-gptq holds 99.5% of STS-B similarity but only 96.8% of nDCG@10, so a model that looks lossless on the cheap screen measurably changes which documents it returns.

Status

Phases 1–9 are done, which is the whole v1.0 scope.

Embedding-model support is beyond the original ten phases: the measurement spine turned out to be model-agnostic, and only the loader, the memory arithmetic, the tasks and the serving flags needed a second implementation.

Both backends are verified end to end against real binaries: vLLM on GPU, and llama.cpp built from source and actually run — CPU-only so far, so GGUF throughput has not been measured on a GPU.

On Windows, llama.cpp's binaries are Linux executables, so build it in WSL and add --llama-cpp-wsl:

uv run autodistiller compress --model Qwen/Qwen3-0.6B --method gguf-q4-k-m \
  --llama-cpp ~/llama.cpp --llama-cpp-wsl

v1.0 targets Hugging Face models on NVIDIA GPUs, vLLM first and llama.cpp next, with INT4/INT8/AWQ/GPTQ and FP8 through existing backends.

Depth pruning landed from Phase 10: --prune 2,4 searches block count as a dimension alongside quantization, and the two compose — prune first, then quantize what is left. It is measured, not assumed, and on small models it usually loses: dropping 4 of Qwen3-0.6B's 28 blocks took wikitext-2 perplexity from 17.0 to 29.0, which the search reports as dominated rather than recommending it. 2:4 sparsity is not offered, because vLLM 0.27 removed sparsity support outright and llama.cpp never had it. Still post-v1: knowledge distillation, student-model search, and Bayesian optimization — the last only if the discrete search space proves limiting.

More

• Design notes — why it works this way, and the failures that shaped it
• vLLM on WSL — four undocumented blockers, and their fixes
• Contributing · Changelog

AutoTrainer is a separate project: it trains and fine-tunes, AutoDistiller takes a trained model and makes it deployable. They stay interoperable rather than merged.

License

Apache 2.0. See LICENSE and NOTICE.