quantpilot 1.5.2

QuantPilot tells you which local LLM to run on your PC, in which format and runtime, and why. It sizes every option from the model's own files, checks it against your free memory, ranks the options for the kind of work you do, and measures speed and quality on your machine instead of guessing. Then it makes the file for you.

• Recommends the model, format (GGUF, AWQ, GPTQ, bitsandbytes, NVFP4), runtime (llama.cpp, vLLM, transformers) and context cache for chat, coding, RAG, agents, reasoning, long context, low latency or throughput, with the reasons and how sure it is.
• Sizes exactly: weights and context memory from config.json and the tensor headers (no download of the weights), checked against llama.cpp's own memory numbers on NVIDIA, Apple Silicon and CPU.
• Measures: speed and time to the first token with llama.cpp's benchmark, and quality as perplexity on WikiText-2, compared with the original model. Every result is kept, and the recommendations learn from it.
• Quantizes: one command (or one button) from a Hugging Face model or your own fine-tune to a checked, benchmarked file.

Install

QuantPilot runs with uv on Linux, macOS and Windows, Python 3.10 or newer.

uv tool install quantpilot          # the ocq and quantpilot commands
uvx quantpilot                      # or try it without installing

Open the app

Type ocq in a terminal: with no command it opens the full-screen app (arrow keys and Enter, no flags to remember). quantpilot and ocq tui do the same, and ocq --help lists the commands.

Quantization formats

Each quantization format is an extra, installed when you want it (they bring PyTorch, a large download):

uv tool install "quantpilot[gguf]"                              # GGUF: any PC
uv tool install "quantpilot[gguf,awq,bnb]" --torch-backend auto # NVIDIA: PyTorch for your driver

ocq doctor then says what is ready on your PC and gives the command for anything that is not. The app's This PC page installs the extras with a button. With pip: pip install "quantpilot[gguf]". From a clone of the repo: uv sync --extra gguf.

Quick start

ocq                                   # the full-screen app (uv run ocq in a clone of the repo)
ocq doctor                            # is everything ready on this PC?
ocq recommend --workload coding       # which model and format to run, and why
ocq feasibility Qwen/Qwen3-8B         # every format of one model on this PC
ocq optimize Qwen/Qwen3-8B -b gguf --gguf-type Q4_K_M --execute
ocq bench ocq-output/Qwen--Qwen3-8B/artifact/Qwen--Qwen3-8B-Q4_K_M.gguf --quality

A model is a Hugging Face id, a folder of your own fine-tune (saved with save_pretrained), a .gguf file, or an Ollama model (ollama:qwen3:8b). ocq models lists the ones already on your PC: Ollama, LM Studio, the Hugging Face cache, your folders.

Your own model

Point QuantPilot at the folder your training saved with save_pretrained (Unsloth, Axolotl, the Hugging Face Trainer, TRL): it needs config.json, the *.safetensors weights and the tokenizer files. Merge LoRA adapters into the model first: an adapter alone is not a model.

ocq feasibility ./my-model                                    # every format, on this PC
ocq optimize ./my-model -b gguf --gguf-type Q4_K_M --execute  # or AWQ, GPTQ, bitsandbytes, NVFP4
ocq bench ocq-output/local--my-model/artifact/local--my-model-Q4_K_M.gguf --quality

The file remembers the folder it was made from, so --quality compares it with your original (measured once): Q4_K_M keeps about 77% of the model's quality. Every format works the same way: GGUF, AWQ, GPTQ, bitsandbytes and NVFP4.

In the app, My models → + Add your model opens a folder browser that checks what you pick (✓ config.json, weights, tokenizer, or what is missing: a LoRA adapter to merge first, .bin weights to save again). Once added, the model is listed with the others and its menu offers Check, Quantize and Measure its quality.

The app

Run ocq for the full-screen app. Check a model gives the pick with its reasons and confidence, and every format's memory, speed and quality side by side. Quantize makes it with live progress. Benchmark measures any model on the PC and keeps the history. Discover ranks models for a kind of work, Settings keeps a Hugging Face token for gated models.

How it decides

Options that cannot run here are ruled out with the reason. The rest are ranked by the workload's priorities in order (coding: quality, then context, then speed), never by one blended score. Each pick says why it won, what the next options would give and cost, and how sure it is (Low to Very High) with the evidence.

Formats and where they run

Checked against llama.cpp's own numbers on Windows 11 with an RTX 4060 Laptop, Linux (WSL2) with NVIDIA and CPU only, and macOS on Apple Silicon, with GPU and CPU only.

Commands

For scripts and agents

recommend, bench, models, inspect, feasibility, profile, history and doctor take --json, and Python gets the same data:

from ocq import api

pick = api.recommend(workload="coding")["best"]
print(pick["model"]["id"], pick["format"]["label"], pick["quality"]["kept"])

The shape is versioned and only grows within a version: see docs/api.md.

Learn more

• User guide: every command and option, how memory, speed and quality are worked out, fine-tunes on disk, GPUs, mixture of experts, environment variables
• JSON and Python API
• Contributing and changelog

MIT license.