TurboQuant
Model quantization & optimization toolkit for edge and resource-constrained deployment.
INT4 · INT8 · FP16 · GPTQ · AWQ · BitsandBytes · Structured pruning · ONNX & TensorRT export
Why TurboQuant?
Modern open-source models are powerful but expensive to serve. Shipping a 7B-parameter LLM in FP16 demands ~14 GB of VRAM; a vision transformer that fits comfortably on a workstation may blow up on a Jetson Orin or a phone. TurboQuant gives you a single, consistent interface to compress, quantize, prune, export, and benchmark models — so you can ship them on the hardware you actually have.
It is built around three principles:
- One API, many backends. Wrap
bitsandbytes,auto-gptq,autoawq, native PyTorch quantization, and ONNX/TensorRT export behind a uniformquantize(model, method=...)interface. - Reproducible benchmarks. Latency, peak memory, model size, and task accuracy (perplexity, classification top-1, etc.) are first-class citizens — every example ships with a comparable benchmark.
- No magic. Each technique is implemented as a small, readable module so it doubles as a reference for how the methods work.
Features
| Category | Techniques |
|---|---|
| Weight quantization | INT8 dynamic & static PTQ, FP16/BF16 casting, INT4 (bitsandbytes NF4 / FP4), GPTQ, AWQ |
| Pruning | Magnitude (unstructured), L1 structured (channel/filter), N:M sparsity helpers |
| Export | ONNX (with onnxslim graph optimization), TensorRT engine builder, ORT quantization |
| Calibration | Per-tensor & per-channel, MinMax / Entropy / Percentile observers |
| Benchmark | Latency (warmup + median + p95), peak GPU/CPU memory, throughput, model size, perplexity, top-k accuracy |
| CLI | turboquant quantize, turboquant prune, turboquant export, turboquant bench |
Installation
The PyPI package is named turboquant-ml (the unsuffixed turboquant
name was taken by an unrelated project). The Python import and CLI are still
just turboquant / tq:
# Core install
pip install turboquant-ml
# With ONNX export
pip install "turboquant-ml[onnx]"
# Full LLM compression stack (GPTQ + AWQ + bitsandbytes)
pip install "turboquant-ml[gptq,awq,bnb,eval]"
# Everything
pip install "turboquant-ml[all]"
import turboquant # import name unchanged
from turboquant import quantize # same API
Note —
bitsandbytes,auto-gptq,autoawqandtensorrtare heavy native dependencies. They are deliberately optional; TurboQuant degrades gracefully when they are missing.
Quick start
Python API
from turboquant import quantize, benchmark
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "meta-llama/Llama-3.2-1B"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto")
# One-line INT4 weight-only quantization via bitsandbytes
qmodel = quantize(model, method="bnb-nf4")
# Benchmark side-by-side
report = benchmark.compare(
baseline=model,
candidate=qmodel,
tokenizer=tok,
prompts=["Explain quantization in one sentence."],
metrics=["latency", "memory", "size", "perplexity"],
)
print(report.as_table())
CLI
# Quantize a HuggingFace model to INT4 with GPTQ + W4A16
tq quantize meta-llama/Llama-3.2-1B \
--method gptq \
--bits 4 \
--group-size 128 \
--calib-dataset wikitext \
--out ./outputs/llama-3.2-1b-gptq
# Structured prune a vision model and re-evaluate
tq prune microsoft/resnet-50 \
--strategy l1-channel \
--sparsity 0.30 \
--eval imagenet-val \
--out ./outputs/resnet50-pruned
# Export to ONNX with INT8 dynamic quantization
tq export ./outputs/resnet50-pruned \
--format onnx \
--quant int8-dynamic \
--opset 17
# Benchmark FP16 vs INT8 vs INT4 on a model
tq bench meta-llama/Llama-3.2-1B --methods fp16,int8-dynamic,bnb-nf4 \
--report ./benchmarks/results/llama32-1b.json
Supported methods at a glance
| Method | Bits | Backend | Calibration | Typical use case |
|---|---|---|---|---|
fp16 / bf16 |
16 | PyTorch | none | Fast, lossless-ish baseline |
int8-dynamic |
8 | PyTorch | none | CPU inference, transformers |
int8-static |
8 | PyTorch | required | CNNs, edge CPUs |
bnb-int8 |
8 | bitsandbytes | none | LLM training & serving on GPU |
bnb-nf4 / bnb-fp4 |
4 | bitsandbytes | none | LLM inference, QLoRA |
gptq |
2–8 | auto-gptq | required | LLM weight-only, best accuracy/bit |
awq |
4 | autoawq | required | LLM weight-only, fast inference |
Reference benchmarks
SmolLM2-135M on CPU (real measured numbers)
python benchmarks/scripts/sweep_cpu.py --model-id HuggingFaceTB/SmolLM2-135M --methods fp32,fp16,bf16,int8-dynamic
| Method | Size (MB) | Forward latency (ms) | Generation throughput (tok/s) |
|---|---|---|---|
| FP32 (baseline) | 513.2 | 31.3 | 32.6 |
| FP16 | 256.7 | 57.2 | 47.5 |
| BF16 | 256.7 | 55.4 | 48.9 |
| INT8 dynamic | 236.6 | 30.7 | 30.0 |
Read this carefully — the result is realistic, not flattering:
- FP16/BF16 cut size in half, and generation throughput goes up ~50% (smaller KV cache wins), but the per-step forward pass is 2× slower because consumer CPUs have no fast FP16 matmul kernel. On a Tensor-Core GPU these numbers flip.
- INT8 dynamic is the smallest (≈54 % off) and matches FP32 forward latency, but generation throughput is similar to FP32 here — the small hidden size of a 135 M model limits how much INT8 GEMM kernels can help.
- The right baseline matters: comparing INT8 to a poorly-quantizable
reference (e.g. GPT-2, which uses
transformers.Conv1Dinstead ofnn.Linear) makes INT8 look bad. Always check what your method actually rewrites —tq methodsplusprint(model)will tell you.
Reproduce
pip install -e ".[viz]" truststore
python benchmarks/scripts/sweep_cpu.py \
--model-id HuggingFaceTB/SmolLM2-135M \
--methods fp32,fp16,bf16,int8-dynamic \
--out benchmarks/results/smollm2_135m.json \
--plot benchmarks/results/smollm2_135m.png
GPU sweeps (Llama-class models with GPTQ / AWQ / NF4) will land here once a CUDA runner is added to CI — contributions welcome.
Architecture
turboquant/
├── quantization/ # Algorithms: int8, fp16, gptq, awq, bnb, observers
├── pruning/ # Magnitude + structured (L1, L2, taylor) + N:M
├── export/ # ONNX, TensorRT, ORT quantization
├── benchmark/ # Latency, memory, perplexity, classification, plot
├── calibration/ # Datasets, dataloaders, observer fitting
├── models/ # Convenience loaders + registry
└── cli.py # Typer-based CLI
Each algorithm lives in a single, readable file with a quantize_* / prune_* function and a short docstring referencing the original paper.
Roadmap
- INT8 dynamic & static PTQ (PyTorch native)
- FP16/BF16 casting
- BitsAndBytes INT8 / NF4 / FP4 wrappers
- GPTQ & AWQ integration
- L1 structured & magnitude pruning
- ONNX export with
onnxslim - Latency / memory / perplexity benchmarks
- TensorRT INT8 calibration cache
- SmoothQuant W8A8
- HQQ (Half-Quadratic Quantization)
- Distillation-aware quantization
- Mobile export (CoreML / TFLite)
- Web dashboard for benchmark comparison
Citing & related work
TurboQuant stands on the shoulders of giants. If you use it in research, please also cite the underlying algorithms:
- GPTQ — Frantar et al., 2023 (arXiv:2210.17323)
- AWQ — Lin et al., 2023 (arXiv:2306.00978)
- LLM.int8() / QLoRA — Dettmers et al., 2022 / 2023 (arXiv:2208.07339, 2305.14314)
- SmoothQuant — Xiao et al., 2022 (arXiv:2211.10438)
Contributing
Contributions are very welcome — see CONTRIBUTING.md. Good first issues are tagged on the issue tracker.
git clone https://github.com/Ademo93/turboquant
cd turboquant
pip install -e ".[dev,all]"
pre-commit install
pytest
License
MIT — do whatever you like, just keep the copyright notice.
Metadata
Release files for turboquant-ml 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| turboquant_ml-0.1.0.tar.gz | 138.4 kB | Without provenance | Details for turboquant_ml-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| turboquant_ml-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for turboquant_ml-0.1.0-py3-none-any.whl |
Total release size: 173.0 kB
Release files / turboquant_ml-0.1.0.tar.gz
| Download URL | turboquant_ml-0.1.0.tar.gz |
|---|---|
| Size | 138.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
cba93d26f7acb8e5b6319c6be1e538a3961e9cf925b0e7e41b8713efef58297a
|
|
BLAKE2b-256 checksum How to use checksums |
63e3dddb236317e6190821d1d31c7c80346bc8d825d3d646fb76dcc2a25c7fc4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.14.2
|
Release files / turboquant_ml-0.1.0-py3-none-any.whl
| Download URL | turboquant_ml-0.1.0-py3-none-any.whl |
|---|---|
| Size | 34.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
d1a0b46661fa7a92d54ec8406eeca4b37cd2f45c52becf5b4869698fe2354696
|
|
BLAKE2b-256 checksum How to use checksums |
d0087212809d0914dadce4323d706a97a485c42d79a31674f4e4807af918abe5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.14.2
|
Security for turboquant-ml 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.