ternary-quant 0.1.6

Post-training ternary quantization for HuggingFace generative models

 _____     ___                _   
|_   _|__ / _ \ _  _ __ _ _ _| |_ 
  | ||___| (_) | || / _` | ' \  _|
  |_|     \__\_\\_,_\__,_|_||_\__|

Component-first post-training ternary quantization for HuggingFace models. HF-native PTQ for VLMs, seq2seq, and audio — with packed runtime modes for selected 7B-class models.

Try the live demo  |  Pre-quantized: Qwen3-1.7B · Qwen2-VL-2B · Gemma4-E2B

GGUF/llama.cpp is the strongest default for local deployment of supported models, and current llama.cpp builds support selected multimodal models through libmtmd (including image input and experimental audio input). ternary-quant targets a different gap: HuggingFace/Python-native, component-first post-training ternary quantization of existing FP16 checkpoints. Decoder-LLM benchmarks retain 95–99% of FP16 quality (97.7% mean across seven models); multimodal/audio quality is model-card and validation-suite dependent.

Qwen2.5-7B:   15.2 GB FP16  →  9.6 GB triton_memory  ✓ fits 12 GB GPU (RTX 4070/3080)
Phi-3.5-mini:  7.6 GB FP16  →  4.6 GB triton_memory  ✓ fits  6 GB GPU (RTX 3060/GTX 1060)
Mistral-7B:   14.0 GB FP16  →  5.0 GB stored           (98.7% ARC quality, stays in Python/HF)
FLAN-T5-small:154 MB FP16  →  125 MB stored            (5.1× decoder component)

---

Quickstart

pip install ternary-quant
# or with uv
uv pip install ternary-quant

# Vision-Language Model
ternary-quant quantize-broad Qwen/Qwen2-VL-2B-Instruct \
  --output ./qwen2vl-quant \
  --components text_backbone vision_backbone \
  --scheme tritplane3

# Causal LM
ternary-quant quantize-broad Qwen/Qwen3-4B \
  --output ./qwen3-4b-quant --components text_backbone --scheme tritplane3

# Seq2seq / Audio
ternary-quant quantize-broad openai/whisper-small \
  --output ./whisper-quant --components decoder

# Generate
ternary-quant generate ./qwen2vl-quant --prompt "Describe this image" --runtime-mode cached

Or use a pre-quantized model directly:

from ternary_quant.inference import load_ternary_model, generate_text

model, tokenizer = load_ternary_model("AsadIsmail/Qwen3-1.7B-ternary", device="cuda", runtime_mode="cached")
print(generate_text(model, tokenizer, "What is ternary quantization?", max_new_tokens=100))

---

What it does

ternary-quant is post-training quantization (PTQ) — it takes supported existing FP16 HuggingFace checkpoints and pushes selected transformer weights toward {-1, 0, +1} (ternary). No retraining required.

PTQ ternary vs trained ternary (BitNet): BitNet-style models are trained from scratch to operate with ternary weights, which is different from converting an existing FP16 checkpoint after training. ternary-quant does PTQ: take an existing FP16 model (Qwen, Mistral, Whisper, VLMs) and quantize selected components without retraining. GGUF/llama.cpp supports trained ternary formats for compatible models, but not this repo's HF-native component-first PTQ workflow for arbitrary checkpoints.

---

Key results

95–99% decoder-LLM quality retention across 7 models (0.5B-7B), measured with lm-eval 0-shot (ARC, HellaSwag, WinoGrande):

Model	Avg retain	Compression
Phi-2 (2.7B)	99.2%	1.8x
Qwen2.5-0.5B	99.1%	1.8x
Mistral-7B	98.7%	2.8x
Qwen3-4B	98.1%	1.3x
Qwen3-1.7B	97.5%	2.7x

Full results: docs/KNOWN_GOOD_MODELS.md

Validated paths cover decoder LLMs, selected VLMs (Qwen2-VL, Gemma-4, SmolVLM), seq2seq (FLAN-T5), and audio (Whisper).

---

How it compares

Tool	LLMs	VLMs	Seq2seq / Audio	PTQ ternary	Component selection	Stays in Python/HF
GGUF / llama.cpp	✓✓✓	✓ (selected models via libmtmd)	partial / experimental	trained only	partial	✗
bitsandbytes	✓✓	partial	✗	✗	✗	✓
GPTQ / AWQ	✓✓	partial	✗	✗	✗	✓
PT²-LLM / PTQTP	✓✓	✗	✗	✓	✗	✗
ternary-quant	✓✓	✓✓	✓✓	✓	✓	✓

Choose the right tool:

• Standard decoder LLM, want max speed locally → GGUF (llama.cpp is faster on CPU, better ecosystem)
• PTQ ternary on decoder LLMs, care about peak quality → PT²-LLM (ICLR 2026, strongest ternary PTQ results on decoder LLMs)
• Need HF-native component-first ternary PTQ for a VLM, seq2seq, audio, or custom model → ternary-quant
• 4-bit LLM with best quality-per-bit → HQQ-4 (outperforms ternary-quant on standard LLMs)

---

Runtime modes

Mode	Hardware	Weights	Speed	Use case
cached	CPU / GPU / MPS	FP16 in memory	fastest	Default — dequantizes once at load
triton_memory	NVIDIA GPU	packed 2-bit	~3-6x slower	Fit oversized models on small GPUs
metal	Apple Silicon	packed 2-bit	~3-4x slower	Fit oversized models on Mac

triton_memory and metal keep weights as packed 2-bit ternary codes throughout inference — no FP16 weight matrix is ever materialized. This trades speed for memory, enabling models that don't fit otherwise.

metal mode requires torch ≥ 2.7 (for torch.mps.compile_shader). On older torch it degrades to cached. Apple-Silicon MPS generation also needs torch ≥ 2.7 to avoid a GQA matmul crash present in 2.6.

Model	FP16 VRAM	triton_memory	Fits on
Phi-3.5-mini (3.8B)	7.6 GB	4.6 GB	RTX 3060 6GB
Qwen2.5-7B (7.6B)	15.2 GB	9.6 GB	RTX 4070 12GB

Model	Cached (Mac)	Metal (Mac)	Memory saved
Qwen3-1.7B	10.4 GB / 46 tok/s	4.6 GB / 12 tok/s	2.3x
Qwen2-VL-2B	16.8 GB / 52 tok/s	7.4 GB / 13 tok/s	2.3x

Full details: docs/RUNTIME_MODES.md

---

Learn more

• Known good models — measured results and commands for validated architecture families
• Runtime modes — triton_memory, metal, CPU inference details
• Research track — RAST + CCRE for true 1.58-bit compression
• Examples — text, seq2seq, VLM walkthroughs
• Paper — full experimental results and method description (LaTeX)
• Release notes — what changed in this version

---

Honest positioning

• HQQ-4 beats ternary-quant on quality-per-bit for standard LLMs — use it when quality is the priority
• PT²-LLM and PTQTP achieve strong PTQ ternary results on decoder-only LLMs — ternary-quant's algorithmic contribution on that axis is parallel, not superior
• The broad path is a ternary PTQ research/runtime path, not a general replacement for strong INT4 deployments
• The research track achieves true near-ternary weights with a real quality cost — this is the active research frontier, not a production-ready path
• GGUF is better for local deployment of supported standard LLMs and selected multimodal models; ternary-quant's value is component-first ternary PTQ in the HuggingFace ecosystem