ternary-quant 0.1.6

 _____     ___                _   
|_   _|__ / _ \ _  _ __ _ _ _| |_ 
  | ||___| (_) | || / _` | ' \  _|
  |_|     \__\_\\_,_\__,_|_||_\__|

GGUF/llama.cpp is the strongest default for local deployment of supported models, and current llama.cpp builds support selected multimodal models through libmtmd (including image input and experimental audio input). ternary-quant targets a different gap: HuggingFace/Python-native, component-first post-training ternary quantization of existing FP16 checkpoints. Decoder-LLM benchmarks retain 95–99% of FP16 quality (97.7% mean across seven models); multimodal/audio quality is model-card and validation-suite dependent.

Qwen2.5-7B:   15.2 GB FP16  →  9.6 GB triton_memory  ✓ fits 12 GB GPU (RTX 4070/3080)
Phi-3.5-mini:  7.6 GB FP16  →  4.6 GB triton_memory  ✓ fits  6 GB GPU (RTX 3060/GTX 1060)
Mistral-7B:   14.0 GB FP16  →  5.0 GB stored           (98.7% ARC quality, stays in Python/HF)
FLAN-T5-small:154 MB FP16  →  125 MB stored            (5.1× decoder component)

Quickstart

pip install ternary-quant
# or with uv
uv pip install ternary-quant

# Vision-Language Model
ternary-quant quantize-broad Qwen/Qwen2-VL-2B-Instruct \
  --output ./qwen2vl-quant \
  --components text_backbone vision_backbone \
  --scheme tritplane3

# Causal LM
ternary-quant quantize-broad Qwen/Qwen3-4B \
  --output ./qwen3-4b-quant --components text_backbone --scheme tritplane3

# Seq2seq / Audio
ternary-quant quantize-broad openai/whisper-small \
  --output ./whisper-quant --components decoder

# Generate
ternary-quant generate ./qwen2vl-quant --prompt "Describe this image" --runtime-mode cached

Or use a pre-quantized model directly:

from ternary_quant.inference import load_ternary_model, generate_text

model, tokenizer = load_ternary_model("AsadIsmail/Qwen3-1.7B-ternary", device="cuda", runtime_mode="cached")
print(generate_text(model, tokenizer, "What is ternary quantization?", max_new_tokens=100))

What it does

ternary-quant is post-training quantization (PTQ) — it takes supported existing FP16 HuggingFace checkpoints and pushes selected transformer weights toward {-1, 0, +1} (ternary). No retraining required.

PTQ ternary vs trained ternary (BitNet): BitNet-style models are trained from scratch to operate with ternary weights, which is different from converting an existing FP16 checkpoint after training. ternary-quant does PTQ: take an existing FP16 model (Qwen, Mistral, Whisper, VLMs) and quantize selected components without retraining. GGUF/llama.cpp supports trained ternary formats for compatible models, but not this repo's HF-native component-first PTQ workflow for arbitrary checkpoints.

Key results

95–99% decoder-LLM quality retention across 7 models (0.5B-7B), measured with lm-eval 0-shot (ARC, HellaSwag, WinoGrande):

Full results: docs/KNOWN_GOOD_MODELS.md

Validated paths cover decoder LLMs, selected VLMs (Qwen2-VL, Gemma-4, SmolVLM), seq2seq (FLAN-T5), and audio (Whisper).

How it compares

Choose the right tool:

• Standard decoder LLM, want max speed locally → GGUF (llama.cpp is faster on CPU, better ecosystem)
• PTQ ternary on decoder LLMs, care about peak quality → PT²-LLM (ICLR 2026, strongest ternary PTQ results on decoder LLMs)
• Need HF-native component-first ternary PTQ for a VLM, seq2seq, audio, or custom model → ternary-quant
• 4-bit LLM with best quality-per-bit → HQQ-4 (outperforms ternary-quant on standard LLMs)

Runtime modes

triton_memory and metal keep weights as packed 2-bit ternary codes throughout inference — no FP16 weight matrix is ever materialized. This trades speed for memory, enabling models that don't fit otherwise.

metal mode requires torch ≥ 2.7 (for torch.mps.compile_shader). On older torch it degrades to cached. Apple-Silicon MPS generation also needs torch ≥ 2.7 to avoid a GQA matmul crash present in 2.6.

Full details: docs/RUNTIME_MODES.md

Learn more

• Known good models — measured results and commands for validated architecture families
• Runtime modes — triton_memory, metal, CPU inference details
• Research track — RAST + CCRE for true 1.58-bit compression
• Examples — text, seq2seq, VLM walkthroughs
• Paper — full experimental results and method description (LaTeX)
• Release notes — what changed in this version

Honest positioning

• HQQ-4 beats ternary-quant on quality-per-bit for standard LLMs — use it when quality is the priority
• PT²-LLM and PTQTP achieve strong PTQ ternary results on decoder-only LLMs — ternary-quant's algorithmic contribution on that axis is parallel, not superior
• The broad path is a ternary PTQ research/runtime path, not a general replacement for strong INT4 deployments
• The research track achieves true near-ternary weights with a real quality cost — this is the active research frontier, not a production-ready path
• GGUF is better for local deployment of supported standard LLMs and selected multimodal models; ternary-quant's value is component-first ternary PTQ in the HuggingFace ecosystem