paroquant 0.1.2

ParoQuant — Pairwise Rotation Quantization for LLMs

ParoQuant

Pairwise Rotation Quantization for Efficient Reasoning LLM Inference

State-of-the-art INT4 quantization for LLMs. ParoQuant uses learned pairwise rotations to suppress weight outliers, closing the accuracy gap with FP16 while running at near-AWQ speed. Supports NVIDIA GPUs (vLLM, Transformers) and Apple Silicon (MLX).

Quick Start

NVIDIA GPU:

pip install "paroquant[vllm]"
python -m paroquant.cli.chat --model z-lab/Qwen3-8B-PARO

# or with Docker
docker run --pull=always --rm -it --gpus all --ipc=host \
  ghcr.io/z-lab/paroquant:chat --model z-lab/Qwen3-8B-PARO

Apple Silicon:

pip install "paroquant[mlx]"
python -m paroquant.cli.chat --model z-lab/Qwen3-8B-PARO

Models

All models are available on Hugging Face. Swap the model name in the commands above to try any of them.

Qwen3

Model	Checkpoint
Qwen3-0.6B	z-lab/Qwen3-0.6B-PARO
Qwen3-1.7B	z-lab/Qwen3-1.7B-PARO
Qwen3-4B	z-lab/Qwen3-4B-PARO
Qwen3-8B	z-lab/Qwen3-8B-PARO
Qwen3-14B	z-lab/Qwen3-14B-PARO
Qwen3-4B-Thinking-2507	z-lab/Qwen3-4B-Thinking-2507-PARO

Llama

Model	Checkpoint
Llama-2-7B	z-lab/Llama-2-7b-hf-PARO
Llama-3-8B	z-lab/Meta-Llama-3-8B-PARO
Llama-3-70B	z-lab/Meta-Llama-3-70B-PARO
Llama-3.1-8B-Instruct	z-lab/Llama-3.1-8B-Instruct-PARO

Want a model that's not listed? Open an issue and let us know.

Installation

git clone https://github.com/z-lab/paroquant && cd paroquant

pip install -e ".[vllm]"            # vLLM backend (GPU, recommended)
pip install -e ".[transformers]"    # Transformers backend (GPU)
pip install -e ".[mlx]"             # MLX backend (Apple Silicon)
pip install -e ".[optim,eval]"      # Optimization & evaluation

Or use Docker: docker run -it --gpus all --ipc=host ghcr.io/z-lab/paroquant:latest

Quantize Your Own Model

# 1. Optimize rotation parameters
experiments/optimize/4bit.sh Qwen/Qwen3-8B

# 2. Export to HF checkpoint (--mode real for INT4, --mode pseudo for FP16)
python -m paroquant.cli.convert \
  --model Qwen/Qwen3-8B \
  --result-dir output/Qwen3-8B \
  --output-path models/Qwen3-8B-PARO

Reproduction

See experiments/README.md for scripts to reproduce all results in the paper.

Docker Images

Image	Purpose
ghcr.io/z-lab/paroquant:latest	Optimization & evaluation
ghcr.io/z-lab/paroquant:chat	Interactive chat
ghcr.io/z-lab/paroquant:chat-cu130	Interactive chat (CUDA 13.0 / ARM64)
ghcr.io/z-lab/paroquant:eval-reasoning	Reasoning task evaluation

Citation

@inproceedings{liang2026paroquant,
  title     = {{ParoQuant: Pairwise Rotation Quantization for Efficient Reasoning LLM Inference}},
  author    = {Liang, Yesheng and Chen, Haisheng and Zhang, Zihan and Han, Song and Liu, Zhijian},
  booktitle = {International Conference on Learning Representations (ICLR)},
  year      = {2026}
}