SINQ: Sinkhorn-Normalized Quantization for Calibration-Free Low-Precision LLMs

⚡️ A fast, plug-and-play, model-agnostic quantization technique delivering state-of-the-art performance for Large Language Models without sacrificing accuracy.

💡 Want to run a large model on your GPU but don’t have enough memory? With SINQ, you can deploy models that would otherwise be too big drastically reducing memory usage while preserving LLM quality.

⏱️ SINQ quantizes Qwen3-14B in just ~21 sec and DeepSeekV2.5-236B in ~5 min

🆕 [17/10/2025] First models on 🤗 Hugging Face Hub:

We’ve started uploading our first pre-quantized models to the 🤗 Hugging Face Hub and will continue adding more soon. Note: We’re also actively working to add support for popular frameworks such as vLLM, SGLang, and llama.cpp to enable fast SINQ-ference (sorry for the joke).
In the meantime, you can ⭐️ star and watch the repo to stay updated!

🚀 Welcome to the official SINQ repository!

SINQ (Sinkhorn-Normalized Quantization) is a novel, fast and high-quality quantization method designed to make any Large Language Models smaller while keeping their accuracy almost intact.

🔍 What You’ll Find Here

• 1. How does SINQ work?
• 2. Why should I use SINQ?
• 3. Quantize (and save) any LLM with SINQ
• 4. How to reproduce paper results
• 5. Ongoing updates on new features and integrations
• 6. How to Cite This Work
• 7. Related Repositories

📊 Feature Comparison: SINQ vs HQQ (calibration-free) and A-SINQ vs AWQ (calibrated)

📄 Want to know more? Read our paper on arXiv!

1. How does SINQ work?

2. Why should I use SINQ?

3. Quantize any LLM with SINQ

Setup & Quick Start

First, install the dependencies and set up the package:

# 1. Clone the repository
git clone https://github.com/huawei-csl/SINQ.git
cd SINQ

# 2. Install dependencies
pip install -r req.txt

# 3. Install SINQ
pip install .

Quantize in a few lines

Quantizing any 🤗 Hugging Face model with SINQ is simple and takes only a few lines of code:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from sinq.patch_model import AutoSINQHFModel
from sinq.sinqlinear import BaseQuantizeConfig

model_name = "Qwen/Qwen3-1.7B"
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_name)

quant_cfg = BaseQuantizeConfig(
    nbits=4,            # quantization bit-width
    group_size=64,      # group size
    tiling_mode="1D",   # tiling strategy
    method="sinq"       # quantization method ("asinq" for the calibrated version)
)

qmodel = AutoSINQHFModel.quantize_model(
    model,
    tokenizer=tokenizer,
    quant_config=quant_cfg,
    compute_dtype=torch.bfloat16,
    device="cuda:0"
)

✅ That’s it. Your model is now quantized with SINQ and ready for inference or saving.

Optional Flags

You can further customize the quantization process to balance accuracy and memory for your needs.
Here’s a summary of the main arguments you can tune:

💡 Tip: For most cases, the defaults (--nbits 4 --tiling_mode 1D --group_size 64 --method sinq) provide an excellent trade-off between compression and accuracy.

Save & reload

If you want to reuse a quantized model later, save it to disk in HF-style sharded safetensors and reload without needing base FP weights.

Requires: pip install safetensors

# --- Save to a folder (sharded safetensors) ---
from sinq.patch_model import AutoSINQHFModel
import torch

save_dir = "qwen3-1.7b-sinq-4bit"  # any path

# 'model' must already be SINQ-quantized (e.g., via AutoSINQHFModel.quantize_model)
AutoSINQHFModel.save_quantized_safetensors(
    qmodel,
    tokenizer,
    save_dir,
    verbose=True,
    max_shard_size="4GB",   # typical HF shard size (use "8GB" if you prefer)
)

# --- Reload later--
from sinq.patch_model import AutoSINQHFModel
import torch

tokenizer = AutoTokenizer.from_pretrained(save_dir)
qmodel = AutoSINQHFModel.from_quantized_safetensors(
    save_dir,
    device="cuda:0",
    compute_dtype=torch.bfloat16,
)

# (optional) quick smoke test
prompt = "Explain neural network quantization in one sentence."
inputs = tokenizer(prompt, return_tensors="pt").to("cuda:0")
with torch.inference_mode():
    out_ids = qmodel.generate(**inputs, max_new_tokens=32, do_sample=False)
print(tokenizer.decode(out_ids[0], skip_special_tokens=True))

# --- Save to a folder (.pt) ---
from sinq.patch_model import AutoSINQHFModel

save_dir = "qwen3-1.7b-sinq-4bit"  # any path
AutoSINQHFModel.save_quantized(qmodel, tokenizer, save_dir, verbose=True)  # creates qmodel.pt

# --- Reload later from .pt ---
from sinq.patch_model import AutoSINQHFModel
import torch

tokenizer = AutoTokenizer.from_pretrained(save_dir)
qmodel = AutoSINQHFModel.from_quantized(
    save_dir,
    device="cuda:0",
    compute_dtype=torch.bfloat16,
)

Compatible with lm-eval evaluation framework

Below is a minimal example showing how to evaluate a SINQ-quantized model on a benchmark dataset:

from lm_eval import evaluator
from lm_eval.models.huggingface import HFLM

# Wrap the already quantized model and tokenizer with HFLM
lm = HFLM(pretrained=qmodel, tokenizer=tokenizer, device="cuda:0")

# Evaluate (many tasks available on lm-eval such as MMLU and HellaSwag)
results = evaluator.simple_evaluate(
    model=lm,
    tasks=["lambada_openai"],  # small and fast benchmark
    device="cuda:0"
)

4. How to reproduce paper results

# 1. Clone the repository
git clone https://github.com/huawei-csl/SINQ.git
cd SINQ

# 2. Install dependencies
pip install -r req.txt

# 3. Install SINQ
pip install .

cd tests
python quant_model_eval.py

python quant_model_eval.py --model_name Qwen/Qwen3-1.7B

python quant_model_eval.py --method sinq_nf4 --model_name Qwen/Qwen3-1.7B

python quant_model_eval.py --method asinq --model_name Qwen/Qwen3-1.7B

5. Ongoing updates on new features and integrations

We are actively expanding SINQ with new features and integrations. Stay tuned here for the latest updates:

• [26/09/2025] - SINQ paper released on arXiv
• [30/09/2025] - SINQ GitHub repository made public
• [02/10/2025] - SINQ paper featured on 🤗 Hugging Face Papers
• [17/10/2025] - First pre-quantized SINQ models available on 🤗Hugging Face Hub! 🆕
• [23/10/2025] - Faster inference with gemlite backend (4-bit 1D tiling)
• 🔜 Coming soon - 🤗 Integration with Hugging Face Transformers
• 🔜 Coming soon - Support for Conv2D layers and timm models for computer vision tasks
• 🔜 Work in progress - Support for mixed-precision quantization (combine multiple bitwidths for optimal accuracy-efficiency balance)
• 🔜 Work in progress - We’re actively working to provide support for popular frameworks such as vLLM, SGLang, and llama.cpp.

6. How to Cite This Work

If you find SINQ useful in your research or applications, please cite our paper:

@misc{muller2025sinq,
      title={SINQ: Sinkhorn-Normalized Quantization for Calibration-Free Low-Precision LLM Weights}, 
      author={Lorenz K. Muller and Philippe Bich and Jiawei Zhuang and Ahmet Celik and Luca Benfenati and Lukas Cavigelli},
      year={2025},
      eprint={2509.22944},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={http://arxiv.org/abs/2509.22944}
}

7. Related Repositories

This project builds upon and extends the excellent work from the following open-source projects:

• Qwen3-Quantization - Base implementation and evaluation scripts for Qwen3 quantization.
• HQQ - High-quality calibration-free quantization baseline.

📜 You can find their original licenses in the corresponding LICENSE files in these repositories.