neural-compressor-pt 3.5

Repository of Intel® Neural Compressor

Intel® Neural Compressor

An open-source Python library supporting popular model compression techniques on all mainstream deep learning frameworks (TensorFlow, PyTorch, and ONNX Runtime)

Architecture   |   Workflow   |   LLMs Recipes   |   Results   |   Documentations

---

Intel® Neural Compressor aims to provide popular model compression techniques such as quantization, pruning (sparsity), distillation, and neural architecture search on mainstream frameworks such as TensorFlow, PyTorch, and ONNX Runtime, as well as Intel extensions such as Intel Extension for TensorFlow and Intel Extension for PyTorch. In particular, the tool provides the key features, typical examples, and open collaborations as below:

• Support a wide range of Intel hardware such as Intel Gaudi Al Accelerators, Intel Core Ultra Processors, Intel Xeon Scalable Processors, Intel Xeon CPU Max Series, Intel Data Center GPU Flex Series, and Intel Data Center GPU Max Series with extensive testing; support AMD CPU, ARM CPU, and NVidia GPU through ONNX Runtime with limited testing; support NVidia GPU for some WOQ algorithms like AutoRound and HQQ.

• Validate popular LLMs such as LLama2, Falcon, GPT-J, Bloom, OPT, and more than 10,000 broad models such as Stable Diffusion, BERT-Large, and ResNet50 from popular model hubs such as Hugging Face, Torch Vision, and ONNX Model Zoo, with automatic accuracy-driven quantization strategies

• Collaborate with cloud marketplaces such as Google Cloud Platform, Amazon Web Services, and Azure, software platforms such as Alibaba Cloud, Tencent TACO and Microsoft Olive, and open AI ecosystem such as Hugging Face, PyTorch, ONNX, ONNX Runtime, and Lightning AI

What's New

• [2024/10] Transformers-like API for INT4 inference on Intel CPU and GPU.
• [2024/07] From 3.0 release, framework extension API is recommended to be used for quantization.
• [2024/07] Performance optimizations and usability improvements on client-side.

Installation

Choose the necessary framework dependencies to install based on your deploy environment.

Install Framework

• Install intel_extension_for_pytorch for CPU
• Install intel_extension_for_pytorch for Intel GPU
• Use Docker Image with torch installed for HPU
  Note: There is a version mapping between Intel Neural Compressor and Gaudi Software Stack, please refer to this table and make sure to use a matched combination.
• Install torch for other platform
• Install TensorFlow

Install Neural Compressor from pypi

# Install 2.X API + Framework extension API + PyTorch dependency
pip install neural-compressor[pt]
# Install 2.X API + Framework extension API + TensorFlow dependency
pip install neural-compressor[tf]

Note: Further installation methods can be found under Installation Guide. check out our FAQ for more details.

Getting Started

After successfully installing these packages, try your first quantization program. Following example code demonstrates FP8 Quantization, it is supported by Intel Gaudi2 AI Accelerator.
To try on Intel Gaudi2, docker image with Gaudi Software Stack is recommended, please refer to following script for environment setup. More details can be found in Gaudi Guide.

Run a container with an interactive shell, more info

docker run -it --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --net=host --ipc=host vault.habana.ai/gaudi-docker/1.22.0/ubuntu24.04/habanalabs/pytorch-installer-2.7.1:latest

Note: Since Habana software >= 1.21.0, PT_HPU_LAZY_MODE=0 is the default setting. However, most low-precision functions (such as convert_from_uint4) do not support this setting. Therefore, we recommend setting PT_HPU_LAZY_MODE=1 to maintain compatibility.

Run the example,

from neural_compressor.torch.quantization import (
    FP8Config,
    prepare,
    convert,
)

import torch
import torchvision.models as models

model = models.resnet18()
qconfig = FP8Config(fp8_config="E4M3")
model = prepare(model, qconfig)

# Customer defined calibration. Below is a dummy calibration
model(torch.randn(1, 3, 224, 224).to("hpu"))

model = convert(model)

output = model(torch.randn(1, 3, 224, 224).to("hpu")).to("cpu")
print(output.shape)

More FP8 quantization doc.

Following example code demonstrates weight-only large language model loading on Intel Gaudi2 AI Accelerator.

from neural_compressor.torch.quantization import load

model_name = "TheBloke/Llama-2-7B-GPTQ"
model = load(
    model_name_or_path=model_name,
    format="huggingface",
    device="hpu",
    torch_dtype=torch.bfloat16,
)

Note: Intel Neural Compressor will convert the model format from auto-gptq to hpu format on the first load and save hpu_model.safetensors to the local cache directory for the next load. So it may take a while to load for the first time.

Documentation

Overview
Architecture		Workflow		APIs		LLMs Recipes	Examples
PyTorch Extension APIs
Overview		Dynamic Quantization		Static Quantization		Smooth Quantization
Weight-Only Quantization		FP8 Quantization		MX Quantization		Mixed Precision
Tensorflow Extension APIs
Overview			Static Quantization			Smooth Quantization
Transformers-like APIs
Overview
Other Modules
Auto Tune				Benchmark

Note: From 3.0 release, we recommend to use 3.X API. Compression techniques during training such as QAT, Pruning, Distillation only available in 2.X API currently.

Selected Publications/Events

• arXiv: Faster Inference of LLMs using FP8 on the Intel Gaudi (Mar 2025)
• PyTorch landscape: PyTorch general optimizations (Mar 2025)
• Blog on SqueezeBits: [Intel Gaudi] #4. FP8 Quantization (Jan 2025)
• EMNLP'2024: Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs (Sep 2024)
• arXiv: Efficient Post-training Quantization with FP8 Formats (Sep 2023)
• arXiv: Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs (Sep 2023)

Note: View Full Publication List.

Additional Content

• Release Information
• Contribution Guidelines
• Legal Information
• Security Policy

Communication

• GitHub Issues: mainly for bug reports, new feature requests, question asking, etc.
• Email: welcome to raise any interesting research ideas on model compression techniques by email for collaborations.
• Discord Channel: join the discord channel for more flexible technical discussion.
• WeChat group: scan the QA code to join the technical discussion.