neural-insights 2.2

Repository of Intel® Neural Compressor

Intel® Neural Compressor

An open-source Python library supporting popular model compression techniques on all mainstream deep learning frameworks (TensorFlow, PyTorch, ONNX Runtime, and MXNet)

Architecture   |   Workflow   |   Results   |   Examples   |   Documentations

---

Intel® Neural Compressor aims to provide popular model compression techniques such as quantization, pruning (sparsity), distillation, and neural architecture search on mainstream frameworks such as TensorFlow, PyTorch, ONNX Runtime, and MXNet, as well as Intel extensions such as Intel Extension for TensorFlow and Intel Extension for PyTorch. In particular, the tool provides the key features, typical examples, and open collaborations as below:

• Support a wide range of Intel hardware such as Intel Xeon Scalable processor, Intel Xeon CPU Max Series, Intel Data Center GPU Flex Series, and Intel Data Center GPU Max Series with extensive testing; support AMD CPU, ARM CPU, and NVidia GPU through ONNX Runtime with limited testing

• Validate more than 10,000 models such as Bloom-176B, OPT-6.7B, Stable Diffusion, GPT-J, BERT-Large, and ResNet50 from popular model hubs such as Hugging Face, Torch Vision, and ONNX Model Zoo, by leveraging zero-code optimization solution Neural Coder and automatic accuracy-driven quantization strategies

• Collaborate with cloud marketplace such as Google Cloud Platform, Amazon Web Services, and Azure, software platforms such as Alibaba Cloud and Tencent TACO, and open AI ecosystem such as Hugging Face, PyTorch, ONNX, and Lightning AI

Installation

Install from pypi

pip install neural-compressor

More installation methods can be found at Installation Guide. Please check out our FAQ for more details.

Getting Started

Quantization with Python API

# Install Intel Neural Compressor and TensorFlow
pip install neural-compressor
pip install tensorflow
# Prepare fp32 model
wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_6/mobilenet_v1_1.0_224_frozen.pb

from neural_compressor.config import PostTrainingQuantConfig
from neural_compressor.data import DataLoader
from neural_compressor.data import Datasets

dataset = Datasets('tensorflow')['dummy'](shape=(1, 224, 224, 3))
dataloader = DataLoader(framework='tensorflow', dataset=dataset)

from neural_compressor.quantization import fit
q_model = fit(
    model="./mobilenet_v1_1.0_224_frozen.pb",
    conf=PostTrainingQuantConfig(),
    calib_dataloader=dataloader,
    eval_dataloader=dataloader)

Documentation

Overview
Architecture		Workflow		Examples		APIs
Python-based APIs
Quantization		Advanced Mixed Precision		Pruning (Sparsity)		Distillation
Orchestration		Benchmarking		Distributed Compression		Model Export
Neural Coder (Zero-code Optimization)
Launcher		JupyterLab Extension		Visual Studio Code Extension		Supported Matrix
Advanced Topics
Adaptor		Strategy		Distillation for Quantization		SmoothQuant
Innovations for Productivity
Neural Insights				Neural Solution

Selected Publications/Events

• Blog on Medium: Intel Optimization at Netflix (May 2023)
• Blog on Medium: Effective Post-training Quantization for Large Language Models with Enhanced SmoothQuant Approach (Apr 2023)
• Blog by Intel: Intel® Xeon® Processors Are Still the Only CPU With MLPerf Results, Raising the Bar By 5x (Apr 2023)
• NeurIPS'2022: Fast Distilbert on CPUs (Oct 2022)
• NeurIPS'2022: QuaLA-MiniLM: a Quantized Length Adaptive MiniLM (Oct 2022)

View our Full Publication List.

Additional Content

• Release Information
• Contribution Guidelines
• Legal Information
• Security Policy

Research Collaborations

Welcome to raise any interesting research ideas on model compression techniques and feel free to reach us ([e-mail]). Look forward to our collaborations on Intel Neural Compressor!