mct-nightly 2.2.0.20241116.529

A Model Compression Toolkit for neural networks

---

Getting Started • Tutorials • High level features and techniques • Resources • Community • License

---

Getting Started

Quick Installation

Pip install the model compression toolkit package in a Python>=3.9 environment with PyTorch>=2.1 or Tensorflow>=2.12.

pip install model-compression-toolkit

For installing the nightly version or installing from source, refer to the installation guide.

Important note: In order to use MCT, you’ll need to provide a floating point .pt or .keras model as an input.

Tutorials and Examples

Our tutorials section will walk you through the basics of the MCT tool, covering various compression techniques for both Keras and PyTorch models. Access interactive notebooks for hands-on learning with popular models/tasks or move on to Resources section.

Supported Quantization Methods

MCT supports various quantization methods as appears below.

Quantization Method	Complexity	Computational Cost	Tutorial
PTQ (Post Training Quantization)	Low	Low (~1-10 CPU minutes)
GPTQ (parameters fine-tuning using gradients)	Moderate	Moderate (~1-3 GPU hours)
QAT (Quantization Aware Training)	High	High (~12-36 GPU hours)

For each flow, Quantization core utilizes various algorithms and hyper-parameters for optimal hardware-aware quantization results. For further details, please see Supported features and algorithms.

Required input:

• Floating point model - 32bit model in either .pt or .keras format
• Representative dataset - can be either provided by the user, or generated utilizing the Data Generation capability

Resources

• User Guide contains detailed information about MCT and guides you from installation through optimizing models for your edge AI applications.

• MCT's API Docs is seperated per quantization methods:

  • Post-training quantization | PTQ API docs
  • Gradient-based post-training quantization | GPTQ API docs
  • Quantization-aware training | QAT API docs

• Debug – modify optimization process or generate explainable report

• Release notes

Supported Versions

Currently, MCT is being tested on various Python, Pytorch and TensorFlow versions:

Supported Versions Table

	PyTorch 2.2	PyTorch 2.3	PyTorch 2.4	PyTorch 2.5
Python 3.9
Python 3.10
Python 3.11
Python 3.12

	TensorFlow 2.12	TensorFlow 2.13	TensorFlow 2.14	TensorFlow 2.15
Python 3.9
Python 3.10
Python 3.11

Supported Features

MCT offers a range of powerful features to optimize neural network models for efficient deployment. These supported features include:

Data Generation *

MCT provides tools for generating synthetic images based on the statistics stored in a model's batch normalization layers. These generated images are valuable for various compression tasks where image data is required, such as quantization and pruning. You can customize data generation configurations to suit your specific needs. Go to the Data Generation page.

Quantization

MCT supports different quantization methods:

• Post-training quantization (PTQ): Keras API, PyTorch API
• Gradient-based post-training quantization (GPTQ): Keras API, PyTorch API
• Quantization-aware training (QAT) *

Quantization Method	Complexity	Computational Cost
PTQ	Low	Low (~CPU minutes)
GPTQ (parameters fine-tuning using gradients)	Moderate	Moderate (~1-3 GPU hours)
QAT	High	High (~12-36 GPU hours)

In addition, MCT supports different quantization schemes for quantizing weights and activations:

• Power-Of-Two (hardware-friendly quantization [1])
• Symmetric
• Uniform

Main features:

• Graph optimizations: Transforming the model to an equivalent (yet, more efficient) model (for example, batch-normalization layer folding to its preceding linear layer).
• Quantization parameter search: Different methods can be used to minimize the expected added quantization-noise during thresholds search (by default, we use Mean-Square-Error, but other metrics can be used such as No-Clipping, Mean-Average-Error, and more).
• Advanced quantization algorithms: To prevent a performance degradation some algorithms are applied such as:
  • Shift negative correction: Symmetric activation quantization can hurt the model's performance when some layers output both negative and positive activations, but their range is asymmetric. For more details please visit [1].
  • Outliers filtering: Computing z-score for activation statistics to detect and remove outliers.
• Clustering: Using non-uniform quantization grid to quantize the weights and activations to match their distributions.*
• Mixed-precision search: Assigning quantization bit-width per layer (for weights/activations), based on the layer's sensitivity to different bit-widths.
• Visualization: You can use TensorBoard to observe useful information for troubleshooting the quantized model's performance (for example, the model in different phases of the quantization, collected statistics, similarity between layers of the float and quantized model and bit-width configuration for mixed-precision quantization). For more details, please read the visualization documentation.
• Target Platform Capabilities: The Target Platform Capabilities (TPC) describes the target platform (an edge device with dedicated hardware). For more details, please read the TPC README.

Enhanced Post-Training Quantization (EPTQ)

As part of the GPTQ we provide an advanced optimization algorithm called EPTQ.

The specifications of the algorithm are detailed in the paper: "EPTQ: Enhanced Post-Training Quantization via Label-Free Hessian" [4].

More details on the how to use EPTQ via MCT can be found in the EPTQ guidelines.

Structured Pruning *

MCT introduces a structured and hardware-aware model pruning. This pruning technique is designed to compress models for specific hardware architectures, taking into account the target platform's Single Instruction, Multiple Data (SIMD) capabilities. By pruning groups of channels (SIMD groups), our approach not only reduces model size and complexity, but ensures that better utilization of channels is in line with the SIMD architecture for a target Resource Utilization of weights memory footprint. Keras API Pytorch API

Experimental features

Some features are experimental and subject to future changes.

For more details, we highly recommend visiting our project website where experimental features are mentioned as experimental.

Results

Pytorch

We quantized classification networks from the torchvision library. In the following table we present the ImageNet validation results for these models:

Network Name	Float Accuracy	8Bit Accuracy	Data-Free 8Bit Accuracy
MobileNet V2 [3]	71.886	71.444	71.29
ResNet-18 [3]	69.86	69.63	69.53
SqueezeNet 1.1 [3]	58.128	57.678

Keras

MCT can quantize an existing 32-bit floating-point model to an 8-bit fixed-point (or less) model without compromising accuracy. Below is a graph of MobileNetV2 accuracy on ImageNet vs average bit-width of weights (X-axis), using single-precision quantization, mixed-precision quantization, and mixed-precision quantization with GPTQ.

For more results, please see [1]

Pruning Results

Results for applying pruning to reduce the parameters of the following models by 50%:

Model	Dense Model Accuracy	Pruned Model Accuracy
ResNet50 [2]	75.1	72.4
DenseNet121 [3]	74.44	71.71

Troubleshooting and Community

If you encountered large accuracy degradation with MCT, check out the Quantization Troubleshooting for common pitfalls and some tools to improve quantized model's accuracy.

Check out the FAQ for common issues.

You are welcome to ask questions and get support on our issues section and manage community discussions under discussions section.

Contributions

MCT aims at keeping a more up-to-date fork and welcomes contributions from anyone.

*Checkout our Contribution guide for more details.

License

MCT is licensed under Apache License Version 2.0. By contributing to the project, you agree to the license and copyright terms therein and release your contribution under these terms.

References

[1] Habi, H.V., Peretz, R., Cohen, E., Dikstein, L., Dror, O., Diamant, I., Jennings, R.H. and Netzer, A., 2021. HPTQ: Hardware-Friendly Post Training Quantization. arXiv preprint.

[2] Keras Applications

[3] TORCHVISION.MODELS

[4] Gordon, O., Habi, H. V., & Netzer, A., 2023. EPTQ: Enhanced Post-Training Quantization via Label-Free Hessian. arXiv preprint