Skip to main content

A Model Compression Toolkit for neural networks

Project description

Model Compression Toolkit (MCT)

Model Compression Toolkit (MCT) is an open-source project for neural network model optimization under efficient, constrained hardware.

This project provides researchers, developers, and engineers tools for optimizing and deploying state-of-the-art neural networks on efficient hardware.

Specifically, this project aims to apply quantization to compress neural networks.

MCT is developed by researchers and engineers working at Sony Semiconductor Israel.

Table of Contents

Getting Started

This section provides an installation and a quick starting guide.

Installation

To install the latest stable release of MCT, run the following command:

pip install model-compression-toolkit

For installing the nightly version or installing from source, refer to the installation guide.

Quick start & tutorials

For an example of how to use MCT with TensorFlow or PyTorch on various models and tasks, check out the quick-start page and the results CSV.

In addition, a set of notebooks are provided for an easy start. For example:

Supported Versions

Currently, MCT is being tested on various Python, Pytorch and TensorFlow versions:

PyTorch 1.12 PyTorch 1.13 PyTorch 2.0
Python 3.8 Run Tests Run Tests Run Tests
Python 3.9 Run Tests Run Tests Run Tests
Python 3.10 Run Tests Run Tests Run Tests
TensorFlow 2.11 TensorFlow 2.12 TensorFlow 2.13
Python 3.8 Run Tests Run Tests Run Tests
Python 3.9 Run Tests Run Tests Run Tests
Python 3.10 Run Tests Run Tests Run Tests

Supported Features

MCT offers a range of powerful features to optimize neural network models for efficient deployment. These supported features include:

Data Generation

MCT provides tools for generating synthetic images based on the statistics stored in a model's batch normalization layers. These generated images are valuable for various compression tasks where image data is required, such as quantization and pruning. You can customize data generation configurations to suit your specific needs. Go to the Data Generation page.

Quantization

MCT supports different quantization methods:

Quantization Method Complexity Computational Cost
PTQ Low Low (order of minutes)
GPTQ (parameters fine-tuning using gradients) Mild Mild (order of 2-3 hours)
QAT High High (order of 12-36 hours)

In addition, MCT supports different quantization schemes for quantizing weights and activations:

  • Power-Of-Two (hardware-friendly quantization [1])
  • Symmetric
  • Uniform

Main features:

  • Graph optimizations: Transforming the model to an equivalent (yet, more efficient) model (for example, batch-normalization layer folding to its preceding linear layer).
  • Quantization parameter search: Different methods can be used to minimize the expected added quantization-noise during thresholds search (by default, we use Mean-Square-Error, but other metrics can be used such as No-Clipping, Mean-Average-Error, and more).
  • Advanced quantization algorithms: To prevent a performance degradation some algorithms are applied such as:
    • Shift negative correction: Symmetric activation quantization can hurt the model's performance when some layers output both negative and positive activations, but their range is asymmetric. For more details please visit [1].
    • Outliers filtering: Computing z-score for activation statistics to detect and remove outliers.
  • Clustering: Using non-uniform quantization grid to quantize the weights and activations to match their distributions.*
  • Mixed-precision search: Assigning quantization bit-width per layer (for weights/activations), based on the layer's sensitivity to different bit-widths.
  • Visualization: You can use TensorBoard to observe useful information for troubleshooting the quantized model's performance (for example, the model in different phases of the quantization, collected statistics, similarity between layers of the float and quantized model and bit-width configuration for mixed-precision quantization). For more details, please read the visualization documentation.
  • Target Platform Capabilities: The Target Platform Capabilities (TPC) describes the target platform (an edge device with dedicated hardware). For more details, please read the TPC README.

Experimental features

Some features are experimental and subject to future changes.

For more details, we highly recommend visiting our project website where experimental features are mentioned as experimental.

Results

Keras

Graph of MobileNetV2 accuracy on ImageNet vs average bit-width of weights, using single-precision quantization, mixed-precision quantization, and mixed-precision quantization with GPTQ.

For more results, please see [1]

Pytorch

We quantized classification networks from the torchvision library. In the following table we present the ImageNet validation results for these models:

Network Name Float Accuracy 8Bit Accuracy Data-Free 8Bit Accuracy
MobileNet V2 [3] 71.886 71.444 71.29
ResNet-18 [3] 69.86 69.63 69.53
SqueezeNet 1.1 [3] 58.128 57.678

For more results, please refer to quick start.

Contributions

MCT aims at keeping a more up-to-date fork and welcomes contributions from anyone.

*You will find more information about contributions in the Contribution guide.

License

Apache License 2.0.

References

[1] Habi, H.V., Peretz, R., Cohen, E., Dikstein, L., Dror, O., Diamant, I., Jennings, R.H. and Netzer, A., 2021. HPTQ: Hardware-Friendly Post Training Quantization. arXiv preprint.

[2] MobilNet from Keras applications.

[3] TORCHVISION.MODELS

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mct-nightly-1.9.0.20230926.post355.tar.gz (388.8 kB view details)

Uploaded Source

Built Distribution

File details

Details for the file mct-nightly-1.9.0.20230926.post355.tar.gz.

File metadata

File hashes

Hashes for mct-nightly-1.9.0.20230926.post355.tar.gz
Algorithm Hash digest
SHA256 c533a84766650750afb1c7f94b97f486a177e0cb763cc21704f79b6da8126ecb
MD5 ce270e72b05459d39b198ff7d2a88a64
BLAKE2b-256 5ad3d40e1be1ad701d7bedb3650b025adeed63dc22d958c38fdc47c2758582fd

See more details on using hashes here.

File details

Details for the file mct_nightly-1.9.0.20230926.post355-py3-none-any.whl.

File metadata

File hashes

Hashes for mct_nightly-1.9.0.20230926.post355-py3-none-any.whl
Algorithm Hash digest
SHA256 17aa9c5d4cee12a0a538bf3924c7e6741e7de026009c6db4078221a1f265ba93
MD5 6e28a5839c7d19fd1fb1e4a33fbf7545
BLAKE2b-256 4246693a52951c4fa8c870e22a96b3e6e2187567b8050f8a3b24023753c47a4a

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page