aidge-quantization 0.9.0

Quantization algorithms to compress aidge networks.

Aidge Quantization Module

You can find in this folder the library that implements the quantization algorithms. For the moment only Post Training Quantization (PTQ) is available. Its implementation does support multiple branch architectures.

[TOC]

Quick Start

Dependencies

Aidge dependencies

• aidge_core
• aidge_backend_cpu
• aidge_backend_cuda, if AIDGE_ENABLE_CUDA is set to ON.
• aidge_onnx

# On Windows, from aidge/
# ----------[ C++ development]----------
setup.ps1 -Modules core, backend_cpu, backend_cuda -Clean -Tests

# ----------[ Python development]----------
setup.ps1 -Modules core, backend_cpu, backend_cuda, onnx -Clean -Tests -Python

# On Unix, from aidge/
# ----------[ C++ development]----------
./setup.sh -m core -m backend_cpu, -m backend_cuda --clean --tests

# ----------[ Python development]----------
./setup.sh -m core -m backend_cpu, -m backend_cuda --clean --tests --python

Using setup.ps1

[!NOTE] Windows only* *Unless you installed powershell on your system

# ----------[ C++ development]----------
# From aidge/
setup.ps1 -Modules quantization -Clean -Tests

# ----------[ Python development]----------
# From aidge/
setup.ps1 -Modules quantization -Clean -Tests -Python

[!TIP] Run Get-Help setup.ps1 --Full to display documentation

Using setup.sh

[!NOTE] Unix only*

# ----------[ C++ development]----------
# From aidge/
./setup.sh -m quantization --clean --tests

# ----------[ Python development]----------
# From aidge/
./setup.sh -m quantization --clean --tests --python

[!TIP] Run setup.sh -h to display documentation

Using pip

[!NOTE] If using virtual environment, make sure to use the same for every installation !

# ----------[ Python development]----------
# only in aidge/aidge/aidge_quantization/
pip install . -v

# If you want to install test, do this instead
pip install .[test] -v

# Launch tests using pytest
pytest

[!TIP] -v is to enable verbose mode !

Development mode install

[!WARNING] Experimental Untested & experimental feature, see https://scikit-build-core.readthedocs.io/en/latest/configuration/index.html#editable-installs.

pip install --no-build-isolation --config-settings=editable.rebuild=true -Cbuild-dir=build -ve.

Using CMake

[!NOTE] Only for C++ development Don't use this method, if you wish to create a pip package Use pip instead.

A CMakePreset.json is available.

# Configure
cmake --preset clang-debug
# Build
cmake --build --preset clang-debug
# Install
cmake --install
# Test
ctest --test-dir build/

Feel free to create your own presets in CMakeUsersPresets.json by inheriting the ones available in CMakePresets.json.

User guide

In order to perform a quantization, you will need an AIDGE model (that can be loaded from an ONNX). Then, you will have to provide a calibration dataset consisting of AIDGE tensors (that can be loaded from some numpy arrays). And finally, you will have to specify the quantization number of bits.

Performing the PTQ on your model will then be a one liner:

aidge_quantization.quantize_network(aidge_model, nb_of_bits, calibration_set)

Technical insights

The PTQ algorithm consists of 3 main steps:

- Normalization of the parameters, so that each node set of weights fits in the [-1:1] range.
- Normalization of the activations, so that each node output value fits in the [-1:1] range.
- Quantization of the scaling nodes previously inserted

To achieve those steps, one must propagate the scaling factors inside the network. One should also balance the different branches when they are merging. A particular care is needed for the biases rescaling at each step.

Doing quantization step by step

It is possible to perform the PTQ step by step, thanks to the exposed functions of the API. In that case, here is the standard pipeline:

- Prepare the network for the PTQ (remove the flatten nodes, fuse the BatchNorms ...)
- Insert the scaling nodes that will allow the model calibration
- Perform the Cross Layer Equalization if possible
- Perform the parameter normalization
- Compute the node output ranges over an input calibration dataset
- Adjust the output ranges using a specified error metric (MSE, KL, ...)
- Perform the activation normalization
- Quantize the normalized network
- Convert the scaling factors to bit-shifting operations if needed

Further work

• add Quantization Aware Training (QAT)