lpot 1.3.1

Repository of Intel® Low Precision Optimization Tool

Intel® Low Precision Optimization Tool

Intel® Low Precision Optimization Tool (Intel® LPOT) is an open-source Python* library that delivers a unified low-precision inference interface across multiple Intel-optimized DL frameworks on both CPUs and GPUs. It supports automatic accuracy-driven tuning strategies, along with additional objectives such as optimizing for performance, model size, and memory footprint. It also provides easy extension capability for new backends, tuning strategies, metrics, and objectives.

Note

GPU support is under development.

Infrastructure	Workflow

Supported Intel optimized DL frameworks are:

• TensorFlow*, including 1.15.0 UP2, 1.15.0 UP1, 2.1.0, 2.2.0, 2.3.0, 2.4.0
• PyTorch*, including 1.5.0+cpu, 1.6.0+cpu
• Apache* MXNet, including 1.6.0, 1.7.0
• ONNX* Runtime, including 1.6.0

Installation

The Intel® LPOT library is released as part of Intel® oneAPI AI Analytics Toolkit (AI Kit). The AI Kit provides a consolidated package of Intel's latest deep learning and machine optimizations all in one place for ease of development. Along with LPOT, the AI Kit includes Intel-optimized versions of deep learning frameworks (such as TensorFlow and PyTorch) and high performing Python libraries to streamline end-to-end data science and AI workflows on Intel architectures.

Install for Linux

You can install just the LPOT library from binary or source, or you can get the Intel optimized framework together with the LPOT library by installing the Intel® oneAPI AI Analytics Toolkit.

Install from binary

# install from pip
pip install lpot

# install from conda
conda install lpot -c conda-forge -c intel 

Install from source

git clone https://github.com/intel/lpot.git
cd lpot
pip install -r requirements.txt
python setup.py install

Install from AI Kit

The AI Kit, which includes the LPOT library, is distributed through many common channels, including from Intel's website, YUM, APT, Anaconda, and more. Select and download the AI Kit distribution package that's best suited for you and follow the Get Started Guide for post-installation instructions.

Download AI Kit	AI Kit Get Started Guide

Install for Windows

Prerequisites

The following prerequisites and requirements must be satisfied in order to install successfully：

• Python version: 3.6 or 3.7 or 3.8

• Download and install anaconda: anaconda

• Create a virtual environment named lpot in anaconda:

  # Here we install python 3.7 for instance. You can also choose python 3.6 & 3.8.
  conda create -n lpot python=3.7
  conda activate lpot
  

Install from binary

# install from pip
pip install lpot

# install from conda
conda install lpot -c conda-forge -c intel 

Install from source

git clone https://github.com/intel/lpot.git
cd lpot
pip install -r requirements.txt
python setup.py install

Getting started

• Introduction explains Intel® Low Precision Optimization Tool's API.
• Transform introduces how to utilize LPOT buildin data processing and how to develop a custom data processing method.
• Dataset introudces how to utilize LPOT buildin dataset and how to develop a custom dataset.
• Metric introduces how to utilize LPOT buildin metrics and how to develop a custom metric.
• Tutorial provides comprehensive instructions on how to utilize Intel® Low Precision Optimization Tool's features with examples.
• Examples are provided to demonstrate the usage of Intel® Low Precision Optimization Tool in different frameworks: TensorFlow, PyTorch, MXNet and ONNX Runtime.
• UX is a web based system to simplify Intel® Low Precision Optimization Tool usage.
• Intel oneAPI AI Analytics Toolkit Get Started Guide explains the AI Kit components, installation and configuration guides, and instructions for building and running sample apps.
• AI and Analytics Samples includes code samples for Intel oneAPI libraries.

Deep Dive

• Quantization is the processes that enable inference and training by performing computations at low precision data type, such as fixed point integers. LPOT supports Post-Training Quantization and Quantization-Aware Training
• Pruning provides a common method for introducing sparsity in weights and activations.
• Benchmarking introduces how to utilize the benchmark interface of LPOT.
• Mixed precision introduces how to enable mixed precision, including BFP16 and int8 and FP32, on Intel platforms during tuning.
• Graph Optimization introduces how to enable graph optimization for fp32 and auto-mixed precision.
• TensorBoard provides tensor histogram and execution graph for tuning debugging purpose.

Advanced Topics

• Adaptor is the interface between LPOT and framework. The method to develop adaptor extension is introduced with ONNX Runtime as example.
• Strategy can automatically optimized low-precision recipes for deep learning models to achieve optimal product objectives like inference performance and memory usage with expected accuracy criteria. The method to develop a new strategy is introduced.

System Requirements

Intel® Low Precision Optimization Tool supports systems based on Intel 64 architecture or compatible processors, specially optimized for the following CPUs:

• Intel Xeon Scalable processor (formerly Skylake, Cascade Lake, and Cooper Lake)
• future Intel Xeon Scalable processor (code name Sapphire Rapids)

Intel® Low Precision Optimization Tool requires installing the pertinent Intel-optimized framework version for TensorFlow, PyTorch, and MXNet.

Validated Hardware/Software Environment

Platform	OS	Python	Framework	Version
Cascade Lake Cooper Lake Skylake	CentOS 7.8 Ubuntu 18.04	3.6 3.7 3.8	TensorFlow	2.4.0
				2.2.0
				1.15.0 UP1
				1.15.0 UP2
				2.3.0
				2.1.0
				1.15.2
			PyTorch	1.5.0+cpu
				1.6.0+cpu
				IPEX
			MXNet	1.7.0
				1.6.0
			ONNX Runtime	1.6.0

Validated Models

Intel® Low Precision Optimization Tool provides numerous examples to show promising accuracy loss with the best performance gain. Below table lists some key models as showcases. Full quantized model list on various frameworks is available in Model List

Framework	version	Model	dataset	Accuracy	Performance speed up
INT8 Tuning Accuracy	FP32 Accuracy Baseline	Acc Ratio[(INT8-FP32)/FP32]	Realtime Latency Ratio[FP32/INT8]
tensorflow	2.4.0	resnet50v1.5	ImageNet	76.70%	76.50%	0.26%	3.23x
tensorflow	2.4.0	Resnet101	ImageNet	77.20%	76.40%	1.05%	2.42x
tensorflow	2.4.0	inception_v1	ImageNet	70.10%	69.70%	0.57%	1.88x
tensorflow	2.4.0	inception_v2	ImageNet	74.10%	74.00%	0.14%	1.96x
tensorflow	2.4.0	inception_v3	ImageNet	77.20%	76.70%	0.65%	2.36x
tensorflow	2.4.0	inception_v4	ImageNet	80.00%	80.30%	-0.37%	2.59x
tensorflow	2.4.0	inception_resnet_v2	ImageNet	80.10%	80.40%	-0.37%	1.97x
tensorflow	2.4.0	Mobilenetv1	ImageNet	71.10%	71.00%	0.14%	2.88x
tensorflow	2.4.0	ssd_resnet50_v1	Coco	37.90%	38.00%	-0.26%	2.97x
tensorflow	2.4.0	mask_rcnn_inception_v2	Coco	28.90%	29.10%	-0.69%	2.66x
tensorflow	2.4.0	vgg16	ImageNet	72.50%	70.90%	2.26%	3.75x
tensorflow	2.4.0	vgg19	ImageNet	72.40%	71.00%	1.97%	3.79x

Framework	version	model	dataset	Accuracy	Performance speed up
INT8 Tuning Accuracy	FP32 Accuracy Baseline	Acc Ratio[(INT8-FP32)/FP32]	Realtime Latency Ratio[FP32/INT8]
pytorch	1.5.0+cpu	resnet50	ImageNet	75.96%	76.13%	-0.23%	2.63x
pytorch	1.5.0+cpu	resnext101_32x8d	ImageNet	79.12%	79.31%	-0.24%	2.61x
pytorch	1.6.0a0+24aac32	bert_base_mrpc	MRPC	88.90%	88.73%	0.19%	1.98x
pytorch	1.6.0a0+24aac32	bert_base_cola	COLA	59.06%	58.84%	0.37%	2.19x
pytorch	1.6.0a0+24aac32	bert_base_sts-b	STS-B	88.40%	89.27%	-0.97%	2.28x
pytorch	1.6.0a0+24aac32	bert_base_sst-2	SST-2	91.51%	91.86%	-0.37%	2.30x
pytorch	1.6.0a0+24aac32	bert_base_rte	RTE	69.31%	69.68%	-0.52%	2.15x
pytorch	1.6.0a0+24aac32	bert_large_mrpc	MRPC	87.45%	88.33%	-0.99%	2.73x
pytorch	1.6.0a0+24aac32	bert_large_squad	SQUAD	92.85%	93.05%	-0.21%	2.01x
pytorch	1.6.0a0+24aac32	bert_large_qnli	QNLI	91.20%	91.82%	-0.68%	2.69x

Known Issues

The MSE tuning strategy does not work with the PyTorch adaptor layer. This strategy requires a comparison between the FP32 and INT8 tensors to decide which op impacts the final quantization accuracy. The PyTorch adaptor layer does not implement this inspect tensor interface. Therefore, do not choose the MSE tuning strategy for PyTorch models.

Incompatible Changes

LPOT v1.2 introduces incompatible changes in user facing APIs. Please refer to incompatible changes to know which incompatible changes are made in v1.2.

LPOT v1.2.1 solves this backward compatible issues introduced in v1.2 by moving new user facing APIs to lpot.experimental package and keep old one as is. Please refer to introduction to know the details of user-facing APIs.

Support

Submit your questions, feature requests, and bug reports to the GitHub issues page. You may also reach out to [e-mail].

Contribution

We welcome community contributions to Intel® Low Precision Optimization Tool. If you have an idea on how to improve the library, refer to the following:

• For changes impacting the public API, submit an RFC pull request.
• Ensure that the changes are consistent with the code contribution guidelines and coding style.
• Ensure that you can run all the examples with your patch.
• Submit a pull request.

For additional details, see contribution guidelines.

This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.

License

Intel® Low Precision Optimization Tool is licensed under Apache License Version 2.0. This software includes components that have separate copyright notices and licensing terms. Your use of the source code for these components is subject to the terms and conditions of the following licenses.

Apache License Version 2.0:

• Intel TensorFlow Quantization Tool

MIT License:

• bayesian-optimization

See the accompanying LICENSE file for full license text and copyright notices.

---

View Legal Information.

Citation

If you use Intel® Low Precision Optimization Tool in your research or you wish to refer to the tuning results published in the Validated Models, use the following BibTeX entry.

@misc{Intel® Low Precision Optimization Tool,
  author =       {Feng Tian, Chuanqi Wang, Guoming Zhang, Penghui Cheng, Pengxin Yuan, Haihao Shen, and Jiong Gong},
  title =        {Intel® Low Precision Optimization Tool},
  howpublished = {\url{https://github.com/intel/lpot}},
  year =         {2020}
}