Optimum Intel
🤗 Optimum Intel is the interface between the 🤗 Transformers, Diffusers, Sentence Transformers and timm libraries and the different tools and libraries provided by OpenVINO to accelerate end-to-end pipelines on Intel architectures.
OpenVINO is an open-source toolkit that enables high performance inference capabilities for Intel CPUs, GPUs, and special DL inference accelerators (see the full list of supported devices). It is supplied with a set of tools to optimize your models with compression techniques such as quantization, pruning and knowledge distillation. Optimum Intel provides a simple interface to optimize your model, convert it to the OpenVINO Intermediate Representation (IR) format and run inference using OpenVINO Runtime.
Installation
To install the latest release of 🤗 Optimum Intel with the corresponding required dependencies, you can use pip as follows:
pip install -U optimum-intel
Optimum Intel is a fast-moving project with regular additions of new model support, so you may want to install from source with the following command:
pip install optimum-intel@git+https://github.com/huggingface/optimum-intel.git
Export:
To export your model to OpenVINO IR format, use the optimum-cli tool. Below is an example of exporting TinyLlama/TinyLlama_v1.1 model:
optimum-cli export openvino --model TinyLlama/TinyLlama_v1.1 ov_TinyLlama_v1_1
To export a model hosted on the Hub you can use our space. After conversion, a repository will be pushed under your namespace, this repository can be either public or private.
Additional information on exporting models is available in the documentation.
Inference:
To load an exported model and run inference using Optimum Intel, use the corresponding OVModelForXxx class instead of AutoModelForXxx:
from optimum.intel import OVModelForCausalLM
from transformers import AutoTokenizer, pipeline
model_id = "ov_TinyLlama_v1_1"
model = OVModelForCausalLM.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
results = pipe("Hey, how are you doing today?", max_new_tokens=100)
For more details on Optimum Intel inference, refer to the documentation.
Note: Alternatively, an exported model can also be inferred using OpenVINO GenAI framework, that provides optimized execution methods for highly performant Generative AI.
Quantization:
Post-training static quantization can also be applied. Here is an example on how to apply static quantization on a Whisper model using the LibriSpeech dataset for the calibration step.
from optimum.intel import OVModelForSpeechSeq2Seq, OVQuantizationConfig
model_id = "openai/whisper-tiny"
q_config = OVQuantizationConfig(dtype="int8", dataset="librispeech", num_samples=50)
q_model = OVModelForSpeechSeq2Seq.from_pretrained(model_id, quantization_config=q_config)
# The directory where the quantized model will be saved
save_dir = "nncf_results"
q_model.save_pretrained(save_dir)
You can find more information in the documentation.
Running the examples
Check out the notebooks directory to see how 🤗 Optimum Intel can be used to optimize models and accelerate inference.
Do not forget to install requirements for every example:
cd <example-folder>
pip install -r requirements.txt
3 maintainers
Release files for optimum-intel 2.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| optimum_intel-2.2.0.tar.gz | 448.4 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| optimum_intel-2.2.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size:928.4 kB
Release files / optimum_intel-2.2.0.tar.gz
| Download URL | optimum_intel-2.2.0.tar.gz |
|---|---|
| Size | 448.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
90fb4cc948315fd1ff19f4d2b4d86e04c190e3f7d418fe1ec7546ec7dc1ff6b3
|
|
BLAKE2b-256 checksum How to use checksums |
ede684d60bd6707d193e2ad2025cbf65d922e334a89c22de8bd18ffcb628a71e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.13
|
Release files / optimum_intel-2.2.0-py3-none-any.whl
| Download URL | optimum_intel-2.2.0-py3-none-any.whl |
|---|---|
| Size | 479.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
ba3c1c5995912fa10717dd76c07a60eff636fec0293e0e64b331bb24bc4d6d65
|
|
BLAKE2b-256 checksum How to use checksums |
7602ab4c4d7efb1799ac1f86f790f37e541c4dac3d6e39c6c248fbce560e0c4b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.13
|
