Optimum Intel
🤗 Optimum Intel is the interface between the 🤗 Transformers, Diffusers, Sentence Transformers and timm libraries and the different tools and libraries provided by OpenVINO to accelerate end-to-end pipelines on Intel architectures.
OpenVINO is an open-source toolkit that enables high performance inference capabilities for Intel CPUs, GPUs, and special DL inference accelerators (see the full list of supported devices). It is supplied with a set of tools to optimize your models with compression techniques such as quantization, pruning and knowledge distillation. Optimum Intel provides a simple interface to optimize your model, convert it to the OpenVINO Intermediate Representation (IR) format and run inference using OpenVINO Runtime.
Installation
To install the latest release of 🤗 Optimum Intel with the corresponding required dependencies, you can use pip as follows:
pip install -U optimum-intel
Optimum Intel is a fast-moving project with regular additions of new model support, so you may want to install from source with the following command:
pip install optimum-intel@git+https://github.com/huggingface/optimum-intel.git
Export:
To export your model to OpenVINO IR format, use the optimum-cli tool. Below is an example of exporting TinyLlama/TinyLlama_v1.1 model:
optimum-cli export openvino --model TinyLlama/TinyLlama_v1.1 ov_TinyLlama_v1_1
To export a model hosted on the Hub you can use our space. After conversion, a repository will be pushed under your namespace, this repository can be either public or private.
Additional information on exporting models is available in the documentation.
Inference:
To load an exported model and run inference using Optimum Intel, use the corresponding OVModelForXxx class instead of AutoModelForXxx:
from optimum.intel import OVModelForCausalLM
from transformers import AutoTokenizer, pipeline
model_id = "ov_TinyLlama_v1_1"
model = OVModelForCausalLM.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
results = pipe("Hey, how are you doing today?", max_new_tokens=100)
For more details on Optimum Intel inference, refer to the documentation.
Note: Alternatively, an exported model can also be inferred using OpenVINO GenAI framework, that provides optimized execution methods for highly performant Generative AI.
Quantization:
Post-training static quantization can also be applied. Here is an example on how to apply static quantization on a Whisper model using the LibriSpeech dataset for the calibration step.
from optimum.intel import OVModelForSpeechSeq2Seq, OVQuantizationConfig
model_id = "openai/whisper-tiny"
q_config = OVQuantizationConfig(dtype="int8", dataset="librispeech", num_samples=50)
q_model = OVModelForSpeechSeq2Seq.from_pretrained(model_id, quantization_config=q_config)
# The directory where the quantized model will be saved
save_dir = "nncf_results"
q_model.save_pretrained(save_dir)
You can find more information in the documentation.
Running the examples
Check out the notebooks directory to see how 🤗 Optimum Intel can be used to optimize models and accelerate inference.
Do not forget to install requirements for every example:
cd <example-folder>
pip install -r requirements.txt
3 maintainers
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file optimum_intel-2.1.0.tar.gz.
File metadata
- Download URL: optimum_intel-2.1.0.tar.gz
- Upload date:
- Size: 385.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.2.0 CPython/3.11.13
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
8d4c0c80af19c9048bc2e58f36f510fa7a66c051089fcf253ffc0ba2b6f1386f
|
|
| MD5 |
f1b8c98e8efd74b7a7f38618b0e50d08
|
|
| BLAKE2b-256 |
9ddf498acf3432bdf4b7e4bf2b5d66a9c6d87275eded2b6d0d2258c7c8490f31
|
File details
Details for the file optimum_intel-2.1.0-py3-none-any.whl.
File metadata
- Download URL: optimum_intel-2.1.0-py3-none-any.whl
- Upload date:
- Size: 415.0 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.2.0 CPython/3.11.13
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
e028c39b66552c7c977968e0df3c160b695fe5cc82d7143f0939576ecb4bb483
|
|
| MD5 |
27e0e5ee0655c50fe3d6e8f1a6756eeb
|
|
| BLAKE2b-256 |
66a4d969866a78e854dcf231f2007f6bca9947f684b83bf72ed8ff5825a21c65
|
