🤗 Optimum
Optimum is an extension of Transformers 🤖 Diffusers 🧨 TIMM 🖼️ and Sentence-Transformers 🤗, providing a set of optimization tools and enabling maximum efficiency to train and run models on targeted hardware, while keeping things easy to use.
Installation
Optimum can be installed using pip as follows:
python -m pip install optimum
If you'd like to use the accelerator-specific features of Optimum, you can check the documentation and install the required dependencies according to the table below:
| Accelerator | Installation |
|---|---|
| ONNX | pip install --upgrade --upgrade-strategy eager optimum[onnx] |
| ONNX Runtime | pip install --upgrade --upgrade-strategy eager optimum[onnxruntime] |
| ONNX Runtime GPU | pip install --upgrade --upgrade-strategy eager optimum[onnxruntime-gpu] |
| OpenVINO | pip install --upgrade --upgrade-strategy eager optimum[openvino] |
| NVIDIA TensorRT-LLM | docker run -it --gpus all --ipc host huggingface/optimum-nvidia |
| AMD Instinct GPUs and Ryzen AI NPU | pip install --upgrade --upgrade-strategy eager optimum[amd] |
| AWS Trainum & Inferentia | pip install --upgrade --upgrade-strategy eager optimum[neuronx] |
| Intel Gaudi Accelerators (HPU) | pip install --upgrade --upgrade-strategy eager optimum[habana] |
| FuriosaAI | pip install --upgrade --upgrade-strategy eager optimum[furiosa] |
The --upgrade --upgrade-strategy eager option is needed to ensure the different packages are upgraded to the latest possible version.
To install from source:
python -m pip install git+https://github.com/huggingface/optimum.git
For the accelerator-specific features, append optimum[accelerator_type] to the above command:
python -m pip install optimum[onnxruntime]@git+https://github.com/huggingface/optimum.git
Accelerated Inference
Optimum provides multiple tools to export and run optimized models on various ecosystems:
- ONNX / ONNX Runtime, one of the most popular open formats for model export, and a high-performance inference engine for deployment.
- OpenVINO, a toolkit for optimizing, quantizing and deploying deep learning models on Intel hardware.
- ExecuTorch, PyTorch’s native solution for on-device inference across mobile and edge devices.
- Intel Gaudi Accelerators enabling optimal performance on first-gen Gaudi, Gaudi2 and Gaudi3.
- AWS Inferentia for accelerated inference on Inf2 and Inf1 instances.
- NVIDIA TensorRT-LLM.
The export and optimizations can be done both programmatically and with a command line.
ONNX + ONNX Runtime
🚨🚨🚨 ONNX integration was moved to optimum-onnx so make sure to follow the installation instructions 🚨🚨🚨
Before you begin, make sure you have all the necessary libraries installed :
pip install --upgrade --upgrade-strategy eager optimum[onnx]
It is possible to export Transformers, Diffusers, Sentence Transformers and Timm models to the ONNX format and perform graph optimization as well as quantization easily.
For more information on the ONNX export, please check the documentation.
Once the model is exported to the ONNX format, we provide Python classes enabling you to run the exported ONNX model in a seamless manner using ONNX Runtime in the backend.
For this make sure you have ONNX Runtime installed, fore more information check out the installation instructions.
More details on how to run ONNX models with ORTModelForXXX classes here.
Intel (OpenVINO + NNCF)
Before you begin, make sure you have all the necessary libraries installed.
pip install --upgrade --upgrade-strategy eager optimum[openvino]
You can find more information on the different integration in our documentation and in the examples of optimum-intel.
ExecuTorch
Before you begin, make sure you have all the necessary libraries installed :
pip install optimum-executorch@git+https://github.com/huggingface/optimum-executorch.git
Users can export Transformers models to ExecuTorch and run inference on edge devices within PyTorch's ecosystem.
For more information about export Transformers to ExecuTorch, please check the doc for Optimum-ExecuTorch.
Quanto
Quanto is a pytorch quantization backend which allows you to quantize a model either using the python API or the optimum-cli.
You can see more details and examples in the Quanto repository.
Accelerated training
Optimum provides wrappers around the original Transformers Trainer to enable training on powerful hardware easily. We support many providers:
- Intel Gaudi Accelerators (HPU) enabling optimal performance on first-gen Gaudi, Gaudi2 and Gaudi3.
- AWS Trainium for accelerated training on Trn1 and Trn1n instances.
- ONNX Runtime (optimized for GPUs).
Intel Gaudi Accelerators
Before you begin, make sure you have all the necessary libraries installed :
pip install --upgrade --upgrade-strategy eager optimum[habana]
You can find examples in the documentation and in the examples.
AWS Trainium
Before you begin, make sure you have all the necessary libraries installed :
pip install --upgrade --upgrade-strategy eager optimum[neuronx]
You can find examples in the documentation and in the tutorials.
7 maintainers
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file optimum-2.3.0.tar.gz.
File metadata
- Download URL: optimum-2.3.0.tar.gz
- Upload date:
- Size: 124.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.11.13
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
aa96ad535a5cec68d12c6372574125452284632fe13699633a61e8bbfb09c4df
|
|
| MD5 |
660a1185668fff95c0c785b337ed3900
|
|
| BLAKE2b-256 |
d976e4ac0c4b398ed3fe2d41e0058002d276896b9a15a54be16889d8e0d3ee92
|
File details
Details for the file optimum-2.3.0-py3-none-any.whl.
File metadata
- Download URL: optimum-2.3.0-py3-none-any.whl
- Upload date:
- Size: 160.9 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.11.13
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
3e9b217b4ab21fd4cf894a987002ee7d3626114e009592babf084c2f1a0f3b5f
|
|
| MD5 |
514120f0fd5974a97ac88cf6b6c9c44c
|
|
| BLAKE2b-256 |
8cf9a16609b4e4fc592653d9f2a0413689da686a94d0040f3a2fabfff5b5894c
|
