Optimum DeepSparse is an extension of the Hugging Face Transformers library that integrates the DeepSparse inference runtime. DeepSparse offers GPU-class performance on CPUs, making it possible to run Transformers and other deep learning models on commodity hardware with sparsity. Optimum DeepSparse provides a framework for developers to easily integrate DeepSparse into their applications, regardless of the hardware platform.
Project description
optimum-deepsparse
Accelerated inference of 🤗 models on CPUs using the DeepSparse Inference Runtime.
Install
Optimum DeepSparse is a fast-moving project, and you may want to install from source.
pip install git+https://github.com/neuralmagic/optimum-deepsparse.git
Installing in developer mode
If you are working on the optimum-deepsparse
code then you should use an editable install by cloning and installing optimum
and optimum-deepsparse
:
git clone https://github.com/huggingface/optimum
git clone https://github.com/neuralmagic/optimum-deepsparse
pip install -e optimum -e optimum-deepsparse
Now whenever you change the code, you'll be able to run with those changes instantly.
How to use it?
To load a model and run inference with DeepSparse, you can just replace your AutoModelForXxx
class with the corresponding DeepSparseModelForXxx
class.
import requests
from PIL import Image
- from transformers import AutoModelForImageClassification
+ from optimum.deepsparse import DeepSparseModelForImageClassification
from transformers import AutoFeatureExtractor, pipeline
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
model_id = "microsoft/resnet-50"
- model = AutoModelForImageClassification.from_pretrained(model_id)
+ model = DeepSparseModelForImageClassification.from_pretrained(model_id, export=True, input_shapes="[1,3,224,224]")
feature_extractor = AutoFeatureExtractor.from_pretrained(model_id)
cls_pipe = pipeline("image-classification", model=model, feature_extractor=feature_extractor)
outputs = cls_pipe(image)
Supported Task | Model Class |
---|---|
"image-classification" | DeepSparseModelForImageClassification |
"text-classification"/"sentiment-analysis" | DeepSparseModelForSequenceClassification |
"audio-classification" | DeepSparseModelForAudioClassification |
"question-answering" | DeepSparseModelForQuestionAnswering |
"image-segmentation" | DeepSparseModelForSemanticSegmentation |
If you find any issue while using those, please open an issue or a pull request.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file optimum-deepsparse-0.1.0.dev1.tar.gz
.
File metadata
- Download URL: optimum-deepsparse-0.1.0.dev1.tar.gz
- Upload date:
- Size: 26.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.10.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8597663583ba2b54e7ebe922a23ad3e0f746e3fcb0d2786d00f422f2a79e3a0c |
|
MD5 | 7de60a9bfe9fb033ca6c34f5af061b3a |
|
BLAKE2b-256 | b8c3dc8e590088b3e3d1f31f48da014c1cc852ba8bd64860abfdd45f34ff2b5b |
File details
Details for the file optimum_deepsparse-0.1.0.dev1-py3-none-any.whl
.
File metadata
- Download URL: optimum_deepsparse-0.1.0.dev1-py3-none-any.whl
- Upload date:
- Size: 27.4 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.10.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7e5d50db484e518ea122e15f81503a8ff3feec93b6a552a8d13af76de7967a48 |
|
MD5 | b156bf0ad9a0044dfb7b82358984b63f |
|
BLAKE2b-256 | df53313636c48365970d6093a4925ef6289c9a9b577b2bf48ca654b0494f6b83 |