Skip to main content

Optimum DeepSparse is an extension of the Hugging Face Transformers library that integrates the DeepSparse inference runtime. DeepSparse offers GPU-class performance on CPUs, making it possible to run Transformers and other deep learning models on commodity hardware with sparsity. Optimum DeepSparse provides a framework for developers to easily integrate DeepSparse into their applications, regardless of the hardware platform.

Project description

optimum-deepsparse

Accelerated inference of 🤗 models on CPUs using the DeepSparse Inference Runtime.

DeepSparse Modeling / Python - Test DeepSparse Modeling Nightly

Install

Optimum DeepSparse is a fast-moving project, and you may want to install from source.

pip install git+https://github.com/neuralmagic/optimum-deepsparse.git

Installing in developer mode

If you are working on the optimum-deepsparse code then you should use an editable install by cloning and installing optimum and optimum-deepsparse:

git clone https://github.com/huggingface/optimum
git clone https://github.com/neuralmagic/optimum-deepsparse
pip install -e optimum -e optimum-deepsparse

Now whenever you change the code, you'll be able to run with those changes instantly.

How to use it?

To load a model and run inference with DeepSparse, you can just replace your AutoModelForXxx class with the corresponding DeepSparseModelForXxx class.

import requests
from PIL import Image

- from transformers import AutoModelForImageClassification
+ from optimum.deepsparse import DeepSparseModelForImageClassification
from transformers import AutoFeatureExtractor, pipeline

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

model_id = "microsoft/resnet-50"
- model = AutoModelForImageClassification.from_pretrained(model_id)
+ model = DeepSparseModelForImageClassification.from_pretrained(model_id, export=True, input_shapes="[1,3,224,224]")
feature_extractor = AutoFeatureExtractor.from_pretrained(model_id)
cls_pipe = pipeline("image-classification", model=model, feature_extractor=feature_extractor)
outputs = cls_pipe(image)
Supported Task Model Class
"image-classification" DeepSparseModelForImageClassification
"text-classification"/"sentiment-analysis" DeepSparseModelForSequenceClassification
"audio-classification" DeepSparseModelForAudioClassification
"question-answering" DeepSparseModelForQuestionAnswering
"image-segmentation" DeepSparseModelForSemanticSegmentation

If you find any issue while using those, please open an issue or a pull request.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

optimum-deepsparse-0.1.0.dev1.tar.gz (26.9 kB view details)

Uploaded Source

Built Distribution

optimum_deepsparse-0.1.0.dev1-py3-none-any.whl (27.4 kB view details)

Uploaded Python 3

File details

Details for the file optimum-deepsparse-0.1.0.dev1.tar.gz.

File metadata

File hashes

Hashes for optimum-deepsparse-0.1.0.dev1.tar.gz
Algorithm Hash digest
SHA256 8597663583ba2b54e7ebe922a23ad3e0f746e3fcb0d2786d00f422f2a79e3a0c
MD5 7de60a9bfe9fb033ca6c34f5af061b3a
BLAKE2b-256 b8c3dc8e590088b3e3d1f31f48da014c1cc852ba8bd64860abfdd45f34ff2b5b

See more details on using hashes here.

File details

Details for the file optimum_deepsparse-0.1.0.dev1-py3-none-any.whl.

File metadata

File hashes

Hashes for optimum_deepsparse-0.1.0.dev1-py3-none-any.whl
Algorithm Hash digest
SHA256 7e5d50db484e518ea122e15f81503a8ff3feec93b6a552a8d13af76de7967a48
MD5 b156bf0ad9a0044dfb7b82358984b63f
BLAKE2b-256 df53313636c48365970d6093a4925ef6289c9a9b577b2bf48ca654b0494f6b83

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page