Skip to main content

CLIP with ONNX Runtime and without PyTorch dependencies.

Project description

onnx_clip

An ONNX-based implementation of CLIP that doesn't depend on torch or torchvision. It also has a friendlier API than the original implementation.

This works by

  • running the text and vision encoders (the ViT-B/32 variant) in ONNX Runtime
  • using a pure NumPy version of the tokenizer
  • using a pure NumPy+PIL version of the preprocess function. The PIL dependency could also be removed with minimal code changes - see preprocessor.py.

Installation

To install, run the following in the root of the repository:

pip install .

Usage

All you need to do is call the OnnxClip model class. An example:

from onnx_clip import OnnxClip, softmax, get_similarity_scores
from PIL import Image

images = [Image.open("onnx_clip/data/franz-kafka.jpg").convert("RGB")]
texts = ["a photo of a man", "a photo of a woman"]

# Your images/texts will get split into batches of this size before being
# passed to CLIP, to limit memory usage
onnx_model = OnnxClip(batch_size=16)

# Unlike the original CLIP, there is no need to run tokenization/preprocessing
# separately - simply run get_image_embeddings directly on PIL images/NumPy
# arrays, and run get_text_embeddings directly on strings.
image_embeddings = onnx_model.get_image_embeddings(images)
text_embeddings = onnx_model.get_text_embeddings(texts)

# To use the embeddings for zero-shot classification, you can use these two
# functions. Here we run on a single image, but any number is supported.
logits = get_similarity_scores(image_embeddings, text_embeddings)
probabilities = softmax(logits)

print("Logits:", logits)

for text, p in zip(texts, probabilities[0]):
    print(f"Probability that the image is '{text}': {p:.3f}")

Building & developing from source

Note: The following may give timeout errors due to the filesizes. If so, this can be fixed with poetry version 1.1.13 - see this related issue.

Install, run, build and publish with Poetry

Install Poetry

curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -

To setup the project and create a virtual environment run the following command from the project's root directory.

poetry install

To build a source and wheel distribution of the library run the following command from the project's root directory.

poetry build

Publishing a new version to PyPI (for project maintainers)

First, remove/move the downloaded LFS files, so that they're not packaged with the code. Otherwise, this creates a huge .whl file that PyPI refuses and it causes confusing errors.

Then, follow this guide. tl;dr: go to the PyPI account page, generate an API token and put it into the $PYPI_PASSWORD environment variable. Then run

poetry publish --build --username "__token__" --password $PYPI_PASSWORD

Help

Please let us know how we can support you: earlyaccess@lakera.ai.

LICENSE

See the LICENSE file in this repository.

The franz-kafka.jpg is taken from here.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

onnx_clip-2.1.1.tar.gz (1.9 MB view hashes)

Uploaded Source

Built Distribution

onnx_clip-2.1.1-py3-none-any.whl (1.8 MB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page