onnx_clip 1.2

Replicating CLIP without PyTorch dependencies.

onnx_clip

About

The purpose of this repository is to replicate the functionality of CLIP without needing the various PyTorch dependencies. We do this by utilising a .onnx format of the model, a pure NumPy version of the tokenizer, and an accurate approximation of the preprocess function. Due to this final approximation, the output logits do not perfectly match those of CLIP but are close enough for our purposes.

git lfs

This repository uses Git LFS for the clip_model.onnx file. Make sure to do git lfs install before cloning.

In case you use the onnx_clip project not as a repo, but as a package, the model will be downloaded from the public S3 bucket.

Installation

To install, run the following in the root of the repository:

pip install .

Usage

All you need to do is call the OnnxClip model class. An example can be seen below.

from onnx_clip import OnnxClip, softmax
from PIL import Image

images = [Image.open("onnx_clip/data/CLIP.png").convert("RGB")]
text = ["a photo of a man", "a photo of a woman"]
onnx_model = OnnxClip()
logits_per_image, logits_per_text = onnx_model.predict(images, text)
probas = softmax(logits_per_image)

Building & developing from source

Note: The following may give timeout errors due to the filesizes. If so, this can be fixed with poetry version 1.1.13 - see this related issue.

Install, run, build and publish with Poetry

Install Poetry

curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -

To setup the project and create a virtual environment run the following command from the project's root directory.

poetry install

To build a source and wheel distribution of the library run the following command from the project's root directory.

poetry build

Instructions to publish the build artifacts for project maintainers

Copy this into your poetry config.toml file (or create a new one).

[repositories]
[repositories.onnx_clip]
url = "https://gitlab.com/api/v4/projects/41150990/packages/pypi"

The file should be located here on MacOs

~/Library/Application Support/pypoetry/config.toml

and here on Linux

~/.config/pypoetry/config.toml

With this setup you can now publish a package like so

poetry publish --repository onnx_clip -u <access_token_name> -p <access_token_key>

WARNING: Do not publish to the public pypi registry, e.g. always use the --repository option. NOTE1: You must generate an access token with scope set to api.
NOTE2: The push will fail if there is already a package with the same version. You can increment the version using poetry

poetry version

or by manually changing the version number in pyproject.toml.

Help

Please let us know how we can support: [e-mail].

LICENSE

See the LICENSE file in this repository.