Modern ONNX CLIP

A modern, lightweight, and robust ONNX runtime for CLIP models.

This library allows you to run OpenAI CLIP and OpenCLIP models in production environments without installing PyTorch. It provides a simple CLI to convert models from the massive OpenCLIP model zoo and a pure-Python inference engine powered by onnxruntime, numpy, and pillow.

Supported Python versions: 3.10 through 3.14.

Features

• Zero PyTorch Dependency in Production: Run inference with just numpy and onnxruntime. Drastically reduces Docker image size and memory usage.
• Easy Conversion: Convert any model from OpenCLIP (ViT-B-32, ViT-L-14, SigLIP, etc.) with a single command.
• Modern Tooling: Built with uv, ruff, and strictly typed with pyright.
• Fast: Leverages ONNX Runtime (CPU or CUDA) for high-performance inference.
• Drop-in Replacement: Designed to replace the unmaintained onnx_clip package with better model support.

Installation

For Production (Inference Only)

If you only need to run models, install the base package. This does not install PyTorch.

uv add "modern-onnx-clip[cpu]"
# or for GPU support
uv add "modern-onnx-clip[gpu]"

For Development & Exporting

To convert models, you need the export dependencies (PyTorch, OpenCLIP).

pip install "modern-onnx-clip[export]"

Usage

1. Convert a Model

First, convert a model from the OpenCLIP registry. You need the [export] extras installed for this step.

# Syntax: onnx-clip convert --model <ARCH> --pretrained <TAG> --output <DIR>

# Example: Standard ViT-B-32
onnx-clip convert --model ViT-B-32 --pretrained laion2b_s34b_b79k --output ./models/vit-b-32

# Example: ViT-L-14 (Higher accuracy)
onnx-clip convert --model ViT-L-14 --pretrained openai --output ./models/vit-l-14

This will create a folder containing visual.onnx, textual.onnx, and configuration files.

2. Run Inference (Python)

Now you can use the model in your application. This step works without PyTorch.

from onnx_clip import OnnxClip
from PIL import Image

# 1. Load the model (Provide the directory where you exported the model)
model = OnnxClip(model_dir="./models/vit-b-32", device="cpu")  # use 'cuda' for GPU

# 2. Get Image Embeddings
image = Image.open("cat.jpg")
image_features = model.get_image_embedding(image)
# shape: (1, 512)

# 3. Get Text Embeddings
text_features = model.get_text_embedding(["a photo of a cat", "a photo of a dog"])
# shape: (2, 512)

# 4. Calculate Similarity
# (The embeddings are already normalized)
similarity = image_features @ text_features.T
print(similarity)
# [[0.28, 0.15]]

3. CLI Inference (Testing)

You can also test a model directly from the CLI:

onnx-clip run --model-dir ./models/vit-b-32 --device cuda --show-providers --image cat.jpg --text "a cute cat"

GPU Support

To run on NVIDIA GPUs, simply install with the [gpu] extra:

uv add "modern-onnx-clip[gpu]"

Then initialize the model with device="cuda".

On Windows, the [gpu] extra also installs the CUDA/cuDNN runtime packages required by ONNX Runtime. If CUDA is requested but ONNX Runtime actually falls back to CPU, OnnxClip now raises a clear error instead of silently continuing on the wrong provider.

Runtime Diagnostics

The CLI can print the execution providers that ONNX Runtime actually uses:

onnx-clip run --model-dir ./models/vit-b-32 --device cuda --show-providers --text "a cute cat"

For long-lived applications, you can explicitly release native session resources:

with OnnxClip(model_dir="./models/vit-b-32", device="cuda") as model:
    image_features = model.get_image_embedding(image)

Project Structure

• exporter.py: Handles loading PyTorch models and exporting them to ONNX graphs.
• model.py: The lightweight inference engine. Abstraction over ONNX Runtime sessions.
• preprocessor.py: Reimplementation of CLIP's image preprocessing using only NumPy and Pillow.
• tokenizer.py: Handles text tokenization (BPE) without heavy external dependencies.

Development & Testing

We use pytest for testing.

Standard Tests

Run the standard test suite (ensure you have installed [cpu] or [gpu] extra):

uv run --extra cpu pytest

Manual Verification Tests

The tests/manual/ directory contains scripts to verify numerical consistency between this library (ONNX) and the original PyTorch CLIP. These tests are skipped by default if dependencies are missing. To run them:

1. Install the clip library manually (it cannot be a package dependency due to PyPI restrictions):

   pip install git+https://github.com/openai/CLIP.git
   

2. Export a model to a local directory (e.g., ../models/ViT-B-32):

   onnx-clip convert --model ViT-B-32 --pretrained laion2b_s34b_b79k --output ../models/ViT-B-32
   

3. Set the environment variable and run:

   # Linux/Mac
   export ONNX_CLIP_MODEL_DIR="../models/ViT-B-32"
   pytest tests/manual/
   
   # Windows (PowerShell)
   $env:ONNX_CLIP_MODEL_DIR="../models/ViT-B-32"
   pytest tests/manual/
   

License

MIT License.

Acknowledgements

Built on top of the incredible work by OpenAI and OpenCLIP. Inspired by the original onnx_clip package.