all_clip
Load any clip model with a standardized interface
Install
pip install all_clip
Python examples
from all_clip import load_clip
import torch
from PIL import Image
import pathlib
model, preprocess, tokenizer = load_clip("open_clip:ViT-B-32/laion2b_s34b_b79k", device="cpu", use_jit=False)
image = preprocess(Image.open(str(pathlib.Path(__file__).parent.resolve()) + "/CLIP.png")).unsqueeze(0)
text = tokenizer(["a diagram", "a dog", "a cat"])
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs) # prints: [[1., 0., 0.]]
Checkout these examples to call this as a lib:
API
This module exposes a single function load_clip:
- clip_model CLIP model to load (default ViT-B/32). See below supported models section.
- use_jit uses jit for the clip model (default True)
- warmup_batch_size warmup batch size (default 1)
- clip_cache_path cache path for clip (default None)
- device device (default None)
Related projects
- clip-retrieval to use clip for inference, and retrieval
- open_clip to train clip models
- CLIP_benchmark to evaluate clip models
Supported models
OpenAI
Specify the model as "ViT-B-32"
Openclip
"open_clip:ViT-B-32/laion2b_s34b_b79k" to use the open_clip
HF CLIP
"hf_clip:patrickjohncyh/fashion-clip" to use the hugging face
Deepsparse backend
DeepSparse is an inference runtime for fast sparse model inference on CPUs. There is a backend available within clip-retrieval by installing it with pip install deepsparse-nightly[clip], and specifying a clip_model with a prepended "nm:", such as "nm:neuralmagic/CLIP-ViT-B-32-256x256-DataComp-s34B-b86K-quant-ds" or "nm:mgoin/CLIP-ViT-B-32-laion2b_s34b_b79k-ds".
Japanese clip
japanese-clip provides some models for japanese.
For example one is ja_clip:rinna/japanese-clip-vit-b-16
How to add a model type
Please follow these steps:
- Add a file to load model in
all_clip/ - Define a loading function, that returns a tuple (model, transform, tokenizer). Please see
all_clip/open_clip.pyas an example. - Add the function into
TYPE2FUNCinall_clip/main.py - Add the model type in
test_main.pyandci.yml
Remarks:
- The new tokenizer/model must enable to do the following things as https://github.com/openai/CLIP#usage
tokenizer(texts).to(device)...textsis a list of stringmodel.encode_text(tokenized_texts)...tokenized_textsis a output fromtokenizer(texts).to(device)model.encode_image(images)...imagesis a image tensor by thetransform
For development
Setup a virtualenv:
python3 -m venv .env
source .env/bin/activate
pip install -e .
to run tests:
pip install -r requirements-test.txt
then
make lint
make test
You can use make black to reformat the code
python -m pytest -x -s -v tests -k "ja_clip" to run a specific test
Metadata
Release files for all-clip 1.3.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| all_clip-1.3.0.tar.gz | 9.5 kB | Without provenance | Details for all_clip-1.3.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| all_clip-1.3.0-py3-none-any.whl | Python 3 | none | any | Details for all_clip-1.3.0-py3-none-any.whl |
Total release size: 21.3 kB
Release files / all_clip-1.3.0.tar.gz
| Download URL | all_clip-1.3.0.tar.gz |
|---|---|
| Size | 9.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
243b273344fc770fef30672e107975bf6644603a034cfa4543daed1a45e35ef8
|
|
BLAKE2b-256 checksum How to use checksums |
413e77353066e47a4b3a27d1c368b47d1ab4ac2f3ae65f811c274fbad4f4714d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.11.13
|
Release files / all_clip-1.3.0-py3-none-any.whl
| Download URL | all_clip-1.3.0-py3-none-any.whl |
|---|---|
| Size | 11.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
fcc16526ede9d5716303dcb55e8207c4fa1c33ef43298d7074869fd7c4eebfb0
|
|
BLAKE2b-256 checksum How to use checksums |
ef5370d10c44f0f5ba3ebdbb681e3e24bc631f2d7258ce0f9d2334eef3e97dd2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.11.13
|
Security for all-clip 1.3.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.