Skip to main content

project_description

Project description

blip-inference

Pretrained BLIP with a similar API to CLIP.

BLIP tends to achieve slightly better accuracy than CLIP with similar inference speed. The CLIP API is much cleaner and more commonly used. This repo refactors BLIP to match the CLIP interface, so that it's easier for practitioners to switch between CLIP / BLIP models.

Install

From PyPI:

pip install blip-inference

From source:

pip install "blip_inference @ git+https://git@github.com/fkodom/blip-inference.git"

Usage

User-facing methods behave similarly to CLIP. A few underlying details change, which will only affect advanced users.

import torch
import blip_inference as blip
from PIL import Image

device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = blip.load("feature_extractor", device=device)

raw_text = ["a diagram", "a dog", "a cat"]
text = blip.tokenize(raw_text).to(device)
image = preprocess(Image.open("kitten.jpeg")).unsqueeze(0).to(device)

with torch.no_grad():    
    logits_per_image, logits_per_text = model(image, text)
probs = torch.softmax(logits_per_image, dim=-1)

print("\nPredictions:\n")
for idx, value in enumerate(probs.squeeze()):
    print(f"{raw_text[idx]:>16s}: {100 * value.item():.2f}%")
probs = logits_per_image.softmax(dim=-1).cpu().numpy()

Zero-Shot Prediction

import blip_inference as blip
import torch
from PIL import Image

device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = blip.load('base', device)

raw_text = ["a diagram", "a dog", "a cat"]
text = blip.tokenize(raw_text).to(device)
image = preprocess(Image.open("kitten.jpeg")).unsqueeze(0).to(device)

with torch.no_grad():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)

image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
similarity = (100 * image_features @ text_features.T).softmax(dim=-1)

print("\nPredictions:\n")
for idx, value in enumerate(similarity.squeeze()):
    print(f"{raw_text[idx]:>16s}: {100 * value.item():.2f}%")

Linear Probe Evaluation

See this example from the CLIP repo. Everything should be identical, except for swapping:

  • import clip --> import blip_inference as blip
  • clip --> blip

API

Similar to CLIP, the blip_inference module provides the following methods:

blip_inference.available_models() -> List[str]

Returns the names of the available BLIP models.

blip_inference.load(name: str, device=...) -> Tuple[BLIP, Callable]

Returns the model and the TorchVision transform needed by the model, specified by the model name returned by blip_inference.available_models(). It will download the model as necessary. The name argument can also be a path to a local checkpoint.

The device to run the model can be optionally specified, and the default is to use the first CUDA device if there is any, otherwise the CPU.

blip_inference.tokenize(text: Union[str, List[str]], context_length: int = 35) -> BatchEncoding

Returns a dictionary with tokenized sequences of given text input(s). This can be used as the input to the model


The model returned by blip_inference.load() supports the following methods:

model.encode_image(image: Tensor) -> Tensor

Given a batch of images, returns the image features encoded by the vision portion of the BLIP model.

model.encode_text(text: BatchEncoding) -> Tensor

Given a batch of text tokens, returns the text features encoded by the language portion of the BLIP model.

model(image: Tensor, text: BatchEncoding) -> Tuple[Tensor, Tensor]

Given a batch of images and a batch of text tokens, returns two Tensors, containing the logit scores corresponding to each image and text input. The values are cosine similarities between the corresponding image and text features.

NOTE: Unlike CLIP, logits for BLIP models do not need to be multiplied by 100 before computing cosine similarity. That scaling factor is built into the BLIP model.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

blip-inference-0.1.0.tar.gz (20.9 kB view details)

Uploaded Source

Built Distribution

blip_inference-0.1.0-py3-none-any.whl (20.7 kB view details)

Uploaded Python 3

File details

Details for the file blip-inference-0.1.0.tar.gz.

File metadata

  • Download URL: blip-inference-0.1.0.tar.gz
  • Upload date:
  • Size: 20.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.9.15

File hashes

Hashes for blip-inference-0.1.0.tar.gz
Algorithm Hash digest
SHA256 6dad3db5e69faff801110e03366c005921ffa92a8b2a02b6991163ee90d990d2
MD5 c47cb40203b69c78e4adb73f3d7eb02a
BLAKE2b-256 33da26ed145d7779e356c73e046fd16c89aa1d9bf751328c2cc6ebc4cd1f8e5b

See more details on using hashes here.

File details

Details for the file blip_inference-0.1.0-py3-none-any.whl.

File metadata

File hashes

Hashes for blip_inference-0.1.0-py3-none-any.whl
Algorithm Hash digest
SHA256 6fbd7447abc2ea1833fc4b4ebd3b81de94c9c496508433e75d4b1603732aefe8
MD5 62773c8996e428f5a631a932fbffff99
BLAKE2b-256 e55d818b15bb7a9f6de104f5bbc23a9014a5d8595387cbcbcf8bf565cd31a3ee

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page