TPIPS API reference

TPIPS is a text-conditioned perceptual image metric. It compares images with respect to a text factor/aspect such as "lighting", "pose", or "overall".

On this page

• tpips.load_model
• model.similarity
• model.distance
• model.embed

TPIPS requires Python 3.10 or later. Install matching PyTorch and torchvision builds from the official PyTorch installer, then install the package:

pip install tpips

For datasets, evaluation, training, and the interactive demo, see the repository README.

tpips.load_model

tpips.load_model(
    model_type,
    model_path=None,
    device="cuda",
    *,
    cache_dir=None,
)

Loads a released TPIPS model from Hugging Face or a compatible local checkpoint. The returned model is placed on device and set to evaluation mode.

Parameters

• model_type (str) – Selects the TPIPS metric architecture.

  Value	Supported methods	Description
  "embedding"	similarity, distance, embed	Encodes each text-image input independently. This is the recommended default and the only variant with reusable image embeddings.
  "early_fusion"	similarity, distance	Jointly encodes each aligned image pair. Its exchange-equivariant attention design makes scores invariant to image order, apart from numerical noise.
  "activation_dist"	distance	Computes a learned distance from text-conditioned features at several backbone layers. It does not define cosine similarity or independent image embeddings.

• model_path (str or os.PathLike, optional) – Hugging Face repository ID or local checkpoint directory. If omitted, loads the official checkpoint for model_type.

• device (str, optional) – Torch destination such as "cuda", "cuda:1", or "cpu". Default: "cuda".

• cache_dir (str or os.PathLike, optional) – Hugging Face cache override for the TPIPS checkpoint, Qwen backbone, and processor. By default, Hugging Face uses its standard cache and environment variables such as HF_HOME.

Returns

A device-resident EmbeddingModel, EarlyFusionModel, or ActivationDistModel in evaluation mode.

Raises

• ValueError – If model_type is unsupported; a local path names a file rather than a directory; or the checkpoint configuration is invalid or does not match the requested model type.
• FileNotFoundError – If a checkpoint directory does not contain config.json or model.pt.
• RuntimeError – If a Hub download, checkpoint read, model construction, state-dict load, or device transfer fails.
• TypeError – If a path or device argument has an unsupported type.

Checkpoint configuration

A compatible checkpoint directory contains config.json and model.pt. The configuration declares the metric architecture and backbone family:

{
  "model": {
    "type": "embedding",
    "backbone_family": "qwen3_vl_embed",
    "name": "Qwen/Qwen3-VL-Embedding-8B"
  }
}

model.type must match model_type. model.backbone_family currently must be qwen3_vl_embed, and model.name identifies the concrete Hugging Face backbone. Official TPIPS releases already contain the correct configuration.

The default checkpoints are:

• Embedding
• Early fusion
• Activation distance

Notes

The first call downloads both the TPIPS checkpoint and its Qwen backbone. Subsequent calls reuse the Hugging Face cache. A CUDA GPU is recommended. FlashAttention is optional because TPIPS falls back to PyTorch SDPA.

Evaluation mode disables dropout but does not disable autograd. Use torch.inference_mode() for ordinary inference, as shown below.

Direct construction is available from the metric packages:

from tpips.embedding import EmbeddingQwen3VLEmbedModel
from tpips.early_fusion import EarlyFusionQwen3VLEmbedModel
from tpips.activation_dist import ActivationDistQwen3VLEmbedModel

Constructing these classes initializes a Qwen backbone and processor but does not load a released TPIPS model.pt. Direct construction is intended for architecture development or manually loading a compatible state dict. Normal inference code should use tpips.load_model.

Examples

import torch
import tpips
from PIL import Image

model = tpips.load_model("embedding", device="cuda")

image_a = Image.open("a.jpg").convert("RGB")
image_b = Image.open("b.jpg").convert("RGB")

with torch.inference_mode():
    similarity = model.similarity(image_a, image_b, factor="lighting")
    distance = model.distance(image_a, image_b, factor="lighting")

print(similarity.item())  # higher means more similar
print(distance.item())    # lower means more similar

Load another Hub repository, a local checkpoint, or a dedicated cache:

hub_model = tpips.load_model(
    "embedding",
    "organization/custom-tpips",
    device="cuda:1",
    cache_dir="/data/huggingface-cache",
)

local_model = tpips.load_model(
    "early_fusion",
    "/models/my-tpips",
    device="cpu",
)

model.similarity

model.similarity(
    img_a,
    img_b,
    factor="overall",
    *,
    value_range=(0.0, 1.0),
    channel_last=False,
)

Computes text-conditioned cosine similarity for aligned image pairs. Higher values mean that the images are more similar with respect to factor.

This method is implemented by the embedding and early_fusion variants. activation_dist.similarity() raises NotImplementedError; use distance() for that model.

Parameters

• img_a – First image or explicit image batch.
• img_b – Second image or explicit image batch. Images are paired element-by-element with img_a.
• factor (str, optional) – One text criterion used for the entire call, such as "lighting", "subject color", or "composition". Use "overall" or None for general perceptual similarity. Default: "overall".
• value_range (two-element tuple, optional) – Inclusive finite bounds for every numeric image. Numeric pixels are linearly mapped from this range to [0, 1]. Use (0, 255) for byte-valued input, including uint8 arrays. PIL images ignore this argument. Default: (0.0, 1.0).
• channel_last (bool, optional) – Numeric image layout. False means CHW or BCHW; True means HWC or BHWC. PIL images ignore this argument. Default: False.

Image inputs

Input	Interpretation	channel_last=False	channel_last=True
PIL.Image.Image	One image	Any PIL mode	Same; converted to RGB
3-dimensional NumPy array	One image	(3, H, W)	(H, W, 3)
3-dimensional Torch tensor	One image	(3, H, W)	(H, W, 3)
4-dimensional NumPy array	Explicit batch	(B, 3, H, W)	(B, H, W, 3)
4-dimensional Torch tensor	Explicit batch	(B, 3, H, W)	(B, H, W, 3)
Python list	Explicit batch	PIL or 3-dimensional numeric items	Layout applies to numeric items

A list or 4-dimensional value is an explicit batch even when it contains one image. A Python list must be non-empty and may mix PIL images, 3-dimensional NumPy arrays, and 3-dimensional Torch tensors. It cannot contain a nested 4-dimensional batch. One value_range and one channel_last setting apply to every numeric image in the call.

Numeric images must have exactly three channels, non-zero height and width, and finite pixels inside value_range. NumPy integer and floating dtypes are supported. Torch tensors must be real, non-boolean, non-complex, and non-quantized.

img_a and img_b must contain the same number of images. TPIPS pairs img_a[i] only with img_b[i]; it does not broadcast inputs or produce an all-pairs matrix. A singular image may be paired with an explicit one-image batch, in which case the output remains an explicit batch.

Returns

A device-resident Torch tensor:

• Shape () when both inputs are singular.
• Shape (B,) when either input is an explicit batch, including (1,) for a one-image batch.

Embedding and early-fusion similarity are cosine similarities. The early-fusion attention design makes the result invariant to swapping the input images, apart from numerical noise.

Raises

• NotImplementedError – If called on activation_dist.
• TypeError – If an image type, numeric dtype, or value_range bound has an unsupported type.
• ValueError – If an input is empty; dimensions, channels, or value_range are invalid; pixels are non-finite or outside the declared range; a list contains a 4-dimensional batch; or pair counts differ.
• RuntimeError – If model preprocessing or inference fails.

Autograd

TPIPS does not enter torch.no_grad() or torch.inference_mode(). Floating Torch inputs retain their autograd connection through normalization, Qwen preprocessing, the backbone, and the TPIPS head. PIL and NumPy inputs cannot carry a PyTorch input-gradient graph.

To compute image gradients without accumulating model-parameter gradients:

model.requires_grad_(False)
image_a = image_a.detach().requires_grad_(True)
image_b = image_b.detach().requires_grad_(True)

score = model.similarity(image_a, image_b, factor="overall")
score.backward()

gradient_a = image_a.grad
gradient_b = image_b.grad

Backpropagation retains the large backbone's activations and uses substantially more memory than inference.

Examples

Compare HWC uint8 images:

import numpy as np
from PIL import Image

image_a = np.asarray(Image.open("a.jpg").convert("RGB"))
image_b = np.asarray(Image.open("b.jpg").convert("RGB"))

score = model.similarity(
    image_a,
    image_b,
    factor="lighting",
    channel_last=True,
    value_range=(0, 255),
)

Compare an aligned batch:

import torch
from PIL import Image

left = [Image.open("a0.jpg"), Image.open("a1.jpg")]
right = [Image.open("b0.jpg"), Image.open("b1.jpg")]

with torch.inference_mode():
    scores = model.similarity(left, right, factor="lighting")

assert scores.shape == (2,)

model.distance

model.distance(
    img_a,
    img_b,
    factor="overall",
    *,
    value_range=(0.0, 1.0),
    channel_last=False,
)

Computes text-conditioned distance for aligned image pairs. Lower values mean that the images are more similar with respect to factor.

This method is implemented by all three model variants.

Parameters

• img_a, img_b – Images or aligned image batches in the formats documented by similarity().
• factor (str, optional) – Text criterion used for the entire call. Default: "overall".
• value_range (two-element tuple, optional) – Inclusive numeric pixel range. Default: (0.0, 1.0).
• channel_last (bool, optional) – Whether numeric inputs use HWC or BHWC rather than CHW or BCHW. Default: False.

Input conversion, validation, aligned batching, output shapes, and autograd behavior are identical to similarity().

Returns

A device-resident Torch tensor with shape () for singular inputs or (B,) for an explicit batch.

For embedding and early_fusion, distance is exactly 1 - similarity. For activation_dist, it is a learned non-negative feature distance and is not a cosine distance. Scores from different variants should not be compared numerically.

Raises

• TypeError, ValueError, RuntimeError – Under the same input and execution conditions documented by similarity().

Notes

Activation distance requires aligned patch grids. When the two images in a pair have different spatial sizes, the model resizes the larger image to the smaller exact height and width with antialiased bicubic Torch interpolation and emits a UserWarning.

Examples

import torch
import tpips

model = tpips.load_model("activation_dist", device="cuda")

with torch.inference_mode():
    distances = model.distance(batch_a, batch_b, factor="composition")

model.embed

model.embed(
    images,
    factor="overall",
    normalized=False,
    *,
    value_range=(0.0, 1.0),
    channel_last=False,
)

Encodes images into reusable text-conditioned vectors. This method is available only on the embedding model. early_fusion.embed() and activation_dist.embed() raise NotImplementedError because those architectures do not define independent image embeddings.

Parameters

• images – One image or an explicit image batch in the formats documented by similarity().
• factor (str, optional) – Text criterion used for every image. Default: "overall".
• normalized (bool, optional) – If True, returns unit-L2 vectors. If False, returns raw vectors. Default: False.
• value_range (two-element tuple, optional) – Inclusive numeric pixel range. Default: (0.0, 1.0).
• channel_last (bool, optional) – Whether numeric inputs use HWC or BHWC rather than CHW or BCHW. Default: False.

Image conversion, validation, explicit-batch semantics, and autograd behavior are the same as for similarity().

Returns

A device-resident float32 Torch tensor:

• Shape (D,) for a singular image.
• Shape (B, D) for an explicit batch, including (1, D) for a one-image batch.

Rows preserve input order. The factor is part of the embedding input, so only compare or index vectors generated with the same factor.

Raises

• NotImplementedError – If called on early_fusion or activation_dist.
• TypeError, ValueError, RuntimeError – Under the same input and execution conditions documented by similarity().

Examples

Reuse normalized embeddings to compare one query against a gallery:

import torch
import tpips

model = tpips.load_model("embedding", device="cuda")

with torch.inference_mode():
    query = model.embed(query_image, factor="style", normalized=True)
    gallery = model.embed(gallery_images, factor="style", normalized=True)
    similarities = gallery @ query

assert similarities.shape == (len(gallery_images),)

Citation

If you find TPIPS useful for your research, please cite:

@article{wang2026tpips,
      title={The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric},
      author={Wang, Sheng-Yu and Nitzan, Yotam and Hertzmann, Aaron and Zhu, Jun-Yan and Shechtman, Eli and Efros, Alexei A. and Zhang, Richard},
      journal={arXiv preprint arXiv:2607.18237},
      year={2026}
}

License

TPIPS is provided under the Adobe Research License for noncommercial research use. See the license for the complete terms.