Welcome to pyvisim!

pyvisim is a Python library for computing image similarities using image encoders and neural networks.

📚 Documentation: https://mechacritter.github.io/Python-Visual-Similarity/

Table of Contents

1. Installation
2. Why pyvisim
3. Pretrained Models
4. Contributing
5. Get in Touch
6. TODO
7. License
8. References

For a technical deep-dive into the library internals and the full API reference, see the hosted documentation (also available in this repository as the developer documentation).

Status

Installation

To use the library, you can simply install it via pip:

pip install pyvisim
# For deep learning features and the OxfordFlowerDataset
pip install "pyvisim[nn]"
# For image search feature
pip install "pyvisim[search]"

or clone the repository and install it locally:

git clone https://github.com/MechaCritter/Python-Visual-Similarity.git
cd Python-Visual-Similarity
pip install .

Note that the notebooks are only available if you clone the repository.

All experiments in this project was made on the Oxford Flower Dataset [7], for which I have created a custom dataset class. To use this class, import it as follows:

from pyvisim.datasets import OxfordFlowerDataset

For more details on the dataset, please refer to the documentation.

Why pyvisim?

pyvisim is designed to provide a simple and efficient way to compare images.

Quick Start

With just a few lines of code, you can compute the similarity score between two images using the VLAD encoder:

Example: Compute Similarity Score Using Vector of Locally Aggregated Descriptors (VLAD) [5]

from pyvisim.encoders import VLADEncoder, PretrainedVLAD
from pyvisim.datasets import OxfordFlowerDataset  # needs "nn" extra: install with `pip install "pyvisim[nn]"`

# Load images from the Oxford Flower Dataset. Has to be NumPy Images!
dataset = OxfordFlowerDataset()
image1, *_  = dataset[0]
image2, *_ = dataset[1]

# Load a bundled pretrained VLAD encoder (RootSIFT features, k=256).
# The feature extractor and similarity metric come with it.
encoder = VLADEncoder.from_pretrained(PretrainedVLAD.OXFORD102_K256_ROOTSIFT)

# Compute the similarity score. By default, cosine similarity is used.
similarity_score = encoder.similarity_score(image1, image2)

print(f"Similarity Score: {similarity_score}")

By default the encoder uses cosine similarity. To use a different metric, pass its name; "cosine", "euclidean", "l1" and "manhattan" are supported:

encoder = VLADEncoder.from_pretrained(PretrainedVLAD.OXFORD102_K256_ROOTSIFT)
encoder.similarity_func = "euclidean"

A fitted encoder can be saved to a .encoder file and restored later:

path = encoder.save_to_disk("vlad_oxford102")  # writes vlad_oxford102.encoder
encoder = VLADEncoder.load_from_disk(path)

You can also visit the introduction notebook for more examples.

I also provided various notebooks for different use-cases. Feel free to check them out, and let me know if you have any suggestions or questions!

1. Image Retrieval
   Retrieve the top-k most similar images from a dataset.

   • Use encoding methods like VLAD or Fisher Vectors to quickly find the most relevant matches. Please visit this juptyer notebook for an example.

   • For large galleries, build an InMemoryImageEmbeddingStore over your image paths; it indexes the embeddings and searches them for you (needs the search extra: pip install "pyvisim[search]"):

     from pyvisim.image_store import InMemoryImageEmbeddingStore
     
     store = InMemoryImageEmbeddingStore(
         gallery_paths, encoder, "ivf-flat",
         quantizer="inner_product", index_params={"nlist": 100},
     )
     results = store.retrieve_top_k_similar(query_images, k=5)
     

     See the retrieval docs for more information.

   • Example use: Building a fast image search engine for photo management software.

2. Deep Learning Embeddings

   • Generate VLAD or Fisher vectors from neural network embeddings, e.g., VGG16 or other models.
   • Enhance your deep learning pipeline by leveraging traditional encoding methods on top of CNN features.
   • Or skip the aggregation entirely and use ClipEmbedder (in pyvisim.neural_networks) for ready-made CLIP embeddings, loaded straight from OpenAI's official checkpoints.
   • The VGG16 deep-feature path (DeepConvFeature) and ClipEmbedder both need the nn extra: pip install "pyvisim[nn]".

3. Image Clustering

   • Cluster images based on their similarities to group them by category or content. An example and benchmarking can be found in this notebook.
   • Useful for organizing unlabeled data or generating pseudo-labels for further training.

4. Pipeline for Combining Multiple Encoders

   • Chain various encoders in a single pipeline. An example can be found in this notebook.
   • Achieve more robust similarity metrics by blending different feature representations.

5. Siamese Network

   • Learn a similarity function directly from pairs of images with a Siamese network (needs the nn extra: pip install "pyvisim[nn]").

   • Score two images out of the box, or fine-tune on your own labelled pairs with the bundled ContrastiveLoss and training script:

     from pyvisim.neural_networks import SiameseNeuralNetwork
     
     model = SiameseNeuralNetwork(backbone="resnet18", embedding_dim=128)
     score = model.similarity_score(image1, image2)  # cosine similarity in [-1, 1]
     

     See the neural networks docs for more details.

   • Possible use cases include face recognition, signature verification, or any image-based identity matching.

Pretrained Models

pyvisim ships ready-to-use pretrained encoders trained on the Oxford-102 flower dataset. Each one is a bundled .encoder file that already includes the right feature extractor and the cosine similarity metric, so loading one gives you a working encoder in a single line:

from pyvisim.encoders import (
    VLADEncoder,
    FisherVectorEncoder,
    PretrainedVLAD,
    PretrainedFisher,
)

vlad = VLADEncoder.from_pretrained(PretrainedVLAD.OXFORD102_K256_ROOTSIFT)
fisher = FisherVectorEncoder.from_pretrained(PretrainedFisher.OXFORD102_K256_VGG16_PCA)

The SIFT/RootSIFT variants work on the base install. The *_VGG16* variants build a DeepConvFeature, so they need the nn extra: pip install "pyvisim[nn]".

All clustering models were trained with k=256. The choice of k was made arbitrarily based on the paper ⁵, where the authors tested with k=32, 64, 128, 256, 512, and so on. Since higher values would take too long, I chose k=256 as a balance between performance and computational cost. Variants ending in _PCA reduce the feature dimensions by half with PCA before clustering.

VLAD encoders (PretrainedVLAD)

Loaded with VLADEncoder.from_pretrained(...).

Fisher Vector encoders (PretrainedFisher)

Loaded with FisherVectorEncoder.from_pretrained(...).

Notes

1. Feature Extraction:
   • Deep Features (VGG16): Feature maps from the last convolutional layer of VGG16. At each spatial location, the relative x and y coordinates are concatenated to the feature vector, resulting in 512 + 2 = 514 dimensions ⁶.
   • SIFT: Scale-Invariant Feature Transform descriptors, which was the original feature used for VLAD and Fisher Vector encoding ⁵.
   • RootSIFT: A variant of SIFT with Hellinger kernel normalization⁴.
2. Dimensionality Reduction:
   • Models with _PCA in their names apply PCA to reduce the feature dimensions to by half.
   • The clustering models will learn from the transformed features after PCA is applied.

Contributing

We love contributions of all kinds—whether it’s suggesting new features, fixing bugs, or writing docs! Here’s how you can get involved:

1. Fork this repository.
2. Create a new branch for your changes.
3. Open a pull request with a clear description of your idea or fix.

We welcome all feedback and hope to build a supportive community around pyvisim!

Get in Touch

If you have any questions or just want to say hi, feel free to:

• Open an issue on GitHub.
• Write me an email at vunhathuy234@gmail.com.
• Connect on LinkedIn to follow my work and share your thoughts.

TODO

The features below are planned for future releases:

• With v1.0.0, remove the deprecated weights constructor argument and the _CLUSTERING_TO_PCA_MAPPING internal variable, since they are no longer needed with the new from_pretrained() API.
• Add the one-shot Siamese network variant (Koch et al., 2015) alongside the existing contrastive implementation (Hadsell, Chopra & LeCun, 2006).
• Add tensor sketch approximation and mutual information analysis for Fisher Vector, according to this paper by Weixia Zhang, Jia Yan, Wenxuan Shi, Tianpeng Feng, and Dexiang Deng ¹
• Add support for vision transformers for the DeepConvFeature class.

You are welcome to implement any of these features or suggest new ones!

License

This project is licensed under the terms of the MIT license.

References

[1] Weixia Zhang, Jia Yan, Wenxuan Shi, Tianpeng Feng, and Dexiang Deng, "Refining Deep Convolutional Features for Improving Fine-Grained Image Recognition," EURASIP Journal on Image and Video Processing, 2017.
[2] Relja Arandjelović and Andrew Zisserman, 'All About VLAD', Department of Engineering Science, University of Oxford.
[3] E. Spyromitros-Xioufis, S. Papadopoulos, I. Kompatsiaris, G. Tsoumakas, and I. Vlahavas, "An Empirical Study on the Combination of SURF Features with VLAD Vectors for Image Search," Informatics and Telematics Institute, Center for Research and Technology Hellas, Thessaloniki, Greece; Department of Informatics, Aristotle University of Thessaloniki, Greece.
[4] Relja Arandjelović and Andrew Zisserman, "Three things everyone should know to improve object retrieval," Department of
Engineering Science, University of Oxford.
[5] Hervé Jégou, Florent Perronnin, Matthijs Douze, Jorge Sánchez, Patrick Pérez, and Cordelia Schmid, "Aggregating Local Image Descriptors into Compact Codes," IEEE.
[6] Liangliang Wang and Deepu Rajan, "An Image Similarity Descriptor for Classification Tasks," J. Vis. Commun. Image R., vol. 71, pp. 102847, 2020.
[7] Oxford Flower Dataset.