Skip to main content

Enhancing Coin Image Retrieval with CLIP

Project description

Coin-CLIP 🪙 : Enhancing Coin Image Retrieval with CLIP

[中文] | [English]

Coin-CLIP breezedeus/coin-clip-vit-base-patch32 is built upon OpenAI's CLIP (ViT-B/32) model and fine-tuned on a dataset of more than 340,000 coin images using contrastive learning techniques. This specialized model is designed to significantly improve feature extraction for coin images, leading to more accurate image-based search capabilities. Coin-CLIP combines the power of Visual Transformer (ViT) with CLIP's multimodal learning capabilities, specifically tailored for the numismatic domain.

Key Features:

  • State-of-the-art coin image retrieval;
  • Enhanced feature extraction for numismatic images;
  • Seamless integration with CLIP's multimodal learning.

To further simplify the use of the Coin-CLIP model, this project provides tools for quickly building a coin image retrieval engine.

Comparison: Coin-CLIP vs. CLIP

Example 1 (Left: Coin-CLIP; Right: CLIP)

1. Coin-CLIP vs. CLIP

Example 2 (Left: Coin-CLIP; Right: CLIP)

2. Coin-CLIP vs. CLIP

More Examples

more

Example 3 (Left: Coin-CLIP; Right: CLIP) 3. Coin-CLIP vs. CLIP

Example 4 (Left: Coin-CLIP; Right: CLIP) 4. Coin-CLIP vs. CLIP

Example 5 (Left: Coin-CLIP; Right: CLIP) 5. Coin-CLIP vs. CLIP

Example 6 (Left: Coin-CLIP; Right: CLIP) 6. Coin-CLIP vs. CLIP

Install

pip install coin_clip

Usage

Code Examples

Extract Feature Vectors from Coin Images

from coin_clip import CoinClip

# Automatically download the model from Huggingface
model = CoinClip(model_name='breezedeus/coin-clip-vit-base-patch32')
images = ['examples/10_back.jpg', 'examples/16_back.jpg']
img_feats, success_ids = model.get_image_features(images)
print(img_feats.shape)  # --> (2, 512)

⚠️ Note:

The above code automatically downloads the breezedeus/coin-clip-vit-base-patch32 model from Huggingface. If you cannot download automatically, please manually download the model locally, and then initialize CoinClip by specifying the local directory of the model through the model_name parameter, like model_name='path/to/coin-clip-vit-base-patch32'.

Command line tools

Building a Vector Retrieval Engine

coin-clip build-db can be used to build a vector search engine. It extracts features from all coin images 🪙 in a specified directory and builds a ChromaDB vector search engine.

$ coin-clip build-db -h
Usage: coin-clip build-db [OPTIONS]

  Extract vectors from a candidate image set and build a search engine based
  on it.

Options:
  -m, --model-name TEXT       Model Name; either local path or huggingface
                              model name  [default: breezedeus/coin-clip-vit-
                              base-patch32]
  -d, --device TEXT           ['cpu', 'cuda']; Either 'cpu' or 'gpu', or
                              specify a specific GPU like 'cuda:0'. Default is
                              'cpu'.  [default: cpu]
  -i, --input-image-dir TEXT  Folder with Coin Images to be indexed. [required]
  -o, --output-db-dir TEXT    Folder where the built search engine is stored.
                              [default: ./coin_clip_chroma.db]
  -h, --help                  Show this message and exit.

For instance,

$ coin-clip build-db -i examples -o coin_clip_chroma.db

Querying

After building the vector search engine with the above command, you can use the coin-clip retrieve command to retrieve the coin images 🪙 most similar to a specified coin image.

$ coin-clip retrieve -h
Usage: coin-clip retrieve [OPTIONS]

  Retrieve images from the search engine, based on the query image.

Options:
  -m, --model-name TEXT  Model Name; either local path or huggingface model
                         name  [default: breezedeus/coin-clip-vit-base-
                         patch32]
  -d, --device TEXT      ['cpu', 'cuda']; Either 'cpu' or 'gpu', or specify a
                         specific GPU like 'cuda:0'. Default is 'cpu'.
                         [default: cpu]
  --db-dir TEXT          Folder where the built search engine is stored.
                         [default: ./coin_clip_chroma.db]
  -i, --image-fp TEXT    Image Path to retrieve  [required]
  -h, --help             Show this message and exit.

For instance,

$ coin-clip retrieve --db-dir coin_clip_chroma.db -i examples/10_back.jpg

Project details


Release history Release notifications | RSS feed

This version

0.1

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

coin_clip-0.1.tar.gz (12.6 kB view details)

Uploaded Source

Built Distribution

coin_clip-0.1-py3-none-any.whl (15.9 kB view details)

Uploaded Python 3

File details

Details for the file coin_clip-0.1.tar.gz.

File metadata

  • Download URL: coin_clip-0.1.tar.gz
  • Upload date:
  • Size: 12.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.9.5

File hashes

Hashes for coin_clip-0.1.tar.gz
Algorithm Hash digest
SHA256 b23e7830875a88693dd2d826712184b5150b6306cfd1a9e0021082b2a561071e
MD5 386499dda76e14fb5e4fdfeee2d43ad8
BLAKE2b-256 e9b47fcd054569570b25a0b684a199b03460306f95110e55efb1bc630afa9d40

See more details on using hashes here.

File details

Details for the file coin_clip-0.1-py3-none-any.whl.

File metadata

  • Download URL: coin_clip-0.1-py3-none-any.whl
  • Upload date:
  • Size: 15.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.9.5

File hashes

Hashes for coin_clip-0.1-py3-none-any.whl
Algorithm Hash digest
SHA256 760bd66cc291cb2aa100333b4392659a4b2c37902d3258d0e7a9ca2782591a46
MD5 376c45c674b093988d071269ba23fb22
BLAKE2b-256 8c27db2140dde6d01ec48f7444aa094b1d9db3a4e763fcdb604a2557414c316c

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page