clustimage 1.3.15

Python package clustimage is for unsupervised clustering of images.

clustimage

• A step-by-step guide for clustering images

The aim of clustimage is to detect natural groups or clusters of images.

Image recognition is a computer vision task for identifying and verifying objects/persons on a photograph. We can seperate the image recognition task into the two broad tasks, namely the supervised and unsupervised task. In case of the supervised task, we have to classify an image into a fixed number of learned categories. Most packages rely on (deep) neural networks, and try solve the problem of predicting "whats on the image". In case of the unsupervised task, we do not depend on the fact that training data is required but we can interpret the input data and find natural groups or clusters. However, it can be quit a breath to carefully group similar images in an unsupervised manner, or simply identify the unique images.

The aim of clustimage is to detect natural groups or clusters of images. It works using a multi-step proces of carefully pre-processing the images, extracting the features, and evaluating the optimal number of clusters across the feature space. The optimal number of clusters can be determined using well known methods suchs as silhouette, dbindex, and derivatives in combination with clustering methods, such as agglomerative, kmeans, dbscan and hdbscan. With clustimage we aim to determine the most robust clustering by efficiently searching across the parameter and evaluation the clusters. Besides clustering of images, the clustimage model can also be used to find the most similar images for a new unseen sample.

A schematic overview is as following:

clustimage overcomess the following challenges:

* 1. Robustly groups similar images.
* 2. Returns the unique images.
* 3. Finds higly similar images for a given input image.

clustimage is fun because:

* It does not require a learning proces.
* It can group any set of images.
* It can return only the unique() images.
* it can find highly similar images given an input image.
* It provided many plots to improve understanding of the feature-space and sample-sample relationships
* It is build on core statistics, such as PCA, HOG and many more, and therefore it does not has a dependency block.
* It works out of the box.

Installation

• Install clustimage from PyPI (recommended). clustimage is compatible with Python 3.6+ and runs on Linux, MacOS X and Windows.
• A new environment can be created as following:

conda create -n env_clustimage python=3.8
conda activate env_clustimage

• Install from pypi

pip install -U clustimage

Import the clustimage package

from clustimage import Clustimage

Example 1: Digit images.

In this example we will be using a flattened grayscale image array loaded from sklearn. The array in NxM, where N are the samples and M the flattened raw rgb/gray image.

# Load library
import matplotlib.pyplot as plt
from clustimage import Clustimage
# init
cl = Clustimage()
# Load example digit data
X = cl.import_example(data='mnist')

print(X)
# Each row is an image that can be plotted after reshaping:
plt.imshow(X[0,:].reshape(8,8), cmap='binary')
# array([[ 0.,  0.,  5., ...,  0.,  0.,  0.],
#        [ 0.,  0.,  0., ..., 10.,  0.,  0.],
#        [ 0.,  0.,  0., ..., 16.,  9.,  0.],
#        ...,
#        [ 0.,  0.,  0., ...,  9.,  0.,  0.],
#        [ 0.,  0.,  0., ...,  4.,  0.,  0.],
#        [ 0.,  0.,  6., ...,  6.,  0.,  0.]])
# 
# Preprocessing and feature extraction
results = cl.fit_transform(X)

# Lets examine the results.
print(results.keys())
# ['feat', 'xycoord', 'pathnames', 'filenames', 'labels']
# 
# feat      : Extracted features
# xycoord   : Coordinates of samples in the embedded space.
# filenames : Name of the files
# pathnames : Absolute location of the files
# labels    : Cluster labels in the same order as the input

# Get the unique images
unique_samples = cl.unique()
# 
print(unique_samples.keys())
# ['labels', 'idx', 'xycoord_center', 'pathnames']
# 
# Collect the unique images from the input
X[unique_samples['idx'],:]

Plot the unique images.

cl.plot_unique()

Scatter samples based on the embedded space.

# The scatterplot that is coloured on the clusterlabels. The clusterlabels should match the unique labels.
# Cluster 1 contains digit 4
# Cluster 5 contains digit 2
# etc
# 
# No images in scatterplot
cl.scatter(zoom=None)

# Include images scatterplot
cl.scatter(zoom=4)
cl.scatter(zoom=8, plt_all=True, figsize=(150,100))

Plot the clustered images

# Plot all images per cluster
cl.plot(cmap='binary')

# Plot the images in a specific cluster
cl.plot(cmap='binary', labels=[1,5])

Dendrogram

# The dendrogram is based on the high-dimensional feature space.
cl.dendrogram()

Make various other plots

# Plot the explained variance
cl.pca.plot()
# Make scatter plot of PC1 vs PC2
cl.pca.scatter(legend=False, label=False)
# Plot the evaluation of the number of clusters
cl.clusteval.plot()
# Make silhouette plot
cl.clusteval.scatter(cl.results['xycoord'])

Example 2: Flower images.

In this example I will be using flower images for which the path locations are somewhere on disk.

# Load library
from clustimage import Clustimage
# init
cl = Clustimage(method='pca')
# load example with flowers
pathnames = cl.import_example(data='flowers')
# The pathnames are stored in a list
print(pathnames[0:2])
# ['C:\\temp\\flower_images\\0001.png', 'C:\\temp\\flower_images\\0002.png']

# Preprocessing, feature extraction and clustering. Lets set a minimum of 1-
results = cl.fit_transform(pathnames)

# Lets first evaluate the number of detected clusters.
# This looks pretty good because there is a high distinction between the peak for 5 clusters and the number of clusters that subsequently follow.
cl.clusteval.plot()
cl.clusteval.scatter(cl.results['xycoord'])

Scatter

cl.scatter(dotsize=50, zoom=None)
cl.scatter(dotsize=50, zoom=0.5)
cl.scatter(dotsize=50, zoom=0.5, img_mean=False)
cl.scatter(dotsize=50, zoom=0.5, img_mean=False)
cl.scatter(zoom=1.2, plt_all=True, figsize=(150,100))

Plot the clustered images

# Plot unique images
cl.plot_unique()
cl.plot_unique(img_mean=False)

# Plot all images per cluster
cl.plot()

# Plot the images in a specific cluster
cl.plot(labels=3)

# Plot dendrogram
cl.dendrogram()
# Plot clustered images
cl.plot()

Make prediction for unseen input image.

# Find images that are significanly similar as the unseen input image. 
results_find = cl.find(path_to_imgs[0:2], alpha=0.05)
cl.plot_find()

# Map the unseen images in existing feature-space.
cl.scatter()

Example 3: Cluster the faces on images.

from clustimage import Clustimage
# Initialize with PCA
cl = Clustimage(method='pca', grayscale=True)
# Load example with faces
X = cl.import_example(data='faces')
# Initialize and run
results = cl.fit_transform(X)

# In case you need to extract the faces from the images
# face_results = cl.extract_faces(pathnames)
# The detected faces are extracted and stored in face_resuls. We can now easily provide the pathnames of the faces that are stored in pathnames_face.
# results = cl.fit_transform(face_results['pathnames_face'])

# Plot the evaluation of the number of clusters. As you can see, the maximum number of cluster evaluated is 24 can perhaps be too small.
cl.clusteval.plot()
# Lets increase the maximum number and clusters and run solely the clustering. Note that you do not need to fit_transform() anymore. You can only do the clustering now.
cl.cluster(max_clust=35)
# And plot again. As you can see, it keeps increasing which means that it may not found any local maximum anymore.
# When looking at the graph, we see a local maximum at 12 clusters. Lets go for that
cl.cluster(min_clust=4, max_clust=20)

# Lets plot the 12 unique clusters that contain the faces
cl.plot_unique()

# Scatter
cl.scatter(zoom=None)
cl.scatter(zoom=0.2)

# Make plot
cl.plot(show_hog=True, labels=[1,7])

# Plot faces
cl.plot_faces()
# Dendrogram depicts the clustering of the faces
cl.dendrogram()

Example 4: Break up the steps

Instead of using the all-in-one functionality: fit_transform(), it is also possible to break-up the steps.

from clustimage import Clustimage

# Initialize
cl = Clustimage(method='pca')

# Import data
Xraw = cl.import_example(data='flowers')
Xraw = cl.import_example(data='mnist')
Xraw = cl.import_example(data='faces')

# Check whether in is dir, list of files or array-like
X = cl.import_data(Xraw)

# Extract features using method
Xfeat = cl.extract_feat(X)

# Embedding using tSNE
xycoord = cl.embedding(Xfeat)

# Cluster
labels = cl.cluster()

# Return
results = cl.results

# Or all in one run
# results = cl.fit_transform(X)

# Plots
cl.clusteval.plot()
cl.scatter()
cl.plot_unique()
cl.plot()
cl.dendrogram()

# Find
results_find = cl.find(Xraw[0], k=0, alpha=0.05)
cl.plot_find()

Example: Extract images belonging to clusters

The results obtained from the cl.fit_transform() or cl.cluster() is a dictionary containing the following keys:

* img       : image vector of the preprocessed images
* feat      : Features extracted for the images
* xycoord   : X and Y coordinates from the embedding
* pathnames : Absolute path location to the image file
* filenames : File names of the image file
* labels    : Cluster labels

# Import library
from clustimage import Clustimage
# Initialize
cl = Clustimage(method='pca')
# Import data
pathnames = cl.import_example(data='flowers')
# Cluster flowers
results = cl.fit_transform(pathnames)

# All results are stored in a dict:
print(cl.results.keys())
# Which is the same as:
print(results.keys())

dict_keys(['img', 'feat', 'xycoord', 'pathnames', 'labels', 'filenames'])

# Extracting images that belong to cluster label=0:
Iloc = cl.results['labels']==0
cl.results['pathnames'][Iloc]

# Extracting xy-coordinates for the scatterplot for cluster 0:
import matplotlib.pyplot as plt
xycoord = cl.results['xycoord'][Iloc]
plt.scatter(xycoord[:,0], xycoord[:,1])

# Plot the images for cluster 0:
# Images in cluster 0
imgs = np.where(cl.results['img'][Iloc])[0]
# Make sure you get the right dimension
dim = cl.get_dim(cl.results['img'][Iloc][0,:])
# Plot
for img in imgs:
  plt.figure()
  plt.imshow(img.reshape(dim))
  plt.title()

Maintainers

• Erdogan Taskesen, github: erdogant
• https://github.com/erdogant/clustimage
• Please cite in your publications if this is useful for your research (see citation).
• All kinds of contributions are welcome!
• If you wish to buy me a Coffee for this work, it is very appreciated :) See LICENSE for details.

Other interesting stuff

• https://towardsdatascience.com/how-to-cluster-images-based-on-visual-similarity-cd6e7209fe34