Skip to main content

difPy Duplicate Image Finder - automated search for duplicate or similar images.

Project description

Duplicate Image Finder (difPy)

Tired of going through all images in a folder and comparing them manually to check if they are duplicates?

The Duplicate Image Finder (difPy) Python package automates this task for you!

Read more on how the algorithm of difPy works in my Medium article Finding Duplicate Images with Python.

For a detailed usage guide, please view the official difPy Usage Documentation.


Description

difPy searches for images in one or more different folders, compares the images it found and checks whether these are duplicates. It then outputs the image files classified as duplicates as well as the images having the lowest resolutions, so you know which of the duplicate images are safe to be deleted. You can then either delete them manually, or let difPy delete them for you.

difPy does not compare images based on their hashes. It compares them based on their tensors i. e. the image content - this allows difPy to not only search for duplicate images, but also for similar images.

difPy leverages Python's multiprocessing capabilities and is therefore able to perform at high performance even on large datasets.

Table of Contents

  1. Basic Usage
  2. Output
  3. Additional Parameters
  4. CLI Usage

Basic Usage

To make difPy search for duplicates within one folder:

import difPy
dif = difPy.build('C:/Path/to/Folder/')
search = difPy.search(dif)

To search for duplicates within multiple folders:

import difPy
dif = difPy.build(['C:/Path/to/FolderA/', 'C:/Path/to/FolderB/', 'C:/Path/to/FolderC/',...])
search = difPy.search(dif)

Folder paths can be specified as standalone Python strings, or within a list. With difPy.build(), difPy first scans the images in the provided folders and builds a collection of images by generating image tensors. difPy.search() then starts the search for duplicate images.

Output

difPy returns various types of output that you may use depending on your use case:

I. Search Result Dictionary

A JSON formatted collection of duplicates/similar images (i. e. match groups) that were found. Each match group has a primary image (the key of the dictionary) which holds the list of its duplicates including their filename and MSE (Mean Squared Error). The lower the MSE, the more similar the primary image and the matched images are. Therefore, an MSE of 0 indicates that two images are exact duplicates.

search.result

> Output:
{'C:/Path/to/Image/image1.jpg' : [['C:/Path/to/Image/duplicate_image1a.jpg', 0.0], 
                                  ['C:/Path/to/Image/duplicate_image1b.jpg', 0.0]],
 'C:/Path/to/Image/image2.jpg' : [['C:/Path/to/Image/duplicate_image2a.jpg', 0.0]],
 ...
}

II. Lower Quality Files

A list of duplicates/similar images that have the lowest quality among match groups:

search.lower_quality

> Output:
['C:/Path/to/Image/duplicate_image1.jpg', 
 'C:/Path/to/Image/duplicate_image2.jpg', ...]

Lower quality images then can be moved to a different location:

search.move_to(destination_path='C:/Path/to/Destination/')

Or deleted:

search.delete(silent_del=False)

III. Process Statistics

A JSON formatted collection with statistics on the completed difPy processes:

search.stats

> Output:
{'directory': ['C:/Path/to/Folder_A/', 'C:/Path/to/Folder_B/', ... ],
 'process': {'build': {'duration': {'start': '2024-02-18T19:52:39.479548',
                                    'end': '2024-02-18T19:52:41.630027',
                                    'seconds_elapsed': 2.1505},
                       'parameters': {'recursive': True,
                                      'in_folder': False,
                                      'limit_extensions': True,
                                      'px_size': 50,
                                      'processes': 5},
                       'total_files': {'count': 3232},
                       'invalid_files': {'count': 0, 
                                         'logs': {}}},
             'search': {'duration': {'start': '2024-02-18T19:52:41.630027',
                                     'end': '2024-02-18T19:52:46.770077',
                                     'seconds_elapsed': 5.14},
                        'parameters': {'similarity_mse': 0,
                                       'rotate': True,
                                       'lazy': True,
                                       'processes': 5,
                                       'chunksize': None},
                        'files_searched': 3232,
                        'matches_found': {'duplicates': 3030, 
                                          'similar': 0}}}}

Additional Parameters

difPy supports the following parameters:

difPy.build(*directory, recursive=True, in_folder=False, limit_extensions=True, 
            px_size=50, show_progress=True, processes=None)
difPy.search(difpy_obj, similarity='duplicates', rotate=True, lazy=True, 
             show_progress=True, processes=None, chunksize=None)

CLI Usage

difPy can also be invoked through the CLI by using the following commands:

python dif.py #working directory

python dif.py -D 'C:/Path/to/Folder/'

python dif.py -D 'C:/Path/to/Folder_A/' 'C:/Path/to/Folder_B/' 'C:/Path/to/Folder_C/'

difPy CLI supports the following arguments:

dif.py [-h] [-D DIRECTORY [DIRECTORY ...]] [-Z OUTPUT_DIRECTORY] 
       [-r {True,False}] [-i {True,False}] [-le {True,False}] 
       [-px PX_SIZE]  [-s SIMILARITY] [-ro {True,False}]
       [-la {True,False}] [-proc PROCESSES] [-ch CHUNKSIZE] 
       [-mv MOVE_TO] [-d {True,False}] [-sd {True,False}]
       [-p {True,False}]

If no directory parameter is given in the CLI, difPy will run on the current working directory.

When running from the CLI, the output of difPy is written to files and saved in the working directory by default. To change the default output directory, specify the -Z / -output_directory parameter. The "xxx" in the output filenames is the current timestamp:

difPy_xxx_results.json
difPy_xxx_lower_quality.json
difPy_xxx_stats.json

For a detailed usage guide, please view the official difPy Usage Documentation.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

difPy-4.1.1.tar.gz (18.4 kB view details)

Uploaded Source

Built Distribution

difPy-4.1.1-py3-none-any.whl (15.2 kB view details)

Uploaded Python 3

File details

Details for the file difPy-4.1.1.tar.gz.

File metadata

  • Download URL: difPy-4.1.1.tar.gz
  • Upload date:
  • Size: 18.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.11.2

File hashes

Hashes for difPy-4.1.1.tar.gz
Algorithm Hash digest
SHA256 b4f2bcca34986199213ef0eaded00836ab9e531d7394a433b8a599b3f4bdfb9a
MD5 755329dffc92d4886b0a06f4ad199d30
BLAKE2b-256 5b8f4e7e5a813f37a9426cc2b999712df363d3dbac9d8f3777c58f486658a19a

See more details on using hashes here.

File details

Details for the file difPy-4.1.1-py3-none-any.whl.

File metadata

  • Download URL: difPy-4.1.1-py3-none-any.whl
  • Upload date:
  • Size: 15.2 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.11.2

File hashes

Hashes for difPy-4.1.1-py3-none-any.whl
Algorithm Hash digest
SHA256 4da1cef096b8b0463861b7551681d9ee44b7886cce512b2b9938c58d9b2f28b3
MD5 79461b9dd309f9558db5cbe4b38c8df4
BLAKE2b-256 e36d6cb5518ca4b1210898f7327ee69eff54903a3cc3c2deb7b07236531c505f

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page