Skip to main content

Duplicate Image Finder (difPy)

Tired of going through all images in a folder and comparing them manually to check if they are duplicates?

The Duplicate Image Finder (difPy) Python package automates this task for you!

Read more on how the algorithm of difPy works in my Medium article Finding Duplicate Images with Python.

For a detailed usage guide, please view the official difPy Usage Documentation.

Try the difPy Web App!


Description

difPy searches for images in one or more different folders, compares the images it found and checks whether these are duplicates. It then outputs the image files classified as duplicates as well as the images having the lowest resolutions, so you know which of the duplicate images are safe to be deleted. You can then either delete them manually, or let difPy delete them for you.

difPy does not compare images based on their hashes. It compares them based on their tensors i. e. the image content - this allows difPy to not only search for duplicate images, but also for similar images.

Basic Usage

To make difPy search for duplicates within one folder:

from difPy import dif
search = dif("C:/Path/to/Folder/")

To search for duplicates within multiple folders:

from difPy import dif
search = dif(["C:/Path/to/Folder_A/", "C:/Path/to/Folder_B/", "C:/Path/to/Folder_C/", ... ])

Folder paths can be specified as standalone Python strings, or within a list.

Output

difPy returns various types of output that you may use depending on your use case:

A JSON formatted collection of duplicates/similar images (i. e. match groups) that were found, where the keys are a randomly generated unique id for each image file:

search.result

> Output:
{20220819171549 : {"location" : "C:/Path/to/Image/image1.jpg",
                   "matches" : {30270813251529 : "location": "C:/Path/to/Image/matched_image1.jpg",
                                                 "mse": 0.0},
                               {72214282557852 : "location": "C:/Path/to/Image/matched_image2.jpg",
                                                 "mse": 0.0},
                   ... }
 ...
}

A list of duplicates/similar images that have the lowest quality among match groups:

search.lower_quality

> Output:
["C:/Path/to/Image/duplicate_image1.jpg", 
 "C:/Path/to/Image/duplicate_image2.jpg", ...]

A JSON formatted collection with statistics on the completed difPy process:

search.stats

> Output:
{"directory" : ("C:/Path/to/Folder_A/", "C:/Path/to/Folder_B/", ... ),
 "duration" : {"start_date" : "2023-02-15",
               "start_time" : "18:44:19",
               "end_date" : "2023-02-15",
               "end_time" : "18:44:38",
               "seconds_elapsed" : 18.6113},
 "fast_search" : True,
 "recursive" : True,
 "match_mse" : 0,
 "px_size" : 50,
 "files_searched" : 1032,
 "matches_found" : {"duplicates" : 52, 
                    "similar" : 0},
 "invalid_files" : {"count" : 4},
 "deleted_files" : {"count" : 0}}

Additional Parameters

difPy supports the following parameters:

dif(*directory, fast_search=True, recursive=True, similarity='duplicates', px_size=50, 
    move_to=None, show_progress=True, show_output=False, delete=False, silent_del=False, 
    logs=False)

CLI Usage

difPy can also be invoked through the CLI by using the following commands:

python dif.py -D "C:/Path/to/Folder/"

python dif.py -D "C:/Path/to/Folder_A/" "C:/Path/to/Folder_B/" "C:/Path/to/Folder_C/"

It supports the following arguments:

dif.py [-h] -D DIRECTORY [-Z OUTPUT_DIRECTORY] [-f {True,False}]
       [-r {True,False}] [-s SIMILARITY] [-px PX_SIZE] 
       [-p {True,False}] [-o {True,False}] [-mv MOVE_TO]
       [-d {True,False}] [-sd {True,False}] [-l {True,False}]

The output of difPy is then written to files and saved in the working directory, where "xxx" in the output filesnames is a unique timestamp:

difPy_results_xxx.json
difPy_lower_quality_xxx.csv
difPy_stats_xxx.json

For a detailed usage guide, please view the official difPy Usage Documentation.

Metadata

Release files for difPy 3.0.9

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for difPy 3.0.9
File Size Uploaded Provenance Actions
difPy-3.0.9.tar.gz 12.7 kB Without provenance Details for difPy-3.0.9.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for difPy 3.0.9
File Interpreter ABI Platform Actions
difPy-3.0.9-py3-none-any.whl Python 3 none any Details for difPy-3.0.9-py3-none-any.whl

Total release size: 23.0 kB

Release files / difPy-3.0.9.tar.gz

Download URL difPy-3.0.9.tar.gz
Size 12.7 kB
Tags Source
SHA-256 checksum
How to use checksums
baab0a535e85d8156ba92815616531cdae5a2c133720b8ac413e2f273506dd54
BLAKE2b-256 checksum
How to use checksums
20029ecbbf9ab9a7ba9c4c5162e7b4d9d21fd75e5e67121fd3ed75bed9999d67
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.11.2

Release files / difPy-3.0.9-py3-none-any.whl

Download URL difPy-3.0.9-py3-none-any.whl
Size 10.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
b87c5cdfe918cc51edcc4e35e02f0c11928d48efa27035f17793e974cfb31c6b
BLAKE2b-256 checksum
How to use checksums
4ec9d8196df223793e714f6252068fff2c81a6e72cc4ecf46c15e95e70d03fc9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.11.2

Security for difPy 3.0.9

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

3.0.9 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page