About Duplicates Finder

Duplicates Finder is a simple Python package that identifies duplicate files in and across folders. There are three ways to search for identical files:

1. List all duplicate files in a folder of interest.
2. Pick a file and find all duplications in a folder.
3. Directly compare two folders against each other.

The results are saved as a Pandas Dataframe or can be exported as .csv files. More information about the underlying concept can also be found in this short article.

Installation

You can either clone the repository directly from the Github webpage or run the following command(s) in your terminal:

Pip Installation:

pip install duplicate-finder

Alternatively you can clone the Git repository:

git clone https://github.com/akcarsten/duplicates.git

Then go to the folder to which you cloned the repository and run:

python setup.py install

Now you can run Python and import the Bitfinex client.

Examples of how to use the package

Example 1: List all duplicate files in a folder of interest.

import duplicates as dup


folder_of_interest = 'C:/manyDuplicatesHere/'
dup.list_all_duplicates(folder_of_interest, to_csv=True, csv_path='C:/csvWithAllDuplicates/', fastscan=True)

Here the fastscan parameter is set to True (default is false). By doing so a pre-selection of potential duplicate files is performed based on the file size. If only a specific type of files is of interest this can be further defined by the 'ext' parameter. For example:

df = dup.list_all_duplicates(folder_of_interest, to_csv=True, csv_path='C:/csvWithAllDuplicates/', ext='.jpg')

Example 2: Pick a file and find all duplications in a folder.

import duplicates as dup


file_of_interest = 'C:/manyDuplicatesHere/thisFileExistsManyTimes.jpg'
folder_of_interest = 'C:/manyDuplicatesHere/'
df = dup.find_duplicates(file_of_interest, folder_of_interest)

Example 3: Directly compare two folders against each other.

import duplicates as dup


folder_of_interest_1 = 'C:/noDuplicatesHere/'
folder_of_interest_2 = 'C:/noDuplicatesHereAsWell/'
df = dup.compare_folders(folder_of_interest_1, folder_of_interest_2)

As in Example 1 above a specific filetype can be selected and the results can be written to a .csv file.