Remove Duplicates

Utility to recursive search and move/delete duplicate files of the same size and context in specified folder.

If the storage folder (recycle_bin) is not specified, then duplicate files will be deleted! If the number of command line parameters is zero, then the search folder = current working folder.

Installation

Requires Python 3.8+.

pip install remove-duplicates

After installation use rmdup --help instead of remove_dup.

Command line parameters

-st, --start_folder     Folder with which the recursive search begins.
-rb, --recycle_bin      Optional folder for storing duplicate files (move file).
-log, --log_file        Log file name.
-fnp, --fn_pattern      File name pattern. Only files matching the pattern are processed!
                        Provides support for Unix shell-style wildcards. Default value is "*"
-nr, --not_recursively  If this parameter is set, then recursive search is disabled!

Call example

rmdup --start_folder=E:\YoutubeChannelsCopy --recycle_bin=E:\reserved --log_file=E:\reserved\logfile.txt --fn_pattern="*.png"

How it works

• Files are enumerated recursively (unless -nr is set), only files matching the file name pattern are considered.
• Files of equal size are compared by content (filecmp with shallow=False), so files with the same size and modification time but different content are never treated as duplicates.
• Found duplicate copies are either moved to the storage folder (if -rb is given) or deleted.
• If the storage folder is located inside the search folder, it is skipped during the search.
• If a file cannot be accessed (permissions, race), it is skipped with a warning and the rest of the files are still processed.

Work log

Internationalization

Most of the program messages are translated into five languages: English, Russian, German, Spanish, French. If the language cannot be detected (e.g. C/POSIX locale), English is used by default. See: https://github.com/octaprog7/PyInternalization.

Development

Run the tests (unittest, no external dependencies):

python3 -m unittest discover -s tests -v

PyPi

https://pypi.org/project/remove-duplicates/