Skip to main content

dupehunt

Find duplicate files by content hash. Zero dependencies, one command, works on Windows, macOS and Linux.

$ dupehunt ~/Downloads
3 group(s), 9 file(s), 412.6 MB reclaimable

[1] 180.0 MB x 3  (360.0 MB wasted)
    /home/me/Downloads/ubuntu.iso
    /home/me/Downloads/ubuntu (1).iso
    /home/me/Downloads/backup/ubuntu.iso

[2] 26.3 MB x 2  (26.3 MB wasted)
    ...

Install

pip install dupehunt

Usage

dupehunt [paths ...] [options]
Option Meaning
paths Files or directories to scan. Default: current directory
-m, --min-size Skip files smaller than this. 1K, 4M, 1G all accepted
-a, --algo blake2b (default), sha256, sha1, md5
-n, --top Show only the N largest groups
--json Machine readable output
--delete Delete extra copies, keeping the first path of each group
--yes Required together with --delete to actually delete

Examples

# biggest wins only, files above 10 MB
dupehunt ~/Pictures -m 10M -n 20

# feed the result to another tool
dupehunt ~/data --json > dupes.json

# preview what a cleanup would remove, without touching anything
dupehunt ~/data --delete -m 1M

# really delete the extra copies
dupehunt ~/data --delete --yes -m 1M

--delete without --yes is always a dry run: it prints the plan and removes nothing. Nothing is ever deleted unless both flags are present.

How it works

Three passes, cheapest first, so a full disk scan stays fast:

  1. Group files by exact size. Unique sizes drop out immediately.
  2. Hash the first 8 KiB of each remaining candidate.
  3. Hash the whole file only for candidates that still collide.

Symlinks are never followed, and hard links to an already seen inode are counted once, so the same physical data is not reported as duplicates of itself.

Use as a library

from pathlib import Path
from dupehunt import find_duplicates, total_wasted

groups = find_duplicates([Path(".")], n_min_size=1024 * 1024)
for group in groups:
    print(group.size, group.wasted, [str(p) for p in group.paths])
print(total_wasted(groups))

License

MIT

Metadata

Release files for dupehunt 0.0.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for dupehunt 0.0.1
File Size Uploaded Provenance Actions
dupehunt-0.0.1.tar.gz 7.2 kB Without provenance Details for dupehunt-0.0.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for dupehunt 0.0.1
File Interpreter ABI Platform Actions
dupehunt-0.0.1-py3-none-any.whl Python 3 none any Details for dupehunt-0.0.1-py3-none-any.whl

Total release size: 15.5 kB

Release files / dupehunt-0.0.1.tar.gz

Download URL dupehunt-0.0.1.tar.gz
Size 7.2 kB
Tags Source
SHA-256 checksum
How to use checksums
cb9cf05ea258f0f903ab5f423c05f941bb8860362944fb66134739ed7f254ca3
BLAKE2b-256 checksum
How to use checksums
cd2d0eb661e558fb51b6bc276cecb5d0f073920a87e679e8fe761c0864d81023
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.10.11

Release files / dupehunt-0.0.1-py3-none-any.whl

Download URL dupehunt-0.0.1-py3-none-any.whl
Size 8.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
1f51fe99827d306c798fdc851ba768316f45ac91a88191441c1a1642b05522c0
BLAKE2b-256 checksum
How to use checksums
1f4192f5979452e4141a193de8d4c97fd44d25ad40577ade509b7d87fd3ce508
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.10.11

Security for dupehunt 0.0.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page