duphunt

Find duplicate files by content — anywhere, with nothing to install. The great duplicate finders (fdupes, jdupes, rdfind, fclones) are native binaries you have to brew/apt/cargo install first — which you can't always do on a locked-down box, a colleague's laptop, a CI runner, or a container. duphunt runs the moment you have Python or Node: pip install duphunt or npx duphunt .. Zero dependencies, no network.

pip install duphunt

$ duphunt ~/Downloads

2 duplicate group(s), 5 files, 8.1 MB reclaimable

  4.1 MB × 2   4.1 MB reclaimable
    /Users/me/Downloads/invoice.pdf
    /Users/me/Downloads/invoice (1).pdf

  2.0 MB × 3   4.0 MB reclaimable
    /Users/me/Downloads/clip.mp4
    /Users/me/Downloads/clip-copy.mp4
    /Users/me/Downloads/old/clip.mp4

Groups are sorted biggest-waste-first, so the files worth deleting are at the top.

This is the Python build. A result-equivalent Node build is on npm: npx duphunt (https://github.com/jjdoor/duphunt).

How it works

1. Group by size. Two files of different sizes can't be identical, so files with a unique size are never even read.
2. Hash the collisions. Within each size group, each file is SHA-256 hashed (streamed in 64 KB chunks, so multi-GB files don't blow up memory).
3. Report identical content. Files with the same hash are true byte-for-byte duplicates, grouped and ranked by reclaimable space.

It reports — it never deletes. You decide what to remove.

Usage

duphunt                      # scan the current directory
duphunt ~/Downloads ~/Desktop   # scan several roots at once
duphunt a.jpg b.jpg c.jpg    # or just compare specific files
duphunt . --json             # machine-readable
duphunt . --min-size 1048576 # ignore files under 1 MB
duphunt . --exit-code        # exit 1 if any duplicates exist (CI gate)

Options

Notes

• Empty files are skipped by default (they all hash alike and are rarely what you mean); pass --min-size 0 to include them.
• Symlinks are skipped unless --follow, so a symlinked tree won't be double-counted or loop forever.
• Each physical file is counted once. Repeated or overlapping roots and symlink aliases (even under --follow) are de-duplicated by real path, so they never inflate the results — while genuine hard links still surface.
• Same tool, two builds. The Python and Node builds hash with SHA-256 and produce identical results — use whichever your environment already has.

--json shape

{
  "groups": [
    { "hash": "9f86d0…", "size": 4300000, "count": 2, "wasted": 4300000,
      "paths": ["/a/invoice.pdf", "/b/invoice (1).pdf"] }
  ],
  "summary": { "groups": 1, "files": 2, "wasted": 4300000 }
}

Exit codes

By default duphunt is a viewer and exits 0; add --exit-code to gate a pipeline on it.

License

MIT