ClearClutter

Non-destructive photo/video dedup for a messy drive, with a Google-Photos-style review UI. Scans a source tree into a SQLite index, puts every file into a group (duplicates together, unique files as groups of one), and gives you a daily verify queue. It never deletes anything — rejected copies move to <root>/ClearClutter_bin/ (folder structure preserved) for you to delete manually, every move is logged, and the last batch is always undoable.

Install

pip install clearclutter            # from PyPI, once published
# or from a checkout:
pip install .
# with face recognition ("People"):
pip install "clearclutter[faces]"

Usage

clearclutter scan /Volumes/WD       # index the drive (incremental)
clearclutter group /Volumes/WD      # build groups (dupes + singles)
clearclutter faces /Volumes/WD      # optional: cluster faces by person
clearclutter serve /Volumes/WD      # review UI at http://127.0.0.1:8177

The index lives on the drive itself — <root>/.clearclutter/ holds the SQLite database, thumbnails, and the move audit log, so every disk is fully self-contained and travels with its data. With several disks plugged in, run one clearclutter serve <root> per disk; each instance picks the next free port automatically and shows its drive name in the top bar.

Staying in sync

• On server start, a fast background probe (walk + stat, no hashing) counts files that are new, changed, or missing versus the index; the Sync button shows the number.
• Sync rescans incrementally (only new/changed files are hashed) and rebuilds groups, with live progress. New files land as unverified groups — straight into your review queue.
• Manually moved files are reconciled by content hash: if you move or rename a verified file outside the app, the rescan matches the "missing" row to the new path and carries its verified flag and tags over.
• Files that are truly gone (deleted or moved off-drive) show as broken links in the top bar with a one-click clean up (index rows only — no files are ever touched).

How grouping works

Names are never used for matching, so same-name-different-photo never groups. The suggested keeper (★) is scored by format (RAW > HEIC > JPEG > PNG), resolution, EXIF date, GPS, camera, and size — override with one click. Degrouping wrongly-matched photos splits them into their own manual group; never-pair rules and manual groups both survive regrouping.

The verify workflow

Every file is unverified until you deal with its group:

• Keep ★, bin rest — queues the duplicates for the bin and marks the whole group verified.
• Verify — marks a group correct as-is (singles, or groups where you keep everything).
• Bulk-select photos in any timeline and hit Verify in the action bar.

Groups (Unverified) in the left nav is your daily queue; the badge shows what's left. The Verified view shows everything you've already cleared.

The UI

Left nav: All Data · Photos · People · Location · Verified · Groups (Unverified) · Videos, plus your tags and the pending-actions queue. Timelines use a justified grid with date headers, hover-select circles, and a bulk action bar (verify / tag / move / bin). Clicking a photo opens a lightbox with arrow-key navigation and an info panel. People clusters faces (incremental — names survive re-runs). Location clusters GPS on a ~1 km grid. Light and dark theme follow the OS.

Applying moves: Pending actions → Dry run → Apply. Undo restores the last batch. Tags live in SQLite only — they're a semantic view, independent of folders.

Built for volume

Designed around a 1 TB / few-hundred-thousand-file library: one streaming read per file (hash computed while reading), incremental rescans skip unchanged files by size+mtime, thumbnails are sharded across 256 subdirectories, group statistics are precomputed at build time (the review list is a pure indexed read — no aggregation per page), timelines use an expression index on the capture date, singleton groups insert in bulk, and every list in the UI is paginated with infinite scroll. Scans and syncs run in a background thread with progress — the UI never blocks.

Notes

• Workspace (index, thumbnails, moves.jsonl): <root>/.clearclutter (--workspace to override)
• exiftool on PATH enables Live Photo pairing and video dates (brew install exiftool)
• phash threshold 6 is conservative; raise toward 10 to catch heavier edits
• NTFS on macOS needs Paragon/Tuxera for the apply step; scanning is read-only