ClearClutter

Non-destructive photo/video dedup for a messy drive, with a Google-Photos-style review UI. Scans a source tree into a SQLite index, puts every file into a group (duplicates together, unique files as groups of one), and gives you a daily verify queue. By default it never deletes anything — rejected copies move to <root>/ClearClutter_bin/ (folder structure preserved) for you to delete manually, every move is logged, and the last batch is always undoable. (Permanent deletion exists as an explicit, confirmed opt-in.)

Install

pip install clearclutter            # from PyPI, once published
# or from a checkout:
pip install .
# with face recognition ("People"):
pip install "clearclutter[faces]"

Usage

clearclutter /Volumes/WD            # that's it — review UI at http://127.0.0.1:8177

Point it at a drive and everything else happens from the UI: Sync scans and groups (with live progress), and People → Detect faces runs face clustering. The individual commands still exist for scripting:

clearclutter scan /Volumes/WD       # index the drive (incremental)
clearclutter scan /Volumes/WD --kind photos   # or videos — skip the other kind's I/O
clearclutter group /Volumes/WD      # build groups (dupes + singles)
clearclutter faces /Volumes/WD      # optional: cluster faces by person
clearclutter serve /Volumes/WD      # review UI

Background jobs

Sync, apply and face detection run as background jobs: a panel in the left nav shows live progress with pause / resume / cancel, and you keep browsing and reviewing while they run. The bandwidth selector in the top bar (Full speed / Balanced / Gentle) caps how fast jobs read the disk, so a scan never makes the UI unusable on a slow drive.

The index lives on the drive itself — <root>/.clearclutter/ holds the SQLite database, thumbnails, and the move audit log, so every disk is fully self-contained and travels with its data. With several disks plugged in, run one clearclutter serve <root> per disk; each instance picks the next free port automatically and shows its drive name in the top bar.

Staying in sync

• On server start, a fast background probe (walk + stat, no hashing) counts files that are new, changed, or missing versus the index; the Sync button shows the number.
• Sync rescans incrementally (only new/changed files are hashed) and rebuilds groups, with live progress. New files land as unverified groups — straight into your review queue.
• The Sync scope selector (All media / Photos only / Videos only) limits what gets hashed, so a quick photo pass doesn't wait on gigabytes of video I/O. Missing-file detection still covers the whole drive either way.
• Manually moved files are reconciled by content hash: if you move or rename a verified file outside the app, the rescan matches the "missing" row to the new path and carries its verified flag and tags over.
• Files that are truly gone (deleted or moved off-drive) show as broken links in the top bar with a one-click clean up (index rows only — no files are ever touched).

How grouping works

Names are never used for matching, so same-name-different-photo never groups. The suggested keeper (★) is scored by format (RAW > HEIC > JPEG > PNG), resolution, EXIF date, GPS, camera, and size — override with one click. Degrouping wrongly-matched photos splits them into their own manual group; never-pair rules and manual groups both survive regrouping.

The verify workflow

Every file is unverified until you deal with its group:

• Keep ★, bin rest — queues the duplicates for the bin and marks the whole group verified.
• Keep ★, bin rest — all — the same, applied to every unverified group matching the current filters in one confirmed click. Filter to Exact + Photos and clear thousands of byte-identical duplicates at once (a dry-run count is shown before anything is queued; non-exact filters get an extra warning).
• Verify — marks a group correct as-is (singles, or groups where you keep everything).
• Bulk-select photos in any timeline and hit Verify in the action bar.

Groups (Unverified) in the left nav is your daily queue; the badge shows what's left. The Verified view shows everything you've already cleared.

The UI

Left nav: All Data · Photos · People · Location · Verified · Favourites · Groups (Unverified) · Videos, plus the bin, the pending-actions queue and the background-jobs panel. Timelines use a justified grid with date headers, hover-select circles, and a bulk action bar (verify / favourite / tag / move / bin — one request for the whole selection). Videos play on hover in the timeline and play in-card (muted, with scrubbing) in group review, so comparing near-identical clips doesn't mean bouncing through a full-screen player; large videos stream with HTTP range requests, so playback and seeking are instant. Clicking a photo opens a lightbox with arrow-key navigation and an info panel. People clusters faces (incremental — names survive re-runs), can be started from the UI, and clusters of the same person can be merged. Location clusters GPS on a ~1 km grid. Tags filter any timeline from a dropdown. Light and dark theme follow the OS.

Applying moves: Pending actions → Dry run → Apply — apply runs in the background with progress. Undo restores the last batch. If you'd rather skip the bin folder entirely, an explicit opt-in checkbox deletes binned files permanently (confirmed, and not undoable — the default remains non-destructive). Tags live in SQLite only — they're a semantic view, independent of folders.

Built for volume

Designed around a 1 TB / few-hundred-thousand-file library: one streaming read per file (hash computed while reading), incremental rescans skip unchanged files by size+mtime, thumbnails are sharded across 256 subdirectories, group statistics are precomputed at build time (the review list is a pure indexed read — no aggregation per page), timelines use an expression index on the capture date, singleton groups insert in bulk, and every list in the UI is paginated with infinite scroll. Scans and syncs run in a background thread with progress — the UI never blocks.

Notes

• Workspace (index, thumbnails, moves.jsonl): <root>/.clearclutter (--workspace to override)
• exiftool on PATH enables Live Photo pairing and video dates (brew install exiftool)
• phash threshold 6 is conservative; raise toward 10 to catch heavier edits
• NTFS on macOS needs Paragon/Tuxera for the apply step; scanning is read-only