viddup 0.1.0

A fast, local video duplicate detector

VidDup 🎬

A fast, local video duplicate detector — finds re-encoded, re-scaled, and compressed copies that other tools miss.

---

Why VidDup?

Most duplicate detectors only find byte-for-byte identical files. VidDup goes further:

Scenario	dupeGuru	Czkawka	VidDup
Exact copy (same file)	✅	✅	✅
Re-encoded (H.264 → HEVC)	❌	❌	✅
Re-scaled (1080p → 720p)	❌	❌	✅
Compressed (high → low bitrate)	❌	❌	✅
Incremental cache (skip unchanged files)	❌	❌	✅
Interactive HTML report with thumbnails	❌	❌	✅

VidDup uses perceptual frame hashing (pHash) across multiple sampled frames per video, with a low-variance frame filter to ignore black/fade frames that cause false positives.

---

Quick Start

# Install system dependency
brew install ffmpeg          # macOS
# sudo apt install ffmpeg   # Ubuntu/Debian

# Install VidDup
pip install viddup

# Scan a directory
viddup scan ~/Movies

That's it. A terminal report is printed and an interactive HTML report (with video thumbnails) is automatically saved to the scanned directory and opened in your browser.

---

Features

• Three-layer detection
  • L1: xxHash3-128 — instant exact-copy detection
  • L2: Duration metadata — groups candidates, skips impossible pairs
  • L3: 10-frame pHash — perceptual similarity across re-encodes and re-scales
• Robust pHash comparison — uses median Hamming distance (not average), resistent to black frames, fade-outs, and title cards
• Smart frame selection — automatically skips solid-color frames and retries with alternate timestamps
• Incremental SQLite cache — fingerprints are cached; re-scanning a library of 500 videos takes seconds after the first run
• Multi-directory scan — finds duplicates across multiple folders in one pass
• .viddup_ignore — exclude directories and files with glob patterns, like .gitignore
• Hardware acceleration — automatically uses VideoToolbox on macOS (Apple Silicon & Intel)
• Interactive HTML report — embedded thumbnails, one-click delete-script generation, copy to clipboard

---

Installation

Requirements

• Python 3.11+
• ffmpeg (system binary)

# macOS
brew install ffmpeg
pip install viddup

# Ubuntu / Debian
sudo apt install ffmpeg
pip install viddup

# From source (development)
git clone https://github.com/your-username/viddup.git
cd viddup
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

---

Usage

viddup scan — find duplicates

# Basic scan
viddup scan ~/Movies

# Scan multiple directories (finds cross-folder duplicates)
viddup scan ~/Movies ~/Downloads/Videos

# Stricter threshold (only report very similar videos)
viddup scan ~/Movies --threshold 0.92

# Looser duration filter (catches clips with trimmed intros)
viddup scan ~/Movies --duration-tol 0.15

# Preview what would be scanned (no DB writes)
viddup scan ~/Movies --dry-run

# Save report to a specific directory
viddup scan ~/Movies --output ~/Desktop

# Don't auto-open the browser
viddup scan ~/Movies --no-open

# Force re-fingerprint everything (ignore cache)
viddup scan ~/Movies --no-cache

Full options:

Options:
  -t, --threshold FLOAT         Similarity threshold (0–1). Default: 0.85
  -f, --frames INTEGER          Frames sampled per video. Default: 10
  --duration-tol FLOAT          Duration tolerance for pre-filter. Default: 0.05 (±5%)
  --db PATH                     Fingerprint database path. Default: ~/.viddup/fingerprints.db
  -w, --workers INTEGER         Parallel worker processes. Default: CPU count
  -o, --output PATH             Report output directory. Default: first scanned directory
  --no-cache                    Ignore cache, recompute all fingerprints
  --recursive / --no-recursive  Recurse into subdirectories. Default: on
  --dry-run                     List files to be processed without writing DB
  --html / --no-html            Generate interactive HTML report. Default: on
  --open / --no-open            Auto-open HTML report in browser. Default: on
  -v, --verbose                 Show per-file progress

viddup status — cache info

viddup status
# Shows: cached video count, orphan records, database size, last scan time

viddup clear — clean up cache

# Remove records whose files no longer exist
viddup clear --orphans-only

# Wipe the entire cache
viddup clear --confirm

---

.viddup_ignore

Place a .viddup_ignore file in any scan directory to exclude files or subdirectories:

# Lines starting with # are comments

# Exclude directories (trailing slash required)
BRaw/
原始素材/
proxies/

# Exclude by filename glob
temp_*
*.tmp
._*

# Exclude by relative path glob
Backups/**
archive/2020/**

VidDup reports how many files were filtered when an ignore file is active.

---

How It Works

Each video goes through three layers:

 ┌─────────────────────────────────────────────────┐
 │  L1 — File Hash (xxHash3-128)                    │
 │  Identical bytes → immediate match, skip L2/L3  │
 └──────────────────────────┬──────────────────────┘
                            │ not exact
 ┌──────────────────────────▼──────────────────────┐
 │  L2 — Metadata Pre-filter (ffprobe)              │
 │  Duration difference > 5% → skip pair            │
 │  Reduces O(n²) comparisons to small groups       │
 └──────────────────────────┬──────────────────────┘
                            │ duration match
 ┌──────────────────────────▼──────────────────────┐
 │  L3 — Perceptual Frame Hash (pHash)              │
 │  Sample 10 frames at 5%, 15%, ..., 95%           │
 │  Skip low-variance frames (black/solid color)    │
 │  Compute pHash per frame → compare sequences     │
 │  Median Hamming distance < threshold → duplicate │
 └─────────────────────────────────────────────────┘

"Suggest keep" priority (shown in HTML report):

1. Highest resolution (width × height)
2. Largest file size (less compression loss)
3. Alphabetical path (deterministic tiebreak)

---

HTML Report

After scanning, VidDup generates a self-contained viddup_report_*.html file saved directly in the scanned directory. No server required — open it with any browser.

The report includes:

• Video thumbnails (extracted from the middle of each file, embedded as base64)
• Grouped duplicate cards sorted by reclaimable space
• "Suggest keep" badge for the recommended file in each group
• Checkboxes to select files for deletion (non-suggested files pre-selected)
• Generate delete script → shell commands you can review and execute in your terminal
• Copy to clipboard button

⚠️ VidDup never deletes files automatically. You always review the generated rm commands before executing them.

---

Performance

Fingerprinting speed is limited by ffmpeg frame extraction (I/O + decode) and scales linearly with video count. The main bottleneck is the first scan of a new library — subsequent scans reuse the SQLite cache and complete in seconds regardless of library size.

Tips for faster scans:

• macOS: VideoToolbox hardware decoding is enabled automatically — no action needed.
• Reduce --frames (e.g. --frames 6) for a 40% speedup with minimal accuracy loss on long videos.
• Increase --workers on machines with many CPU cores.
• Use --duration-tol conservatively — a tighter window means fewer L3 comparisons.

---

Configuration Reference

Setting	Default	Notes
--threshold	0.85	Lower = more matches (more false positives). Try 0.90–0.95 for stricter.
--frames	10	More frames = slower but more accurate. 6 is usually sufficient for short clips.
--duration-tol	0.05	Increase to 0.10–0.15 if you expect trimmed intros/outros.
--workers	CPU count	Reduce if your system slows down during scanning.

---

Limitations

• No semantic understanding: VidDup cannot detect if two videos are "about" the same topic but visually different (e.g., two different recordings of the same lecture).
• Duration pre-filter: Videos differing by more than --duration-tol will not be compared at L3, even if they are perceptually similar. Increase the tolerance if needed.
• Corrupt files: Videos that ffprobe cannot parse are skipped and reported in the terminal output.

---

Contributing

Contributions are welcome! See CONTRIBUTING.md for guidelines.

git clone https://github.com/Pengfei-Kou/viddup.git
cd viddup
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest          # run tests
ruff check .    # lint

Roadmap

• v0.3: BK-tree for large libraries (10,000+ videos), audio fingerprint verification, .viddup_ignore negation patterns
• v1.0: PyQt desktop GUI option, PyPI publishing, Windows testing

---

License

MIT © VidDup Contributors