VisualDedup 🔍✨

VisualDedup is a high-performance Python toolkit designed to find exact and visually similar images in large collections, cluster them into duplicate groups, explain why they match, recommend which copy to retain, and safely clean datasets without destructive surprises.

Existing libraries such as imagededup and Perception provide basic perceptual hashing. VisualDedup goes beyond basic hashing to solve the real operational challenges of large-scale dataset management:

1. Explains Why Images Match: Classifies transformations (resizing, JPEG recompression, format conversion, and cropping).
2. Sub-Quadratic Indexing: Uses BK-Trees and Multi-Index Hamming Hashing to eliminate the $O(N^2)$ pairwise comparison bottleneck, scaling to hundreds of thousands of images.
3. Intelligent Keeper Recommendation: Recommends the optimal file to retain based on resolution, lossless format preservation, compression quality, and original timestamps.
4. Safe Dataset Cleanup: Non-destructive by default with strict dry-run modes, quarantine isolation, and full JSON audit logs.
5. Interactive Visual Reports: Generates standalone HTML dashboards and image contact sheets with side-by-side previews, difference metrics, and keeper badges.
6. Incremental SQLite Cache: Avoids recomputing hashes on subsequent scans by tracking file modification times and byte sizes.

⚡ Quickstart

1. Installation

# Clone the repository
git clone https://github.com/your-org/visualdedup.git
cd visualdedup

# Create virtual environment and install
python -m venv .venv
# Windows:
.venv\Scripts\activate
# Linux/macOS:
source .venv/bin/activate

pip install -e ".[dev]"

💻 Command-Line Interface (CLI)

Scan a Folder and Generate Reports

# Scan with pHash (DCT) and export JSON + interactive HTML reports
visualdedup scan ./photos --output report.json --html report.html

# Scan with custom Hamming threshold and contact sheets
visualdedup scan ./dataset --hash phash --threshold 8 --contact-sheet sheet.png

# Fast candidate indexing with Multi-Index Hashing for large collections
visualdedup scan ./large_collection --index multi_index --keeper best_quality

# Output structured tabular CSV for data pipelines / Excel
visualdedup scan ./dataset --csv report.csv

# Output pure pipeable JSON directly to stdout
visualdedup scan ./dataset --json | jq .

# Store duplicate images in a separate folder organized by cluster subfolders
visualdedup scan ./dataset --store-duplicates ./duplicates_output

# Move duplicate discards into a separate folder (cleaning the source directory)
visualdedup scan ./dataset --move-duplicates ./quarantine_folder

Inspect Two Images Side-by-Side

# Terminal table view:
visualdedup inspect original.png compressed_downscaled.jpg

# Structured JSON view for scripts and CI:
visualdedup inspect original.png compressed_downscaled.jpg --json

Outputs a detailed comparison table with SHA-256, dHash, pHash, aHash, wHash distances, resolution differences, and an automated diagnosis (e.g. Resized copy: downscaled ~2.0x; JPEG recompression).

Safe Dataset Cleanup

# Dry run simulation (default: does not touch files)
visualdedup clean ./dataset --mode quarantine --dry-run

# Execute quarantine move with confirmation
visualdedup clean ./dataset --mode quarantine --target-dir ./quarantine --confirm

🐍 Python Library API

from visualdedup import VisualDedup, KeeperStrategy, CleanMode

# 1. Initialize engine with incremental caching and sub-quadratic indexing
vd = VisualDedup(
    cache_path=".visualdedup_cache.db",
    hash_type="phash",       # "phash", "dhash", "ahash", "whash", "exact"
    threshold=8,             # Max Hamming distance
    index_type="bktree",     # "bktree" or "multi_index"
)

# 2. Scan folders
clusters, summary = vd.scan(["./dataset"], recursive=True)

print(f"Scanned {summary.total_scanned} files.")
print(f"Found {summary.total_clusters} duplicate clusters.")
print(f"Reclaimable space: {summary.reclaimable_bytes / (1024 * 1024):.2f} MB")

# 3. Inspect keeper recommendations
for cluster in clusters:
    print(f"\nCluster {cluster.cluster_id} ({cluster.match_category}):")
    print(f"  Keeper: {cluster.keeper_decision.keeper_path}")
    print(f"  Reason: {cluster.keeper_decision.reason}")
    print(f"  Discards: {cluster.keeper_decision.discard_paths}")

# 4. Export visual reports
vd.export_html(clusters, summary, "report.html")
vd.export_json(clusters, summary, "report.json")

# 5. Execute safe cleanup (e.g. quarantine mode with dry run)
clean_result = vd.clean(
    clusters,
    mode=CleanMode.QUARANTINE,
    target_dir="./quarantine_folder",
    dry_run=True,
    audit_log_path="audit.json"
)

🚀 Performance & Sub-Quadratic Indexing

Naive deduplication compares every image pair ($O(N^2)$ comparisons). For 5,000 images, that requires 12.5 million comparisons.

VisualDedup implements metric BK-Trees and Multi-Index Hamming Hashing (Pigeonhole block search):

Run the benchmarks yourself:

python benchmarks/benchmark_indexing.py

🛡️ Keeper Selection Strategies

🧪 Testing

pytest -v

All 22 unit and integration tests validate hashing invariance, tree search pruning, cluster disjoint sets, transformation explanations, safe deletion guards, and report exports.

📄 License

VisualDedup is distributed under the MIT License.