dupefinder

dupefinder is a small, zero-dependency Python library and CLI tool for finding duplicate files using content hashes.

Requires Python 3.10 or later. Detects exact duplicates (identical byte content) only.

Features

• Simple: one function for common use, a full report for advanced use.
• Safe by default: read-only. Never deletes, moves, or modifies files.
• Zero dependency: uses only the Python standard library.
• Modular: each responsibility lives in its own module.
• Memory-friendly: files are hashed in configurable chunks, not loaded fully into RAM.
• Fast: groups by file size before hashing — only candidates are hashed.
• Typed: ships with inline type annotations.
• Observable: typed event system for progress callbacks and integrations.
• Cancellable: abort scans via a callback or timeout.
• Cached: optional SQLite hash cache for repeated scans.

Installation

pip install dupefinder

For development:

git clone https://github.com/igors93/dupefinder.git
cd dupefinder
pip install -e ".[dev]"

Quick start

As a library

from dupefinder import find_duplicates

groups = find_duplicates("./Downloads")

for group in groups:
    print(f"{group.count} duplicate files — {group.size} bytes each")
    for path in group.files:
        print(f"  {path}")

Full report

from dupefinder import scan
from dupefinder.models import ScanOptions
from dupefinder.report import format_report

report = scan(
    "./Downloads",
    options=ScanOptions(
        min_size=1024,          # ignore files smaller than 1 KB
        ignore_hidden=True,     # skip dotfiles and dotfolders
        follow_symlinks=False,  # safe default
    ),
)

print(format_report(report))
print(f"Wasted space: {report.total_wasted_space} bytes")

JSON output

from dupefinder import scan
from dupefinder.report import report_to_json

report = scan("./Downloads")
print(report_to_json(report))

DupeFinder with events

from dupefinder import DupeFinder, ScanOptions

def on_event(event):
    if event.type == "file_discovered":
        print(f"\rFound {event.scanned_files} files...", end="", flush=True)
    elif event.type == "issue":
        print(f"\nWarning: {event.message}")
    elif event.type == "scan_completed":
        print(f"\nDone in {event.elapsed_seconds:.2f}s")

finder = DupeFinder(
    options=ScanOptions(min_size=1024),
    on_event=on_event,
)
report = finder.scan("./Downloads")

Progress callback

from dupefinder import DupeFinder, ScanOptions

def on_progress(progress):
    print(f"[{progress.phase}] {progress.scanned_files} files scanned, "
          f"{progress.hashed_files}/{progress.total_candidates} hashed")

finder = DupeFinder(
    options=ScanOptions(min_size=1024),
    on_progress=on_progress,
)
report = finder.scan("./Downloads")
print(f"Total bytes read: {report.total_bytes_read:,}")

Cancellation

import threading
from dupefinder import DupeFinder

cancel_flag = threading.Event()
threading.Timer(5.0, cancel_flag.set).start()  # cancel after 5 seconds

finder = DupeFinder(should_cancel=cancel_flag.is_set)
report = finder.scan("./Downloads")

if report.cancelled:
    print(f"Cancelled after {report.elapsed_seconds:.2f}s — partial results")

SQLite cache

from dupefinder import DupeFinder
from dupefinder.cache import SQLiteHashCache

with SQLiteHashCache(".dupefinder-cache.sqlite") as cache:
    finder = DupeFinder(cache=cache)
    report = finder.scan("./media")  # second run will be much faster

Scan limits

from dupefinder import DupeFinder, ScanOptions

finder = DupeFinder(options=ScanOptions(
    max_files=1000,       # stop after 1000 files
    max_depth=3,          # scan at most 3 levels deep
    timeout_seconds=30.0, # stop after 30 seconds
))
report = finder.scan("./data")

CLI

# Basic scan
dupefinder ./Downloads

# JSON output
dupefinder ./Downloads --json

# Ignore files smaller than 1 MB
dupefinder ./Downloads --min-size 1MB

# Only scan images
dupefinder ./Pictures --include-ext .jpg,.jpeg,.png,.webp

# Ignore temp and log files
dupefinder . --ignore-ext .tmp,.log

# Exit with code 2 if any duplicates are found (useful in scripts/CI)
dupefinder . --fail-on-duplicates

# Strict mode: raise errors instead of skipping inaccessible files
dupefinder . --strict

# Follow symbolic links (disabled by default)
dupefinder . --follow-symlinks

Run dupefinder --help to see all options.

Exit codes

Notes:

• --strict turns otherwise non-fatal file access errors into exit code 1.
• JSON output can contain an issues list even when the exit code is 0.
• Exit code 3 takes priority over --fail-on-duplicates.

CLI reference

API summary

JSON schema

All JSON output includes a schema_version field (currently "1.1") for forward compatibility.

See docs/api.md for the full reference.

Project structure

src/dupefinder/
├── api.py        public functions: scan, find_duplicates
├── cli.py        terminal command
├── engine.py     DupeFinder class: events, cancellation, cache
├── events.py     ScanEvent dataclass
├── cache.py      HashCache protocol and SQLiteHashCache
├── scanner.py    file discovery (os.scandir, loop detection, max_depth)
├── hashing.py    chunked file hashing with optional cache
├── grouping.py   group by size then by hash
├── filters.py    ignore/include rules
├── models.py     frozen dataclasses
├── report.py     text and JSON output
├── safety.py     path/options validation, helpers
├── constants.py  default values
└── errors.py     custom exceptions

Safety and design constraints

dupefinder is intentionally read-only:

• Does not delete, move, or rename files.
• Does not connect to the internet.
• Does not follow symbolic links by default. Pass --follow-symlinks or ScanOptions(follow_symlinks=True) to opt in.
• Reads files in chunks — no large allocations.
• Permission errors are recorded and skipped by default.
• SQLite cache writes occur only when the user explicitly passes --cache PATH or constructs SQLiteHashCache. The cache file is written to the path chosen by the user.
• Detects exact duplicates only — files with identical byte content. Near-duplicates, similar images, or renamed files are not detected.

See SECURITY.md for more details.

Running tests

pytest

With coverage:

pytest --cov=dupefinder --cov-report=term-missing

Contributing

Contributions are welcome. Please open an issue first to discuss what you want to change.

1. Fork the repository.
2. Create a branch: git checkout -b feature/your-feature.
3. Make your changes and add tests.
4. Run pytest and make sure all tests pass.
5. Open a pull request.

License

MIT — Igor Souza