mail-deduplicate 9.6.0

What is Mail Deduplicate?

Provides the mdedup CLI, an utility to deduplicate mails from a set of boxes.

Features

• Duplicate detection based on cherry-picked and normalized mail headers.
• Presets: a conservative one that only treats identical mails as copies, to run on every box before tuning anything, and an aggressive one that trusts Message-ID alone.
• Fetch mails from multiple sources.
• Reads and writes to mbox, maildir, babyl, mh, mmdf and eml formats.
• Deduplication strategies based on size, timestamp, file path or quantity, chainable as fallbacks.
• Copy, move, delete or hardlink the resulting set of duplicates.
• Dry-run mode.
• Protection against false-positives with safety checks on size and content differences. Two mails sharing every hashed header but not their body are left alone, while their real duplicate is acted on, here by mdedup --strategy discard-matching-path --regexp old --action delete-discarded --dry-run new.mbox old.mbox:
• Optional cache reusing the hashes computed by previous runs.
• Parallel hashing and selection over several processes.
• Supports macOS, Linux and Windows.
• Standalone executables for Linux, macOS and Windows.
• Shell auto-completion for Bash, Zsh and Fish.

Installation

All installation methods are available in the documentation. Below are the most popular ones:

Try it now

uv is the fastest way to run mdedup on any platform, thanks to its uvx command:

$ uvx --from mail-deduplicate -- mdedup

macOS

mdedup is part of the official Homebrew default tap, so you can install it with:

$ brew install mail-deduplicate

Executables

Standalone binaries of mdedup's latest version are available as direct downloads for several platforms and architectures:

Quickstart

Duplicate mails pile up whenever boxes get copied around: backups of the same account taken at different times, per-folder exports where a mail tagged with several labels lands in several boxes, archives consolidated from multiple clients, or IMAP synchronizations gone wrong.

mdedup cleans this up. Start with a conservative pass over all your boxes: --preset conservative treats as copies only the mails whose bytes are identical, keeps one copy of each, and writes every mail that survives into a brand new box. The sources are only read, so there is nothing to rehearse:

$ mdedup --preset conservative --export merged.mbox 2024.mbox 2025.mbox

Copies that differ somewhere, by a header a mailing list added or a Re: prefix, are left in place. mdedup catches those with its default hashing: it groups copies of the same mail by hashing a curated set of normalized headers, applies a --strategy to pick which copy to keep in each group, then performs an --action on the result. The default action is the safest one: sources are only read, and the deduplicated selection is written to a brand new box.

So to merge the same two archives into a single clean box, keeping the copy each mail has in the older one:

$ mdedup --strategy select-matching-path --regexp 2024 --export merged.mbox 2024.mbox 2025.mbox

Mails found in both archives are copied once into merged.mbox, along with all the mails that were unique to each source. Backups usually leave copies byte-identical, which no date or size strategy splits apart: select-one then keeps one copy of each mail, the one whose path sorts first.

To remove duplicates in place instead, switch to a destructive action, and rehearse with --dry-run:

$ mdedup --strategy select-matching-path --regexp 2024 --action delete-discarded --dry-run 2024.mbox 2025.mbox

The trail names each mail the run would delete, and alters nothing.

The hands-on tutorial builds a small playground of duplicated mails, then walks through strategies, actions and safeguards on it. The design page explains how duplicates are detected.

You can support this project with pull requests, business support 🤝 and sponsorship 🫶.