mail-deduplicate 9.2.0

What is Mail Deduplicate?

Provides the mdedup CLI, an utility to deduplicate mails from a set of boxes.

Features

• Duplicate detection based on cherry-picked and normalized mail headers.
• Fetch mails from multiple sources.
• Reads and writes to mbox, maildir, babyl, mh, mmdf and eml formats.
• Deduplication strategies based on size, timestamp, file path or random choice, chainable as fallbacks.
• Copy, move or delete the resulting set of duplicates.
• Dry-run mode.
• Protection against false-positives with safety checks on size and content differences.
• Supports macOS, Linux and Windows.
• Standalone executables for Linux, macOS and Windows.
• Shell auto-completion for Bash, Zsh and Fish.

Installation

All installation methods are available in the documentation. Below are the most popular ones:

Try it now

uv is the fastest way to run mdedup on any platform, thanks to its uvx command:

$ uvx --from mail-deduplicate -- mdedup

macOS

mdedup is part of the official Homebrew default tap, so you can install it with:

$ brew install mail-deduplicate

Executables

Standalone binaries of mdedup's latest version are available as direct downloads for several platforms and architectures:

Quickstart

Duplicate mails pile up whenever boxes get copied around: backups of the same account taken at different times, per-folder exports where a mail tagged with several labels lands in several boxes, archives consolidated from multiple clients, or IMAP synchronizations gone wrong.

mdedup cleans this up. Point it at your boxes: it groups copies of the same mail by hashing a curated set of headers, applies a --strategy to pick which copy to keep in each group, then performs an --action on the result. The default action is the safest one: sources are only read, and the deduplicated selection is written to a brand new box.

So to merge two overlapping archives into a single clean box:

$ mdedup --strategy select-one --export merged.mbox archive-2024.mbox archive-2025.mbox

Mails found in both archives are copied once into merged.mbox, along with all the mails that were unique to each source.

To remove duplicates in place instead, switch to a destructive action, and rehearse with --dry-run:

$ mdedup --strategy select-one --action delete-discarded --dry-run ~/Maildir

The hands-on tutorial builds a small playground of duplicated mails, then walks through strategies, actions and safeguards on it. The design page explains how duplicates are detected.

[!NOTE] Performance and memory usage: each mail is reduced to a lightweight stub as soon as it is hashed, and its content is re-read from its source box whenever a later step needs it again. The memory footprint no longer grows with the size of the boxes: deduplicating a 215 MB maildir of 1,500 mails peaks at 48 MB of resident memory.

It still scales with the number of mails, at a few hundred bytes each, so a large enough collection can reach the memory limits of your machine. In which case mdedup will exit abruptly, zapped by the OOM killer of your OS. Re-reading mail content also costs extra disk I/O. Of course your mileage may vary depending on your hardware.

You can influence implementation of this feature with pull requests, purchasing business support 🤝 and sponsorship 🫶.