kdeldycke/mail-deduplicate

What is Mail Deduplicate?

Provides the mdedup CLI, an utility to deduplicate mails from a set of boxes.

Features

• Duplicate detection based on cherry-picked and normalized mail headers.
• Fetch mails from multiple sources.
• Reads and writes to mbox, maildir, babyl, mh, mmdf and eml formats.
• Deduplication strategies based on size, content, timestamp, file path or random choice, chainable as fallbacks.
• Copy, move or delete the resulting set of duplicates.
• Dry-run mode.
• Protection against false-positives with safety checks on size and content differences.
• Supports macOS, Linux and Windows.
• Standalone executables for Linux, macOS and Windows.
• Shell auto-completion for Bash, Zsh and Fish.

Installation

All installation methods are available in the documentation. Below are the most popular ones:

Try it now

uv is the fastest way to run mdedup on any platform, thanks to its uvx command:

$ uvx --from mail-deduplicate -- mdedup

macOS

mdedup is part of the official Homebrew default tap, so you can install it with:

$ brew install mail-deduplicate

Executables

Standalone binaries of mdedup's latest version are available as direct downloads for several platforms and architectures:

Quickstart

Duplicate mails pile up whenever boxes get copied around: backups of the same account taken at different times, per-folder exports where a mail tagged with several labels lands in several boxes, archives consolidated from multiple clients, or IMAP synchronizations gone wrong.

mdedup cleans this up. Point it at your boxes: it groups copies of the same mail by hashing a curated set of headers, applies a --strategy to pick which copy to keep in each group, then performs an --action on the result. The default action is the safest one: sources are only read, and the deduplicated selection is written to a brand new box.

So to merge two overlapping archives into a single clean box:

$ mdedup --strategy select-one --export merged.mbox archive-2024.mbox archive-2025.mbox

Mails found in both archives are copied once into merged.mbox, along with all the mails that were unique to each source.

To remove duplicates in place instead, switch to a destructive action, and rehearse with --dry-run:

$ mdedup --strategy select-one --action delete-discarded --dry-run ~/Maildir

The hands-on tutorial builds a small playground of duplicated mails, then walks through strategies, actions and safeguards on it. The design page explains how duplicates are detected.

[!WARNING] Performance and memory usage: mdedup implementation is quite naive and everything resides in memory.

If this is good enough for a volume of a couple of gigabytes, the more emails mdedup try to parse, the closer you'll reach the memory limits of your machine. In which case mdedup will exit abruptly, zapped by the OOM killer of your OS. Of course your mileage may vary depending on your hardware.

You can influence implementation of this feature with pull requests, purchasing business support 🤝 and sponsorship 🫶.