mailvault -- Back up and archive email from IMAP and Microsoft 365 mailboxes

[!IMPORTANT] 0.8.0 changes how an archive stores its metadata. The SQLite database that used to live inside the archive is gone; what it knew is now kept in files that are only ever written once or replaced atomically.

Your existing archive keeps working and needs nothing from you. The next backup migrates it automatically, and nothing is deleted — the old database is renamed to store.db.migrated and left where it is, so you can go back to it until you are satisfied and remove it yourself.

If you queried that database with SQL, read this. There is no longer a database inside the archive to point your tools at. You build one from the archive whenever you need it, wherever you want it — or set --index-db (or index_db = true in the config) to maintain a fresh index.db beside the archive after every backup. Either way it is a snapshot of the archive, not part of it — see Querying an archive with SQL.

See the CHANGELOG for the full list.

Why an archive holds no database

The archive is often the only copy of your mail — that is the point of the tool: it takes mail out of a mailbox. So the interesting question is not what happens in the good case, but what happens when a write goes wrong.

Message files are safe by construction. They are named after the hash of their content, never modified, and never written twice; a half-written one is recognizable and can be thrown away without touching anything else.

An SQLite database is the opposite. It rewrites pages in place and relies on fsync behaving. Over SMB or NFS fsync is not reliably honoured, so a torn write does not cost you the last change — it can cost you the file. And that file held the one thing the messages themselves cannot tell you: which mailbox and which folder each message was seen in. Subject, sender, recipients and date are all in the message; that single fact was not, and losing it meant losing it for good.

So a backup no longer writes anything that is modified in place. Everything it records is either immutable or replaced atomically, and the database became what it should have been all along: a tool you build when you want to run a query, not part of the archive.

Installation

After years of Python packaging being an adventure in its own right -- virtualenvs, pip, pipx, setup.py, setuptools, poetry, and whatever else came and went -- uv has finally brought sanity to the table. The recommended way to install mailvault is from PyPI:

$ uv tool install mailvault

This installs the mailvault command into an isolated environment and makes it available on your PATH -- no manual virtualenv juggling required. Support for Microsoft 365 mailboxes via MS Graph is built in; no extra is needed. If you prefer other tooling, pipx install mailvault or pip install mailvault work just as well.

To pin a specific release, or to install the current development state straight from the repository:

$ uv tool install mailvault==0.8.0
$ uv tool install git+https://github.com/sniner/mailvault

Wheels and pre-compiled Windows executables are also available on the GitHub Releases page.

Overview

mailvault is a single command with several subcommands:

• mailvault folders | backup | verify -- back up IMAP or Microsoft 365 mailboxes to a local archive and verify the result
• mailvault archive ... -- manage the local email archive (import, compress, statistics, build a database for querying, etc.)

Global options (--config, -v/--verbose, -q/--quiet, --log-file, --allow-exec, --job) are given before the command.

Backing up mailboxes

mailvault backup downloads emails from one or more mailboxes and stores them in a local content-addressed archive. The backup can be repeated at regular intervals without creating duplicates, as long as you always export from the same mailbox.

A configuration file defines the accounts and options for the backup job (see Configuration file below).

First, you may want to get an overview of all available folders:

$ mailvault --config example.toml folders
example.org::Trash
example.org::Archive
example.org::Archive/2022
example.org::Archive/2021
example.org::Archive/2020
example.org::INBOX

Then run the backup:

$ mailvault --config example.toml backup ./backup
2024-08-15 10:05:52,275 INFO -- START
2024-08-15 10:05:52,276 INFO -- Processing mailbox: example.org
2024-08-15 10:05:52,527 INFO -- example.org::INBOX: found 3 messages
2024-08-15 10:05:52,799 INFO -- example.org::INBOX[1]: NEW: id=25652e390168...a234
2024-08-15 10:05:52,799 INFO -- example.org::INBOX[2]: NEW: id=fa1f63a13f91...c9ee
2024-08-15 10:05:52,799 INFO -- example.org::INBOX[3]: NEW: id=800be881dc38...7fa8

On subsequent runs, already archived messages are recognized and skipped:

$ mailvault --config example.toml backup ./backup
2024-08-15 10:09:28,248 INFO -- START
2024-08-15 10:09:28,250 INFO -- Processing mailbox: example.org
2024-08-15 10:09:28,531 INFO -- example.org::INBOX: found 3 messages
2024-08-15 10:09:28,820 INFO -- example.org::INBOX[1]: EXISTS: id=25652e390168...a234
2024-08-15 10:09:28,820 INFO -- example.org::INBOX[2]: EXISTS: id=fa1f63a13f91...c9ee
2024-08-15 10:09:28,820 INFO -- example.org::INBOX[3]: EXISTS: id=800be881dc38...7fa8

Use --compress to store emails compressed with zstd. Use --job NAME to run only specific jobs from the configuration file.

Deleting from the server after export

With delete_after_export = true a message is removed from the mailbox once it is archived — and only once its location has been written down durably, never before. What "removed" then means is up to the server, and the two big hosted providers do not mean what you probably expect:

On both hosted services the mailbox therefore does not actually shrink. The mail is out of your way, but it still occupies the quota. Each has an option to finish the job — one per backend, and each is refused on the other:

Gmail — name the trash folder, and it is emptied after every backup pass:

delete_after_export = true
trash_folder = "[Gmail]/Trash"

You have to supply the name because Gmail localises it: [Gmail]/Papierkorb on a German account, [Google Mail]/… on some older ones. Use mailvault --config … folders to see what yours is called.

[!WARNING] trash_folder empties the folder completely — including messages you put there yourself and messages that were never archived. It is the one place where mailvault deletes mail it did not archive, so point it at a trash folder and nothing else.

Microsoft 365 — delete for good instead of into Deleted Items:

delete_after_export = true
permanent_delete = true

This is the tidier of the two: it hard-deletes exactly the messages that were just archived, one by one, and never touches anything else that happens to be in the bin. Retention policies, litigation hold and the recoverable-items dumpster still apply — this is not a way around them, and in a tenant with a hold in place the mail remains recoverable by an administrator.

Neither option means anything without delete_after_export, and a job that sets one anyway is refused rather than run: an option that decides the fate of mail must never look effective while doing nothing.

Exchange journal mailboxes

Exchange journaling wraps every mail it records in an envelope: the journal report is the message, and the original mail is an attachment inside it. Backing such a mailbox up as-is would archive the envelopes, not the mail. With exchange_journal = true each item is unwrapped and the original is what goes into the archive. This works on both backends -- IMAP and Microsoft 365.

Journal mailboxes collect other things too, though: a bounce, a notification, or a mail someone filed there by hand. Those have no envelope to unwrap, so they cannot be archived, and they would be examined again on every run. error_folder is where they go:

[[job]]
name = "journal"
server = "exchange.example.org"
username = "journal@example.org"
exchange_journal = true
error_folder = "Journal/NotAJournalItem"

The folder is created if it does not exist, so a job that runs unattended does not stop because someone tidied it away. Without error_folder such items are reported and left in place — never deleted, in either case, since they were never archived. This is the only situation in which mailvault moves mail around in your mailbox; an ordinary backup reads, and with delete_after_export deletes, but never relocates. Setting error_folder on a job that is not a journal job therefore does nothing, and says so when the config is loaded.

[!NOTE] On Microsoft 365 the application registration needs Mail.ReadWrite for this, not just Mail.Read — moving a message and creating a folder are both writes. Without it the job stops with a message naming the missing permission rather than failing obscurely. On IMAP no MOVE capability is required: where the server lacks it (Exchange's own IMAP service often does), the older COPY + \Deleted sequence it replaced is used instead.

Verify and repair

A backup run can lose individual messages: the server may answer a single download with a 504 Gateway Timeout or drop the connection, and that message is skipped. With incremental = true the gap is invisible to every later run, because the message is older than the snapshot date and thus filtered out.

verify compares the mailbox against the archive and reports what is missing:

$ mailvault --config example.toml verify ./backup
example.org::INBOX: 77,592 on server, 43 not archived
example.org: 43 message(s) missing, run again with --repair

The comparison only lists the folder's message headers, which costs a handful of requests instead of one download per message — checking a large mailbox takes minutes, not hours. With --repair the missing messages are downloaded and added to the archive:

$ mailvault --config example.toml verify --repair ./backup
example.org::INBOX: 77,592 on server, 43 not archived, 43 restored
example.org: 43 of 43 message(s) restored

Messages are matched by their Message-ID. A message whose Message-ID is missing or ambiguous is treated as not archived and fetched again, which is harmless: the content-addressed storage recognizes the duplicate and discards it. verify does not support exchange_journal jobs, because there the archived message and the server's journal envelope carry different Message-IDs.

verify is a last resort, not routine. A folder whose downloads partly failed does not advance its snapshot, so the next ordinary backup fetches it again by itself. What is left for verify are archives from older versions and mail moved into a folder with an internal date older than the snapshot.

Managing the archive

mailvault archive provides several subcommands for working with the local archive.

Import emails

Import existing .eml files into the archive. For example, to consolidate emails from ./my_mails into ./backup:

$ mailvault --verbose archive import ./my_mails ./backup

Use --move to remove source files after import, --compress to store them compressed, and --docuware if the source is a Docuware email archive.

Statistics

Show the number of emails and total size of an archive:

$ mailvault archive stats ./backup
./backup: 1,234 emails, 567.8 MiB total

Compress / Decompress

Retroactively compress all uncompressed files in an archive with zstd, or revert compressed files back to plain .eml:

$ mailvault archive compress ./backup
./backup: 1,234 files compressed, 0 already compressed

$ mailvault archive decompress ./backup
./backup: 1,234 files decompressed, 0 already plain

Email addresses

List all sender and recipient addresses found in the archive:

$ mailvault archive addresses ./backup

Querying an archive with SQL

The archive itself holds no database. To run SQL against it, build one:

$ mailvault archive create-db ./backup ./backup.db
./backup: 130,997 message(s) read from the archive
./backup: metadata log: 60 file(s), 219,690 of 219,690 location(s) applied
./backup.db: written -- a snapshot, stale from the next backup onwards

It is assembled from the messages -- which carry their own subject, sender, recipients and date -- and from the archive's record of where each message was seen. It goes wherever you say and is not part of the archive. It is a snapshot: correct for the moment it was built, out of date from the next backup onwards. Build it again when that matters -- or have one maintained for you.

Maintaining an index.db. Pass --index-db to backup (or set index_db = true in the [global] config) and mailvault keeps a queryable index.db beside the archive, refreshed after every backup. The refresh is incremental -- only the log files added since the last one are folded in -- and it stays a projection, never a source of truth: a database that is missing or unreadable is rebuilt from scratch, so you can delete it at any time. Mail added by archive import writes no log and is not picked up this way; rebuild with create-db when that matters.

An existing file is refused unless you pass --force, which replaces it rather than adding to it -- merging two runs into one file would give you neither snapshot:

$ mailvault archive create-db ./backup ./backup.db
./backup.db: already exists, use --force to replace it

Two views are there for convenience, v_messages and v_duplicates:

$ sqlite3 ./backup.db "SELECT date, sender, subject FROM v_messages LIMIT 5"

Use --mailbox NAME for archives that predate the location record, where nothing says which job a message came from; every message is then attributed to that one name.

Migrating an older archive

Archives written before 0.8.0 keep their metadata in store.db inside the archive. The next backup moves it out by itself, but you can also do it deliberately:

$ mailvault archive migrate ./backup
./backup: 130,887 message(s) moved into 59 mailbox/folder place(s)
./backup: 46 resume timestamp(s) moved into state.json
./backup: the database is now store.db.migrated and is no longer used
./backup: delete it once you are satisfied with the archive

Nothing is deleted. The database is renamed and left alone, so you keep a way back until you remove it yourself. Running the command again on a migrated archive does nothing, and an interrupted migration is simply repeated.

Consolidating the metadata log

Every backup writes a small file into meta/ for each folder that received mail, and because incremental runs overlap, the same messages are recorded again in each run's file. Over months these add up, and everything that reads the log -- create-db, verify, --index-db -- reads all of them. compact folds them back down:

$ mailvault archive compact ./backup
./backup: 1,204 log file(s) -> 59 across 59 mailbox/folder place(s)
./backup: 41,388 duplicate observation(s) dropped

It rewrites one file per mailbox/folder holding each observation once, verifies the new files, and only then removes the originals -- so it is lossless and safe to interrupt: a half-done run just leaves both, and the next one finishes. Run it occasionally; there is no hurry, but do not put it off for years.

Migrating from ib-*

The former ib-mailbox and ib-archive commands are now subcommands of a single mailvault command. Global options are given before the command.

The third tool, ib-copy, has no successor. It transferred mail between two IMAP mailboxes, was declared "work in progress and not yet usable" when it was first committed in 2022, and never became usable; it was removed in 0.9.0. For that job use imapsync or mbsync, which do it properly. The last release that still carried it is v0.8.2.

To keep using the old ib-* commands, pin to v0.5.0, the last release before the rename.

Mail archive structure

Emails are stored as RFC 822 .eml files in a content-addressed directory structure:

./archive
├── 00
│   ├── 00
│   │   └── 00003c6ec5464cca9...7af8.eml
│   ├── 0f
│   │   └── 000ffe5b49390d9b2...26eb.eml
│   ├── 11
│   │   └── 001124d77ce778289...4fd8.eml
│   ├── 30
│   │   └── 0030f33161416b03e...97aa.eml

The filename is the SHA-384 hash of the file content and serves as the key to the archive. This makes it easy to verify file integrity by comparing the hash with the filename.

Two more things live beside the messages:

./archive
├── 00/ … ff/            the messages
├── meta/                where each message was seen
│   └── a1/
│       └── a1b2c3….jsonl
└── state.json           where the next incremental run picks up

meta/ answers the one question the messages cannot: which mailbox and which folder each was seen in. One file is one place -- its first line names a mailbox and a folder, the rest name the messages seen there. A message that belongs to several places simply appears in several files, so nothing is ambiguous. These files are content-addressed exactly like the messages, so each one carries its own integrity check:

$ sha384sum meta/a1/a1b2c3….jsonl     # matches the filename, or the file is damaged

They are written once and never modified. state.json holds the timestamps that decide where the next incremental run resumes; it is small and always replaced atomically, never edited in place.

Both are plain text. If you ever want to know what an archive thinks it contains, you can read it without mailvault and without SQL.

Emails with the same Message-ID are considered identical from a user perspective, but if their RFC 822 representation differs, they are stored separately because the hashes differ. MS Exchange in particular tends to produce different versions of the same email -- journal copies, for instance, often differ from mailbox copies by an additional Received header and replaced MIME multipart delimiters.

Configuration file

mailvault reads its configuration from a TOML file. The file name is irrelevant -- the content is always parsed as TOML.

Basic example

A simple example for Google Mail:

[[job]]
name = "gmail.com"
server = "imap.gmail.com"
username = "john.doe@gmail.com"
password = "123456"
folders = ["All Mail"]

A more complete example with folder exclusions:

[[job]]
name = "example.org"
server = "imap.example.org"
username = "john.doe@example.org"
password = "123456"
port = 993
tls = true
ignore_folder_flags = ["Junk", "Drafts", "Trash"]
ignore_folder_names = ['.*/Calendar/?.*']
folders = ["INBOX", "Archive"]

An example for MS Exchange journal export:

[[job]]
name = "exchange.example.org"
server = "exchange.example.org"
username = "john.doe@example.org"
password = "123456"
tls_check_hostname = false
exchange_journal = true
delete_after_export = true
folders = ["INBOX"]

MS Graph backend

As an alternative to IMAP, mailvault can access Microsoft 365 mailboxes via the MS Graph API. This avoids the quirks of Microsoft's IMAP implementation and uses OAuth2 client credentials for authentication, which is well suited for unattended backup scenarios.

To use this backend, set backend = "msgraph" in the job configuration. Authentication requires an Azure AD app registration with Mail.Read (or Mail.ReadWrite if using delete_after_export) application permissions.

[[job]]
name = "m365-backup"
backend = "msgraph"
tenant_id = "your-azure-tenant-id"
client_id = "your-app-client-id"
client_secret_cmd = "pass show m365/client-secret"
username = "john.doe@example.com"
folders = ["Inbox", "Archive"]

The username is the email address of the mailbox to back up. Most other options (folders, ignore_folder_names, exchange_journal, error_folder, etc.) work the same as with IMAP. Three do not:

• ignore_folder_flags has no effect — Graph folders have no IMAP-style flags
• delete_after_export is a soft delete: the message moves to Deleted Items and stays there, see Deleting from the server
• trash_folder is an IMAP option and is refused here — permanent_delete is its counterpart on this backend

Global options

Some options apply to the whole run rather than to a single mailbox and are set in a [global] section: compress, index_db, and incremental. They are marked (global option) in the tables below.

[global]
compress = true
incremental = true   # the default; set to false to re-fetch every folder in full

[[job]]
name = "gmail.com"
server = "imap.gmail.com"
# ...

Dynamic values

All string values in the configuration support environment variable expansion using ${VAR} or ${VAR:-default} syntax:

[[job]]
name = "example.org"
server = "${IMAP_SERVER:-imap.example.org}"
username = "${IMAP_USER}"

Additionally, any string field can be replaced by a _cmd variant that runs a shell command and uses its output as the value. This works for any field, not just passwords:

[[job]]
name = "example.org"
server = "imap.example.org"
username = "john.doe@example.org"
password_cmd = "pass show email/example.org"
client_secret_cmd = "az keyvault secret show --name my-secret --query value -o tsv"

For security, _cmd fields are only evaluated when the --allow-exec flag is passed on the command line. Without it, _cmd fields are silently ignored with a warning.

Parameters

Options

Metadata

Besides the messages, a backup records two things: where each message was seen, into meta/, and where the next run should resume, into state.json. That is all -- there is no database in the archive, and nothing is modified in place. See Why an archive holds no database.

This is not optional. Both files are small, immutable or atomically replaced, and they are what makes an incremental backup and verify possible at all. The option that used to switch them off (with_db, later with_metadata) existed because of the SQLite database, and went with it.

To query an archive with SQL, build a database from it when you need one: see Querying an archive with SQL.

MS Windows

A pre-compiled mailvault.exe for Windows is provided as an asset on the GitHub Releases page. You can download it directly without needing Python or any dependencies.