bdbackup

Backups by design.

bdbackup is a Python library and command-line tool for file and MySQL/MariaDB backups with verification, retention, and guided recovery. Define repeatable jobs, track their outcomes, and prepare recovery copies with explicit safeguards at each step.

Choose the backup method that fits your data:

• file backups from a plain-text template file into a tar archive
• mysqldump logical backups (gzip-compressed)
• xtrabackup / mariabackup physical full and incremental backups

Built for day-to-day system administration: TOML configuration, credential files, verification before publication, process locking, atomic publication, and optional SQLite history keep backup operations explicit and inspectable.

Support and safety

Python 3.12+ on POSIX systems (Linux/macOS); Windows is not supported. tar and tar.gz work on every supported Python; tar.zst requires Python 3.14 with Zstandard support. Database binaries are separate system dependencies.

Every built-in backend verifies staging output before publishing it. A missing or unreadable file fails the backup and preserves the previous archive. A process lock prevents concurrent writers to the same file destination or physical root. --no-verify skips only the additional verification pass after publication. Verification checks archive readability or physical checkpoints; it does not replace testing a restore. File backups are not filesystem snapshots: quiesce applications or back up snapshots when files can change during a run.

Safe relative symlinks, hardlinks, empty directories and ordinary directory permissions/timestamps round-trip. Restore rejects escaping paths/links and special files on all supported Python versions. Ownership and privileged permission bits are intentionally not restored. Use a destination that is not being modified by another process during extraction.

Installation

Install the current code from the original GitHub repository:

pip install "git+https://github.com/ayder/bdbackup.git"

For versions published to PyPI:

pip install bdbackup

With optional MySQL metadata support:

pip install "bdbackup[mysql]"

For development:

git clone https://github.com/ayder/bdbackup.git
cd bdbackup
pip install -e ".[dev]"

Global options

Place the global option before the command: bdbackup --logging DEBUG file .... Levels: DEBUG, INFO, WARNING, ERROR. The CLI uses these exit codes:

File backup

Create a template file listing paths (one per line, # for comments, whitespace allowed):

# backup-template.txt
Documents
Videos
data/projects

Run the backup:

bdbackup file --template backup-template.txt -d /backup/files/daily

Gzip-compress:

bdbackup file --template backup-template.txt -d /backup/files/daily --format tar.gz

Options:

Restore a file archive:

bdbackup restore /backup/files/daily.tar -d /restore/here

Exclusion templates

--exclude-template applies a named bundle of gitignore-style exclusion patterns so you don't have to repeat common artifact rules:

bdbackup file --template backup-template.txt -d /backup/files/daily \
    --exclude-template python-dev

The built-in python-dev template skips __pycache__/, *.pyc, .venv/, venv/, uv.lock, Pipfile.lock, poetry.lock, *.egg-info/, build/, dist/, and common tool caches (.mypy_cache/, .pytest_cache/, .ruff_cache/, .tox/, ...). Templates are repeatable and comma-separated values work too; they combine with -x/--exclude and --exclude-pattern.

Patterns use gitignore-style syntax: *.pyc matches at any depth, a trailing / matches directories only, patterns containing a / (e.g. tests/containers/*img) match relative to the backup root, a leading / anchors to the root, and ! negates a previous pattern.

Custom templates are plain Python files in ~/.config/bdbackup/templates/ (honours $XDG_CONFIG_HOME and $BDBACKUP_CONFIG_DIR) that self-register — no existing code needs to change to add one:

# ~/.config/bdbackup/templates/go_dev.py
from bdbackup.templates import ExclusionTemplate

TEMPLATE = ExclusionTemplate(
    name="go-dev",
    patterns=("vendor/", "vendor/**", "*.test", "go.work"),
    description="Go development artifacts",
)

Database engines

Database engines are pluggable and live in per-database family packages (bdbackup/mysql/ today; bdbackup/postgres/ is planned). Each engine maps a config type name to a backend class; adding one never requires editing existing wiring code:

To add an engine, drop a self-registering module into either bdbackup/mysql/ (shipped with the package) or ~/.config/bdbackup/engines/ (user-level, no package changes) — a versioned xtrabackup variant or a mysql-shell util.dump() engine both follow the same recipe:

# ~/.config/bdbackup/engines/mysql_shell.py
from bdbackup.engines import EngineInfo

class MySQLShellDump:
    def __init__(self, out_dir: str = ".", **params): ...
    def backup(self, name=None): ...          # BackupBackend protocol
    def verify(self, result=None): ...
    def prune(self): ...

ENGINE = EngineInfo(
    name="mysql-shell",          # usable as type = "mysql-shell" in config
    backend=MySQLShellDump,
    description="MySQL Shell util.dump() backups",
    family="mysql",
)

A config type is validated against the live registry, so the new engine is immediately usable from config.toml. MySQL-specific helpers shared by the family (e.g. mysql_cnf_file for credential-safe defaults files) live in bdbackup/mysql/helpers.py. A new database family means creating bdbackup/postgres/ and appending "bdbackup.postgres" to bdbackup.engines.FAMILIES — the single deliberate modification point.

mysqldump

Dump a single database:

bdbackup mysqldump --database mydatabase -o /backup/mysql/dumps -u root -p

Dump all databases:

bdbackup mysqldump --full -o /backup/mysql/dumps -u root -p

Parallel dumps of multiple databases:

bdbackup mysqldump --database db1,db2,db3 -o /backup/mysql/dumps -u root -p --jobs 4

Options:

Use a bare -p to enter a password securely. Passing -p PASSWORD puts it in bdbackup's own process arguments and potentially shell history. Child database processes receive only a temporary credentials-file path; its contents are quoted, its permissions are 0600, and it is removed after the run.

Mysqldump retains --single-transaction, --routines, --events and --triggers by default. --options adds options; explicit --skip-* flags can override applicable defaults. --full retains these defaults and adds --all-databases. Single-transaction consistency applies to transactional tables; quiesce writes to nontransactional tables and avoid schema changes during a dump.

xtrabackup

Full backup:

bdbackup xtrabackup full --database production -r /backup/mysql -u xtrabackup -p

Incremental backup (chains to the latest successful full):

bdbackup xtrabackup incremental --database production -r /backup/mysql -u xtrabackup -p

Prune old backups by retention days:

bdbackup xtrabackup prune --database production -r /backup/mysql --retention 7

Prepare a full backup or an incremental recovery point into a new directory:

bdbackup xtrabackup prepare /backup/mysql/production/2026-09-10/Full_ID \
    -r /backup/mysql/production -d /restore/production -u xtrabackup -p

Use the exact path printed by the backup command in place of Full_ID. Passing an incremental path prepares its full and every prerequisite incremental up to that point. Preparation copies sources to private working directories, decompresses compressed copies, applies the increments in dependency order, and publishes the recovery directory only on success. The destination must be new and outside the backup root. Original backups remain available for new incrementals and repeated recovery attempts.

Use XtraBackup matching your MySQL/Percona server series (8.0 with 8.0, 8.4 with 8.4); use mariabackup or mariadb-backup matching your MariaDB installation. MariaDB preparation omits XtraBackup's --apply-log-only option. Compression is off by default. For a compatible recent XtraBackup, select --compress zstd; MariaDB's deprecated built-in compression accepts only quicklz and requires qpress for decompression. Compatibility must be established with an actual recovery test for the exact server and backup binary versions in use.

New physical backups record parent/full identities and LSNs in bdbackup.json. Incrementals live under DATE/Incremental/FULL_ID/UNIQUE_ID, and cannot attach to another full taken on the same day. Older backups without this metadata require a new full before taking further incrementals; full backups can still be prepared as recovery copies. Retention removes complete dated chains, preserves the newest successful full's date, and refuses to prune without a valid full.

Options:

A process lock prevents two backup runs from corrupting the same backup root. Failed backups are written to a temporary directory first and cleaned up on error, so a partial backup can never be mistaken for a complete one.

Encrypted physical backups

Percona XtraBackup jobs can optionally encrypt full and incremental backups with AES256. In the existing job's TOML section, add:

encrypt = true
encrypt_key_file = "/etc/mysql/xtrabackup.key"

TOML uses true/false, not yes/no. The key file must already exist, be readable by the backup account, and contain exactly 32 bytes. To create a new key once (this command refuses to overwrite an existing key):

sudo python3 - <<'PY'
import os
from pathlib import Path

key_path = Path("/etc/mysql/xtrabackup.key")
fd = os.open(key_path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
with os.fdopen(fd, "wb") as key_file:
    key_file.write(os.urandom(32))
PY

Keep this key outside the backup tree and store a separate secure copy: losing it makes its encrypted backups unrecoverable. Take a new full backup when changing keys or encryption settings; incrementals must use their parent's settings and key. Encryption is not supported by the mariabackup backend.

Run the configured job normally, or use the direct command:

bdbackup run --config /opt/dbs/config.toml mysql-prod
bdbackup --config /opt/dbs/config.toml xtrabackup full \
  --database production --root /backup/mysql \
  --encrypt --encrypt-key-file /etc/mysql/xtrabackup.key -u mysql -p

History records encrypted = 1 in SQLite's backup_runs table for encrypted attempts (0 otherwise), including failed attempts. The status column indicates whether an attempt succeeded. History also stores the algorithm and key-file path for recovery, never the key itself. Existing history databases upgrade automatically on the next backup; old entries remain unencrypted.

History restore uses the recorded key path. If the key has moved, provide its new location:

bdbackup restore --config /opt/dbs/config.toml --backup-id 12 \
  --dst /restore/mysql-prod-12 --encrypt-key-file /secure/saved-xtrabackup.key --yes

Recovery decrypts each full/incremental work copy, then decompresses and prepares it. Original backups remain encrypted; the recovery directory contains plaintext. For manual xtrabackup prepare, supply --encrypt-key-file as well. See Percona's encryption documentation.

Retention policy (GFS)

The bdbackup retention command applies a grandfather-father-son policy to a backup tree: keep every backup for the last N days, then one per ISO week for N weeks, then one per calendar month for N months. Tiers are sequential and never overlap, so the retained set is deterministic and easy to reason about.

# Dry run (default) -- lists what would be deleted, changes nothing
bdbackup retention --full-dir /backup/mysql/production \
    --incr-dir /backup/mysql/production/incr \
    --log-dir /backup/mysql/production/log \
    --daily 7 --weekly 4 --monthly 6 \
    --incr-days 7 --log-days 30 \
    --min-keep-fulls 2 --pick first

# Actually delete
bdbackup retention ... --apply

Chain safety: this flat-file policy expects one chronological backup stream: each new full starts a chain, and subsequent increments continue that chain until the next full. A retained incremental pins its full and every preceding incremental in that chain, including prerequisites older than the age window. A missing full makes an incremental an orphan. If a full/incremental scan skips a file (for example while it is being written), expiration is deferred because its dependencies are uncertain. Keep all files for a stream together; arbitrary overlapping chains or missing intermediate backups require explicit external metadata and are not supported by this filename-based policy.

This command handles flat backup files such as full_*.mbi, not the dated XtraBackup directory tree. Use bdbackup xtrabackup prune for physical backups.

Only --full-dir is required. --incr-dir and --log-dir are optional. --full-glob, --incr-glob, and --log-glob default to full_*.mbi, incr_*.mbi, and *.log.

Config-driven jobs

For scheduled or multi-job usage, define a TOML config file. Convention: keep all bdbackup settings under ~/.config/bdbackup/ (honours $XDG_CONFIG_HOME / $BDBACKUP_CONFIG_DIR) — config.toml, the files.template path lists, templates/*.py exclusion presets, and engines/*.py database engines.

# ~/.config/bdbackup/config.toml
[files-daily]
type = "file"
template_filename = "~/.config/bdbackup/files.template"
backup_dst = "/backup/files/daily"
chdir = "/srv/www"   # source path: template/exclude entries resolve against it
format = "tar.gz"  # tar.zst requires Python 3.14+
exclude_pattern = ["*.log", "node_modules"]
exclude_templates = ["python-dev"]

[mysql-prod]
type = "xtrabackup"
backup_root = "/backup/mysql/production"
user = "xtrabackup"
retention_days = 7
parallel = 2

[mysqldump-all]
type = "mysqldump"
out_dir = "/backup/mysql/dumps"
user = "backup"
options = ["--single-transaction", "--all-databases"]

[mysql-prod-retention]
type = "retention"
full_dir = "/backup/flat/full"
incr_dir = "/backup/flat/incr"
log_dir = "/backup/flat/log"
daily = 7
weekly = 4
monthly = 6
incr_days = 7
log_days = 30
min_keep_fulls = 2
pick = "first"
apply = false           # set true once the dry-run output looks right

Configuration paths expand ~; relative config paths resolve against the config file's directory. Template entries and exclude entries resolve against chdir (or the process working directory if omitted). For a single-database mysqldump job, set database = "mydatabase"; for all databases include --all-databases in options.

Except for the optional [history] settings, each top-level table is one job; its keys (except type and optional schedule) are passed to the backend constructor, so they use Python-style underscores (exclude_templates), not CLI dashes. See config.toml.example for a fully annotated config with every option explained.

Run one job:

bdbackup run --config ~/.config/bdbackup/config.toml mysql-prod

Run every job:

bdbackup run --config ~/.config/bdbackup/config.toml --all

Validate configuration and permissions

Check all configured jobs without creating a backup or running retention:

bdbackup --config /opt/dbs/config.toml --validate
# Or validate only one job:
bdbackup run --config /opt/dbs/config.toml mysql-prod --validate

Run validation as the OS account that will run your cron jobs. It checks backend settings, required executables, destination access, SQLite/recovery directories, file-template sources, and encryption key requirements. Missing directories are reported with mkdir -p guidance and the required OS-user permissions. A missing directory that the backend can create under a writable parent is reported as creatable; validation does not create it.

For MySQL jobs, the installed mysql/mariadb client authenticates using the job's credentials and reads CURRENT_USER(), server version, datadir and SHOW GRANTS. Passwords are passed through a temporary mode-0600 options file, removed afterward. No CREATE USER or GRANT is executed. Missing privileges produce SQL for an administrator, for example:

GRANT RELOAD, BACKUP_ADMIN, REPLICATION CLIENT, PROCESS, LOCK TABLES
ON *.* TO 'xtrabackup_user'@'localhost';
GRANT SELECT ON `performance_schema`.`log_status`
TO 'xtrabackup_user'@'localhost';

The helper also checks the performance-schema tables used by current Percona XtraBackup and prints CREATE TABLESPACE separately as an optional privilege for importing individual tables. MariaDB Backup gets its own privilege recommendations. See Percona privileges and MariaDB Backup privileges.

Checks cover direct grants; role-derived privileges and partial revokes may need manual review. Physical datadir checks cover root-directory access, not every data file or external tablespace. Custom mysqldump options, routine visibility, GTID settings and exact server/binary version compatibility still need review. An unreachable database or missing client is a failed/incomplete check, not a pass. Retention validation checks settings and directory permissions without scanning for deletions, even when apply = true. Helpers do not add SQLite history rows.

Recommend cron entries

Inspect the invoking OS account's crontab -l and print suggested entries:

bdbackup --config /opt/dbs/config.toml --cron
bdbackup run --config /opt/dbs/config.toml mysql-prod --cron
# Both helpers can be used together:
bdbackup --config /opt/dbs/config.toml --validate --cron

Nothing is installed or edited. Defaults are daily backups starting at 02:00 and retention starting at 04:00, staggered by 15 minutes within each group. Override the time inside any job's existing TOML section:

[mysql-prod]
type = "xtrabackup"
backup_root = "/backup/mysql/production"
schedule = "30 1 * * *"

[mysql-prod-retention]
type = "retention"
full_dir = "/backup/flat/full"
schedule = "0 5 * * 0"
apply = false

schedule accepts five numeric cron fields with wildcards, lists, ranges and steps. It is job metadata, never passed to the backup backend. The recommendations use absolute executable/config paths, quote shell arguments, preserve the current working directory and suggest the current PATH for cron. Times use the cron daemon's timezone. Allow enough time for backups before retention; separate cron entries do not establish a dependency.

Matching active bdbackup run entries for the same config/job (or --all) are reported without suggesting duplicates. Commented entries and other configs do not count. Shell wrappers such as flock, scripts, and system-wide cron files are not inspected; review those separately. If crontab cannot be read, suggested entries are still shown but the helper exits with status 1.

XtraBackup job entries run full backups, including their built-in pruning. Separate retention jobs remain for flat backup files; apply = false stays a dry run in cron too. Validation/cron helpers exit 0 when their checks pass, 1 for failed or incomplete checks, and 2 for invalid command/configuration syntax.

Backup history and guided restore

Enable SQLite history in your TOML configuration. No extra Python dependency or database server is required:

[history]
database = "state/history.sqlite3"
restore_root = "/restore/bdbackup"

Both paths expand ~; relative paths resolve against the configuration file. restore_root defaults to restores beside that file. Omitting [history] disables history. File jobs automatically exclude this live database and its SQLite journal files; keep it outside backup inputs when possible. New history databases are created with permissions 0600.

run --config records every selected backup attempt. For direct commands, place --config before the command:

bdbackup run --config config.toml files-daily
bdbackup --config config.toml mysqldump --database app,analytics --jobs 2
bdbackup history --config config.toml
bdbackup history --config config.toml --job files-daily --successful

Each record contains an ID, job name, backup type, UTC start/completion times, status (running, success, or failed), and the successful artifact's absolute path and size. Direct file commands use the destination name as the job name; direct database commands use the database name. Parallel dumps get one record per database, including when another dump fails. Physical full and incremental commands record their respective types and the root/binary needed for preparation. Credentials and raw error messages are not stored; failed rows contain only the exception class. Retention, restore operations, and file dry runs are not backup attempts and do not create rows. Direct Python backend calls are not automatically recorded; applications can wrap them with History.run.

History is written before work starts, and success only after the backup and its requested verification finish. If history cannot be written, the command fails; an artifact already created is preserved. A forcibly killed process may leave a running row, which is never offered for restore. Existing backups are not imported automatically.

Select a successful backup interactively:

bdbackup restore --config config.toml
bdbackup restore --config config.toml --job files-daily

Restore lists successful, available artifacts, asks for the backup ID, suggests <restore_root>/<job>-<id>, and asks for confirmation. You can edit the suggested path. Repeated restores suggest a numbered alternative; history-based restore requires a new directory even when --dst is supplied. For automation, specify the exact backup ID and use --yes:

bdbackup restore --config config.toml --backup-id 12 --dst /restore/job-12 --yes

• File archives are extracted into the selected directory with the existing safe extraction filters.
• MySQL dumps are decompressed and checked into <destination>/backup.sql. Importing SQL into a running server is a separate administrator action.
• XtraBackup/MariaDB backups produce a prepared recovery directory, including prerequisite increments. The destination must be outside the backup root. Server ownership, copy-back, and startup remain administrator actions.
• Custom engines are recorded, but require their own restore support unless they inherit a supported backend.

Deleted artifacts remain in history as unavailable. File identity, size, and modification time detect replaced archives, so older rows for a reused filename are not offered as older recovery points. These checks are not cryptographic integrity checks; restoration still validates the actual archive or physical dependency chain. History stores references to artifacts and does not preserve an archive that a later backup replaces.

Development

Run tests and linting:

pytest
ruff check bdbackup tests scripts

Optional database integration tests use disposable, network-isolated Docker containers with synthetic data. They remove only the containers and volumes created for that run. Pull the matching images before running:

python scripts/integration_mysql.py           # mysql:8.4
python scripts/integration_physical.py percona # percona-server/xtrabackup:8.4
python scripts/integration_physical.py percona-encrypted # AES256 + zstd recovery
python scripts/integration_physical.py mariadb # mariadb:11.4

Physical recovery returns a prepared data directory; copying it into a server's data directory, assigning ownership to the server user, and starting that server are separate administrator actions. Use the same database version and filesystem case-sensitivity as the source.

pyproject.toml (project.version) is the sole version source. The package's __version__ and CLI read installed distribution metadata generated from it. After changing the version, run uv lock to refresh the generated lockfile and reinstall with pip install -e '.[dev]' to refresh local metadata. Source development requires this editable installation.

Build and validate a release:

python -m build
twine check dist/*
python scripts/smoke_install.py dist

Commit the validated changes and create an annotated tag named v<project.version>. The publishing workflow accepts only that matching tag. It runs the reusable CI workflow on that commit (tests on Python 3.12–3.14, lint, real MySQL/Percona/MariaDB recovery tests, source/wheel build, Twine, and an installed-wheel recovery smoke test), then uploads those exact artifacts. Configure the PyPI trusted publisher for ayder/bdbackup, publish.yml, environment pypi before releasing. Account configuration and actual database recovery evidence are release prerequisites.

License

MIT License. See LICENSE.