Declarative Data Backup

Declarative Data Backup creates encrypted Borg archives for configured source directories. Each source gets its own archive series in one Borg repository, so Borg can deduplicate chunks across all sources. The application validates the complete configuration before invoking Borg and prunes a source's archives only after every create operation in the run has succeeded.

Requirements

• Python 3.11 or newer.
• Borg 1.4.x installed separately and available as borg on PATH.
• A Borg repository that has already been initialized in keyfile encryption mode, its matching keyfile, and a protected passphrase source.
• For remote repositories, OpenSSH and a trusted known_hosts file.

The application never initializes a repository. Borg 2 is not supported.

Installation

Install the package from PyPI:

python -m pip install declarative-data-backup

Or install a checkout with the development dependencies:

python -m pip install -e ".[dev]"

The PyPI distribution is named declarative-data-backup; its importable package is declarative_data_backup.

GitLab release automation

The pipeline in .gitlab-ci.yml publishes releases when a Git tag in major.minor.patch format is pushed, for example 1.2.3. Use the same version for the GitLab Release name and tag. Tags with a v prefix, prerelease suffix, or leading zeros are not published. Branch pushes do not run this release pipeline.

For each version tag, the pipeline:

1. Sets the package version in the CI checkout's pyproject.toml, builds a wheel and source distribution, and checks them with Twine.
2. Uploads declarative-data-backup to PyPI.
3. Builds the existing Python 3.13 Dockerfile with the same package version, verifies the installed version, and pushes ${CI_REGISTRY_IMAGE}/declarative-data-backup:1.2.3-python3.13 to the project's GitLab Container Registry.

The source checkout's version is not committed back to the repository. The two publishing jobs run independently after the package build succeeds; publishing is not atomic across PyPI and the container registry.

Before the first release:

• Enable the project's GitLab Container Registry.
• Add PYPI_API_TOKEN in GitLab Settings > CI/CD > Variables as a masked, protected variable. Use a PyPI API token authorized to publish declarative-data-backup; for a new PyPI project, its first upload may require an account-wide token, which should then be replaced with a project-scoped token.
• Protect release tags in GitLab (for example, *.*.*) and restrict who can create them. Protected variables are only available on protected refs.
• Use a trusted Docker-executor runner configured for Docker-in-Docker with TLS: privileged = true and volumes = ["/certs/client", "/cache"] in its [runners.docker] configuration. GitLab.com instance runners support this setup. Privileged runners should not execute untrusted code. If your runner requires a tag, add it to publish_container.

Registry authentication uses GitLab's built-in CI_REGISTRY_USER and CI_REGISTRY_PASSWORD; no separate registry password is needed.

Create and push a release tag from the commit you want to publish:

git tag 1.2.3
git push origin 1.2.3

Creating a GitLab Release with a new 1.2.3 tag also triggers the tag pipeline. Creating or editing a Release on an already existing tag does not trigger a new pipeline. PyPI versions cannot be overwritten: do not move or reuse release tags. If only the container job fails, retry that job rather than rerunning a successful PyPI upload.

Configure and run

Create a YAML file accessible to the backup process:

media_backup:
  folders:
    - /server_root/media/photos
    - /server_root/media/music
  compression: auto,zstd,22
  retention_days: 30
  required_mounts:
    - /server_root/media
  allow_empty: true

postgres_backup:
  folders:
    - /server_root/path/to/postgres-backups
  required_mounts:
    - /server_root/path/to/postgres-backups

Set the required environment variables before starting the process:

Variable	Purpose
BACKUP_CONFIG_PATH	Absolute path to the YAML configuration.
SERVER_ROOT	Absolute path to the root containing all configured source directories.
BORG_REPO	Existing Borg repository location, local or remote.
BORG_KEY_FILE	Path to the external keyfile for the encrypted repository.
BORG_BASE_DIR	Writable directory for Borg state and the backup coordination lock.
BORG_RSH	SSH command and options used by Borg for a remote repository.
BORG_PASSCOMMAND	Command Borg runs to obtain the repository passphrase.

Optional variables:

Variable	Purpose
BORG_REMOTE_PATH	Remote Borg executable path, passed as Borg's --remote-path. Leave unset to use Borg's default.
BORG_EXIT_CODES	Borg exit-code mode: legacy or modern.

Provide credentials through a secret manager or protected files; do not put passphrases, private keys, or credentials in the YAML file, source control, or shell history. For example, in a controlled environment where these variables have already been securely populated:

python -m declarative_data_backup

The command exits with status 0 on success, nonzero on configuration or Borg failure, and 130 when interrupted. Borg's exit code 1 indicates warnings: the run fails and pruning is skipped.

Configuration reference

The top level must be a nonempty YAML mapping of backup group names to group mappings. Group names may contain ASCII letters, digits, _, and -, and must be at most 64 characters. Duplicate YAML keys, unknown group fields, invalid values, and malformed or empty configuration are rejected before any create or prune operation.

Each group supports:

• folders (required): a nonempty list of absolute directory paths. Each source must exist, be readable, and resolve inside SERVER_ROOT. Duplicate canonical paths and parent/child overlaps across groups are rejected.
• compression (optional): defaults to auto,zstd,22. Supported Borg 1.4 codecs include none, lz4, zstd, zlib, and lzma; auto may precede a codec. Codec levels are checked against supported ranges: zstd 1-22, zlib 0-9, and lzma 0-9. Borg's obfuscate compression specifier is also accepted for specifier values 1-6, 110-123, or 250; it is useful only with repository encryption.
• retention_days (optional): positive integer, default 30. This controls Borg's --keep-daily policy.
• required_mounts (optional): list of exact mount points that must appear in the process's /proc/self/mountinfo.
• allow_empty (optional): boolean, default true. If set to false, an empty source is rejected.

Configure required_mounts explicitly for every source that depends on a separately mounted filesystem. The program does not infer mount requirements from source paths. In a container or Kubernetes pod, mounting /server_root does not prove that nested host filesystems are mounted. In particular, allow_empty: true does not prove that a disk is present: an empty source may be legitimate and can be archived.

Paths are resolved before backup. A top-level source symlink is allowed only when its canonical target remains inside SERVER_ROOT; symlinks within a source are stored as links rather than followed.

Archives, retention, and compaction

The application creates one archive per configured source in the single BORG_REPO. Names have this form:

bbkp__<source-series>__<UTC timestamp>__<run UUID>

The stable source series is derived from the canonical source path and includes a SHA-256 suffix. Borg receives the canonical absolute path as the source, so the original directory hierarchy is retained inside the archive.

The application creates all configured archives before pruning any source. If a create fails or Borg reports warnings, the run stops and no pruning is performed. Retries use a new run UUID and may create another archive for that day; archives are never overwritten to handle a collision.

--keep-daily N retains the latest archive in up to N daily buckets. It does not guarantee N successful calendar days of recovery if scheduled runs fail or are missed. Removing a source from the YAML does not delete its old archives. The source-series selector does not match archives made with the previous non-namespaced naming scheme; retain those archives unless a separately reviewed migration or cleanup is performed.

Pruning marks repository data for deletion; it does not reclaim the space. Repository compaction is a separate maintenance operation and is not run by this application. Any scheduled compact job or manual maintenance must first acquire an exclusive flock on:

<BORG_BASE_DIR>/backup.lock

Use the same persistent state volume and filesystem as the backup job, with working advisory-lock semantics. The backup process waits up to 3600 seconds for this lock. Kubernetes concurrencyPolicy: Forbid does not coordinate separate CronJobs, and Borg's own repository lock does not replace this application-level lock. Never unlink or replace the lock file while any process could hold it.

Deployment and operational notes

The Dockerfile builds an image with Borg 1.4.1 and the Python package. The application itself does not set CPU or I/O priority; a scheduled deployment may wrap the command with nice -n 19 and ionice -c 3. For Kubernetes or another container runtime:

• Run as a non-root user and confirm it can read the keyfile and credentials and write the Borg state volume.
• Mount source data read-only. Keep Borg cache, configuration, security state, keyfile, and the coordination lock on persistent writable storage.
• For nested source filesystems, configure and verify the corresponding required_mounts.
• Use a versioned image tag or digest and confirm private-registry pull credentials have only the required read access.
• Do not mount a service-account token unless the workload needs one.
• Ensure any compact job mounts the same state volume and honors the lock contract above.
• Monitor failed and warning backup jobs. Retention cannot make up for days without successful archives.
• A fixed schedule does not ensure upstream jobs, such as Unmanic processing, have completed; establish that dependency separately.

The default auto,zstd,22 may be slow or exceed a 1 GiB memory limit on some data sets. It has not been benchmarked under that limit and may be terminated for memory use. Compression can be overridden per group without recreating the repository.

Kubernetes Secret projection permissions, storage ownership, mount behavior, node placement, registry access, and workload scheduling must be verified in the target cluster. They are not verified by this Python package or its container integration test.

Testing

Run the unit tests:

python -m pytest -m "not integration"

The opt-in local Borg integration test uses a temporary encrypted repository:

RUN_BORG_LOCAL_INTEGRATION=1 python -m pytest -m integration \
  tests/integration/test_local_borg_integration.py

The opt-in Docker SSH integration test uses disposable containers and a temporary repository; it does not use production credentials or BorgBase:

RUN_BORG_DOCKER_INTEGRATION=1 python -m pytest -m integration \
  tests/integration/test_docker_integration.py

Do not treat skipped integration tests as successful restore validation.

Recovery

Recovery requires the repository and the matching external keyfile and passphrase. Keep independent recovery copies of the keyfile and passphrase; the repository must not be their only copy. Also retain the SSH private key and verified known_hosts entry for a remote repository.

With the required Borg environment and credentials securely configured, list archives and inspect the desired archive:

borg list "$BORG_REPO"
borg list "$BORG_REPO"::'bbkp__<source-series>__<timestamp>__<run-uuid>'

Extract into an isolated recovery directory and verify the restored data before returning it to service:

mkdir -p /restore
cd /restore
borg extract "$BORG_REPO"::'bbkp__<source-series>__<timestamp>__<run-uuid>'

Test recovery periodically using a disposable or approved recovery environment. This repository's tests do not verify access to a live remote repository or actual Kubernetes Secret projection and storage behavior.