django-mirroring

Django add-on for production database mirror refresh and staging restore. Uses Dumpling for in-stream anonymisation and Postgres shadow-database cutover so consumers never see a half-loaded mirror.

Links

• Changelog
• Contributing
• Security
• Issues

Supported versions

This package supports Django 6.0 and Python 3.12+. Dependencies are compatible-release pinned; see Pinned dependencies.

Installation

Pick the command for your preferred package installer:

uv add django-mirroring
pip install django-mirroring
# or from GitHub while unpublished:
pip install git+https://github.com/ababic/django-mirroring.git

Add "mirroring" to INSTALLED_APPS and run migrations:

python manage.py migrate mirroring

Pinned dependencies

Every dependency is pinned to a compatible release (~=) so a mirror run cannot silently pick up an incompatible Dumpling policy format or AWS SDK behaviour.

dumpling-cli ships the dumpling executable, so installing this package also pins the CLI. The commands verify it at start-up and refuse to run against a different minor series.

Postgres client tools (pg_dump, psql) are system packages, not pip dependencies, so they are pinned as a minimum major — pg_dump refuses to dump a server newer than itself, while newer clients read older servers fine:

Commands fail fast with a clear error when a tool is missing or too old, rather than part-way through a dump.

Dumpling policy (project-owned)

Anonymisation rules live in a project-owned Dumpling TOML file, not inside this package. Point the refresh command at it with MIRROR_DUMPLING_CONFIG (env) or MIRROR_DUMPLING_CONFIG (Django setting). The effective config may also embed generated row_filters and staff username keep rules at refresh time.

Settings overview

Configure via Django settings and/or environment variables (host projects typically wire env → settings in one place).

DUMPLING_GLOBAL_SALT must be set in the environment for Dumpling lint/run.

Endpoint guidance (operators)

Refresh and restore only refuse when source and destination resolve to the same host/port/database (restore/revert also require MIRROR_RESTORE_ALLOW=1 and --confirm). Which databases those env URLs point at is otherwise an operator responsibility — document your project's URLs carefully; there is no hostname allow/block list in the package:

• Prefer pointing MIRROR_SOURCE_DATABASE_URL at a full-access follower or offline replica so pg_dump can read every table Dumpling anonymises, and so refresh load does not compete with live writes. Dumping the primary is allowed but not recommended under load.
• Prefer pointing MIRROR_DATABASE_URL at a separate mirror database from the live app primary. The destination role needs CREATEDB for shadow load + rename cutover.
• Point MIRROR_RESTORE_TARGET_DATABASE_URL only at a disposable staging (or equivalent) database you intend to replace.
• A restricted / allow-listed role that cannot SELECT PII tables will produce an incomplete or failing dump — use full-access credentials for the source.
• Put sslmode on connection URLs when the server requires TLS (no hostname-based SSL inference).
• Omit provider schemas (e.g. Heroku's heroku_ext / _heroku) via MIRROR_EXCLUDED_SCHEMA in the host project when needed.

Management commands

Selective media sync (separate buckets)

When staging must not share the production media bucket, run sync_referenced_media after restore_from_mirror. It collects keys from every FileField / ImageField (honouring private storage location prefixes) plus optional host collectors, then CopyObjects only those keys.

python manage.py sync_referenced_media --dry-run
MEDIA_SYNC_ALLOW=1 python manage.py sync_referenced_media --confirm

Default behaviour skips keys already present on the destination (--skip-existing). Missing source keys are counted and skipped (common when DB rows outlive deleted objects).

Anonymising PII media

List models or fields that must not be copied as-is. Those keys are not copied from production; instead a placeholder is PutObject'd at the same destination key. Image/PDF placeholders are seeded from the source object's ETag (content fingerprint) so they stay visually distinct without copying real bytes.

# settings.py
MIRRORING_ANONYMISE_MEDIA_FIELDS = [
    "listing.shipment",  # dispatch/return labels + courier XML
    "ebay.ebaycoupondownload",  # coupon transaction CSVs
    "data_reporting.exporteddata",  # admin exports
    # or field-level: "reskinned_inventory.picture.preview",
]
# Optional override; return None to fall back to suffix defaults:
# MIRRORING_ANONYMISE_MEDIA_PROVIDER = "myapp.media_sync.anonymise_for_ref"

Omit a collector from MEDIA_SYNC_EXTRA_COLLECTORS to skip that path bag entirely.

Admin

MirrorDatabaseState is a read-only singleton watermark (generation + restore time). By default it registers on django.contrib.admin.site. Set MIRRORING_ADMIN_SITE to your project's admin site (for example "core.admin.site") or call mirroring.admin.register_admin(site) yourself with MIRRORING_AUTO_REGISTER_ADMIN = False.

Development

just install
just test
just lint
just coverage

See CONTRIBUTING.md for the full recipe list and release process.