django-mirroring

Django add-on for production database mirror refresh and staging restore. Uses Dumpling for in-stream anonymisation and Postgres shadow-database cutover so consumers never see a half-loaded mirror.

Links

• Quickstart (Heroku)
• Changelog
• Contributing
• Security
• Issues

Supported versions

This package supports Django 6.0 and Python 3.12+. Dependencies are compatible-release pinned; see Pinned dependencies.

Installation

uv add "django-mirroring~=0.2.4"
# or: pip install "django-mirroring~=0.2.4"

Add "mirroring" to INSTALLED_APPS and run migrations:

python manage.py migrate mirroring

Quickstart (Heroku)

End-to-end setup for a Django app that already runs on Heroku Postgres + S3. Substitute your app names and plan sizes. The package does not invent hostnames or “safe” targets — you choose which URLs and buckets are written.

Architecture once configured:

production follower ──pg_dump──► Dumpling ──► mirror DB (prod addon)
                                              │
                         restore_from_mirror ──┘──► staging primary
                         sync_referenced_media ──► staging media bucket

1. Install and wire settings

uv add "django-mirroring~=0.2.4"
python manage.py migrate mirroring

In the host project (typical pattern):

# settings.py
import mirroring.defaults as mirror_defaults
from myproject import mirror_policy  # project-owned lists + Dumpling path

INSTALLED_APPS = [
    # ...
    "mirroring",
]

MIRRORING_ADMIN_SITE = "myproject.admin.site"  # optional custom AdminSite
MIRROR_DUMPLING_CONFIG = mirror_defaults.mirror_dumpling_config_path(
    default=mirror_policy.DEFAULT_DUMPLING_CONFIG,
)
MIRROR_EXCLUDED_SCHEMA = mirror_policy.MIRROR_EXCLUDED_SCHEMA  # e.g. heroku_ext
MIRROR_EXCLUDED_TABLE_DATA = mirror_defaults.build_mirror_excluded_table_data(
    mirror_policy.MIRROR_EXTRA_EXCLUDED_TABLE_DATA,
)
MIRROR_RETAIN_MONTHS = mirror_defaults.mirror_retain_months()
MIRROR_ROW_RETAIN = mirror_policy.MIRROR_ROW_RETAIN
MIRROR_RESTORE_STAFF_EMAIL_DOMAINS = mirror_policy.mirror_restore_staff_email_domains()

2. Mirror database (production)

Provision the mirror database

Create a separate Postgres addon on the production app (not a fork of live data for day-to-day use — refresh will replace its contents nightly). The role behind MIRROR_DATABASE_URL needs CREATEDB (shadow load + rename cutover).

# Example: empty Standard-0 on the production app (adjust plan as needed)
heroku addons:create heroku-postgresql:standard-0 \
  -a your-app-production \
  --as MIRROR_DATABASE

heroku pg:wait -a your-app-production
heroku config:get MIRROR_DATABASE_URL -a your-app-production

Heroku will set MIRROR_DATABASE_URL when you use --as MIRROR_DATABASE. If you attach under another colour name, copy that URL into MIRROR_DATABASE_URL yourself.

Point the dump source at a full-access follower

Prefer a follower/replica of the primary so pg_dump does not compete with writes, and so the role can SELECT every table Dumpling anonymises.

# Example: follower of DATABASE_URL on the same app
heroku addons:create heroku-postgresql:standard-0 \
  -a your-app-production \
  --as MIRROR_SOURCE_DATABASE \
  -- --follow DATABASE_URL

heroku pg:wait -a your-app-production
# Ensure MIRROR_SOURCE_DATABASE_URL is set (Heroku sets it with --as above)
heroku config:get MIRROR_SOURCE_DATABASE_URL -a your-app-production

Do not use a restricted / allow-listed read role (missing SELECT on PII tables) as the dump source — the dump will be incomplete or fail.

Exclude noisy / non-portable table data

Omit provider schemas and empty high-churn or secret tables (schema still created). Common Heroku + Django starting point:

# myproject/mirror_policy.py
MIRROR_EXCLUDED_SCHEMA = ["heroku_ext", "_heroku"]

MIRROR_EXTRA_EXCLUDED_TABLE_DATA = [
    "django_session",
    "django_admin_log",
    # tokens, rate-limit buckets, task trackers, bulk exports, …
]

Wire with build_mirror_excluded_table_data(...) as shown above. Optionally set MIRROR_ROW_RETAIN + MIRROR_RETAIN_MONTHS so refresh keeps only recent rows for large timestamped tables (orders, etc.).

Add a Dumpling policy and MIRROR_DUMPLING_CONFIG

Anonymisation rules are project-owned TOML (not shipped in this package).

# Draft from a fresh plain-SQL dump when the schema is ready (dumpling-cli >= 0.9):
pg_dump "$MIRROR_SOURCE_DATABASE_URL" -Fp --no-owner --no-acl -f /tmp/src.sql
dumpling scaffold-config -i /tmp/src.sql --infer-json-paths -o myproject/dumplingconf.toml

Edit rules for emails, names, addresses, JSON paths, etc. Keep salts out of git:

heroku config:set DUMPLING_GLOBAL_SALT="$(openssl rand -hex 32)" -a your-app-production

Point Django at the file (path on the slug, or resolve via mirror_dumpling_config_path(default=...)). Set staff domains used both for username keep on the mirror and rematerialisation on restore:

heroku config:set MIRROR_RESTORE_STAFF_EMAIL_DOMAINS="example.com" -a your-app-production

Include a # MIRROR_STAFF_USERNAME_KEEP_RULE marker in the TOML where the refresh command should inject the staff username keep rule (see host examples).

First refresh (dry-run, then live)

heroku run python manage.py refresh_database_mirror --dry-run -a your-app-production
# When ready (scheduler or one-off):
heroku run python manage.py refresh_database_mirror --confirm -a your-app-production

Schedule refresh_database_mirror nightly on production once dry-runs look good.

Read-only URL for staging (store on production for safekeeping)

Staging should restore from the mirror using a credential that cannot DROP/CREATE databases or write production data. Create a read-only role on the mirror instance, put the URL in production config as MIRROR_DATABASE_READONLY_URL (documentation / copy source only — this package does not read that name), then set staging’s MIRROR_DATABASE_URL to that read-only URL.

# On the mirror Postgres (example — adjust role/db names):
# CREATE ROLE mirror_readonly LOGIN PASSWORD '...';
# GRANT CONNECT ON DATABASE <mirror_db> TO mirror_readonly;
# GRANT USAGE ON SCHEMA public TO mirror_readonly;
# GRANT SELECT ON ALL TABLES IN SCHEMA public TO mirror_readonly;
# ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO mirror_readonly;

heroku config:set MIRROR_DATABASE_READONLY_URL='postgres://mirror_readonly:...@.../<mirror_db>?sslmode=require' \
  -a your-app-production

3. Mirror media (production safekeeping + policy)

When staging uses a different S3 bucket than production, restore only brings DB rows; object bytes are copied afterward with sync_referenced_media.

Read-only AWS credentials (store on production)

Create an IAM user/role that can s3:GetObject / s3:ListBucket on the production media bucket only. Store the keys on the production app for safekeeping (operators copy them to staging when configuring sync):

heroku config:set \
  MEDIA_SYNC_SOURCE_AWS_ACCESS_KEY_ID=... \
  MEDIA_SYNC_SOURCE_AWS_SECRET_ACCESS_KEY=... \
  -a your-app-production

(Your host may map those into the process env that boto3 uses for the source client, or you may pass a dedicated profile — the package reads MEDIA_SYNC_SOURCE_BUCKET / region and the usual AWS credential chain.)

Skip or anonymise selected media

In host settings:

# Skip CopyObject for PII-bearing models/fields; plant placeholders instead.
MIRRORING_ANONYMISE_MEDIA_FIELDS = [
    "listing.shipment",  # shipping labels
    "data_reporting.exporteddata",
    # or field-level: "myapp.model.field",
]
# Optional: "myapp.media_sync.anonymise_for_ref"
# MIRRORING_ANONYMISE_MEDIA_PROVIDER = "..."

# Extra keys not stored in FileField/ImageField (JSON path bags, CharFields, …).
# Omit a collector entirely to skip that bag.
MEDIA_SYNC_EXTRA_COLLECTORS = [
    "myapp.media_sync.collect_extra_keys",
]

Anything not referenced from the restored DB (and collectors) is never copied — that is how “skip certain media completely” works at scale.

4. Mirror restore (staging)

Env vars on the staging app

heroku config:set \
  MIRROR_DATABASE_URL="$(heroku config:get MIRROR_DATABASE_READONLY_URL -a your-app-production)" \
  MIRROR_RESTORE_TARGET_DATABASE_URL="$(heroku config:get DATABASE_URL -a your-app-staging)" \
  MIRROR_RESTORE_STAFF_EMAIL_DOMAINS="example.com" \
  MEDIA_SYNC_SOURCE_BUCKET=your-prod-media-bucket \
  -a your-app-staging

Ensure staging dynos have pg_dump / psql at least major MIRRORING_POSTGRES_CLIENT_MAJOR (default 15).

Run restore, then media sync

heroku run python manage.py restore_from_mirror --dry-run -a your-app-staging

MIRROR_RESTORE_ALLOW=1 heroku run \
  python manage.py restore_from_mirror --confirm -a your-app-staging

# Optional rollback while {db}_preswap still exists:
# MIRROR_RESTORE_ALLOW=1 heroku run python manage.py revert_mirror_restore --confirm -a your-app-staging

heroku run python manage.py sync_referenced_media --dry-run -a your-app-staging
MEDIA_SYNC_ALLOW=1 heroku run \
  python manage.py sync_referenced_media --confirm -a your-app-staging

restore_from_mirror rematerialises staff email / password / names from the pre-restore staging DB onto users matched by UserModel.USERNAME_FIELD (usernames were kept on the mirror for allowlisted email domains).

5. Checklist

• mirroring installed, migrated, admin watermark visible
• Production: MIRROR_SOURCE_DATABASE_URL (follower), MIRROR_DATABASE_URL (CREATEDB)
• Production: Dumpling TOML + DUMPLING_GLOBAL_SALT + exclude/retain lists
• Production: nightly refresh_database_mirror
• Production: MIRROR_DATABASE_READONLY_URL (+ optional media source IAM) stored
• Staging: restore target + readonly mirror URL + staff domains
• Staging: restore_from_mirror then sync_referenced_media with anonymise list

Pinned dependencies

Every dependency is pinned to a compatible release (~=) so a mirror run cannot silently pick up an incompatible Dumpling policy format or AWS SDK behaviour.

dumpling-cli ships the dumpling executable, so installing this package also pins the CLI. The commands verify it at start-up and refuse to run against a different minor series.

Postgres client tools (pg_dump, psql) are system packages, not pip dependencies, so they are pinned as a minimum major — pg_dump refuses to dump a server newer than itself, while newer clients read older servers fine:

Commands fail fast with a clear error when a tool is missing or too old, rather than part-way through a dump.

Dumpling policy (project-owned)

Anonymisation rules live in a project-owned Dumpling TOML file, not inside this package. Point the refresh command at it with MIRROR_DUMPLING_CONFIG (env) or MIRROR_DUMPLING_CONFIG (Django setting). The effective config may also embed generated row_filters and staff username keep rules at refresh time.

Settings overview

Configure via Django settings and/or environment variables (host projects typically wire env → settings in one place).

DUMPLING_GLOBAL_SALT must be set in the environment for Dumpling lint/run.

Endpoint guidance (operators)

Refresh and restore only refuse when source and destination resolve to the same host/port/database (restore/revert also require MIRROR_RESTORE_ALLOW=1 and --confirm). Which databases those env URLs point at is otherwise an operator responsibility — document your project's URLs carefully; there is no hostname allow/block list in the package:

• Prefer pointing MIRROR_SOURCE_DATABASE_URL at a full-access follower or offline replica so pg_dump can read every table Dumpling anonymises, and so refresh load does not compete with live writes. Dumping the primary is allowed but not recommended under load.
• Prefer pointing MIRROR_DATABASE_URL at a separate mirror database from the live app primary. The destination role needs CREATEDB for shadow load + rename cutover.
• Point MIRROR_RESTORE_TARGET_DATABASE_URL only at a disposable staging (or equivalent) database you intend to replace.
• A restricted / allow-listed role that cannot SELECT PII tables will produce an incomplete or failing dump — use full-access credentials for the source.
• Put sslmode on connection URLs when the server requires TLS (no hostname-based SSL inference).
• Omit provider schemas (e.g. Heroku's heroku_ext / _heroku) via MIRROR_EXCLUDED_SCHEMA in the host project when needed.

Management commands

Selective media sync (separate buckets)

When staging must not share the production media bucket, run sync_referenced_media after restore_from_mirror. It collects keys from every FileField / ImageField (honouring private storage location prefixes) plus optional host collectors, then CopyObjects only those keys.

python manage.py sync_referenced_media --dry-run
MEDIA_SYNC_ALLOW=1 python manage.py sync_referenced_media --confirm

Default behaviour skips keys already present on the destination (--skip-existing). Missing source keys are counted and skipped (common when DB rows outlive deleted objects).

Anonymising PII media

List models or fields that must not be copied as-is. Those keys are not copied from production; instead a placeholder is PutObject'd at the same destination key. Image/PDF placeholders are seeded from the source object's ETag (content fingerprint) so they stay visually distinct without copying real bytes.

# settings.py
MIRRORING_ANONYMISE_MEDIA_FIELDS = [
    "listing.shipment",  # dispatch/return labels + courier XML
    "ebay.ebaycoupondownload",  # coupon transaction CSVs
    "data_reporting.exporteddata",  # admin exports
    # or field-level: "reskinned_inventory.picture.preview",
]
# Optional override; return None to fall back to suffix defaults:
# MIRRORING_ANONYMISE_MEDIA_PROVIDER = "myapp.media_sync.anonymise_for_ref"

Omit a collector from MEDIA_SYNC_EXTRA_COLLECTORS to skip that path bag entirely.

Admin

MirrorDatabaseState is a read-only singleton watermark (generation + restore time). By default it registers on django.contrib.admin.site. Set MIRRORING_ADMIN_SITE to your project's admin site (for example "core.admin.site") or call mirroring.admin.register_admin(site) yourself with MIRRORING_AUTO_REGISTER_ADMIN = False.

Development

just install
just test
just lint
just coverage

See CONTRIBUTING.md for the full recipe list and release process.