scalio-backup

Encrypted export and import of a whole Scalio app instance, shared by the Scalio apps (Nexus first, then Citadel, Foundry, Rosetta, Beacon and Pulse, ADR 0069 wartung/W5). Used for backup/restore and for moving an instance between self-hosted and SaaS. Concept and decisions E1-E12: https://claude.ai/artifact/Ej8DFrgi4hcFDQ69dvKWZ7

Nothing in this package is secret. The password is the customer's; without it nobody (Scalio included) can open a bundle.

Bundle format (.scaliobak)

• Outside: standard age v1 with a passphrase (scrypt, work factor 2^18), ChaCha20-Poly1305 in 64 KiB chunks. age -d backup.scaliobak > backup.tar opens it without Scalio.
• Inside: a tar stream. manifest.json first (format scalio-backup/1, product, app and schema version, source, options, SHA-256 per part), then collections/<name>.jsonl (MongoDB Extended JSON, canonical, so dates, Int64 and ObjectIds stay exact), meta.jsonl, and for file folders files/<dir>.index.jsonl plus the files. The import checks every hash; a changed or incomplete bundle is refused before anything is written.

Classes

Every collection (and every document of a mixed meta collection; meta_key names its id field, or several, e.g. ("id", "_id") when the app and its packages mix them) has one class: config (always exported), data (exported unless "configuration only"), cache, ephemeral, instance (never exported). Each app keeps its classes in its own backup_spec.py and a test that fails on an unclassified collection.

Use in an app

from scalio_backup import AppSpec, Preserve, apply_import, export_instance, open_bundle, recover, rollback, purge

SPEC = AppSpec(product="scalio-nexus", app_version=VERSION, collections=..., meta_ids=...,
               secret_fields={"settings": ("entra.clientSecret", ...)}, file_dirs={"hr": HR_UPLOAD_DIR},
               schema_version=current_schema, export_hook=decrypt_instance_secrets, import_hook=encrypt_again)

# files the instance keeps encrypted with its own key (Rosetta's raw exports, W6): file_export_hook(dir, rel, path,
# work_dir) returns a plain copy in work_dir (or None: the file as it is), file_import_hook(dir, rel, path) encrypts
# the file in the opened bundle again with the target's key, before anything is swapped

# export (background job): writes the encrypted bundle to a file object
manifest = await export_instance(db, SPEC, out, password, include_data=True, include_secrets=True,
                                 created_by=user_email, source={"mode": "selfhosted", "host": host, "tenant": tid})

# import, step 1: decrypt and verify into a work folder, nothing changes yet (WrongPassword, BundleError,
# IncompatibleBundle when the product differs or the bundle is newer than this instance)
bundle = open_bundle(upload, password, SPEC, target_schema=await current_schema(db))
show(bundle.preview())          # source, counts, warnings; the app adds its own (other tenant, license limits)

# import, step 2: inside the app's maintenance window (no writes, background jobs paused)
await apply_import(db, SPEC, bundle, preserve=(Preserve("app_users", {"email": me}),), imported_by=me)
bundle.cleanup()
# then: migrations, end all sessions, re-sync caches, show the checklist

await rollback(db, SPEC)        # back to the state before the import (until purge)
await recover(db, SPEC)         # on startup: cleans up an import that stopped half way
await purge(db, SPEC)           # daily: drops the rollback copy after 7 days

How the swap works: documents go into _import_<name>, then each collection is renamed to _preimport_<name> and the staging copy into its place. A journal (_backup_journal) records each step, so an interrupted import is rolled back on the next start. File folders keep their old content in .preimport/.

Routes and the classification test (0.2.0)

scalio_backup.web.backup_router(...) (extra web, FastAPI) gives every app the page Einstellungen -> Sicherung with the same paths as Nexus (/backup/status, /backup/export…, /backup/import/…): the export job in its own thread, the chunked upload (2 GB), the preview with warnings (another tenant needs a confirmation), the import under the maintenance lock (pass scalio-platform maintenance as lock: the API answers 423, no job starts, the import waits for running jobs) and the rollback for 7 days. Confirmation like «Zurücksetzen»: the password when the account has one, and always the second factor. The app gives its pieces as hooks (source_info, preserve, after_import, migrate, preview_extra, audit); call router.backup.startup() once when the API starts and purge_rollback(db, spec, lock) from the worker.

scalio_backup.scan.scan(backend_dir) finds the collections and meta ids in the app's source, check(spec, found) lists what has no class or is stale; each app runs it as a test.

Since 0.2.0 the import keeps the indexes of the collections it replaces (before, they were gone until the app created them again).

Development

pip install -e ".[dev]", then pytest -q and ruff check .. Engine tests use mongomock; with MONGO_URL set they run against a real MongoDB (CI does both). tests/vector.scaliobak is the shared test bundle (tests/make_vector.py); the age layer is checked against rage (pyrage) in both directions.