swiss-ai-hub-backup
Backup, restore, and continuous Postgres maintenance for the AI-Hub data services. Runs as an independent Dagster instance (3 containers: gRPC code server, daemon, webserver) inside the Docker Compose project.
Requires the Docker socket (/var/run/docker.sock) to discover and manage platform containers via the
com.docker.compose.project label.
Dagster UI: http://localhost:3004
What this package does
- Daily backup (1 AM Europe/Zurich) of PostgreSQL × 2, Milvus, Neo4j, ClickHouse, Valkey, NATS to S3 (SeaweedFS).
- Restore from any prior backup, partition-selected through the Dagster UI.
- Weekly Postgres cleanup (Sundays 3 AM) — prunes verbose Python logs and transient framework-internal events from
the
dagsterdatabase'sevent_logstable. UI-safe by construction:ASSET_MATERIALIZATION,STEP_SUCCESS,STEP_FAILURE, therunstable, and the asset catalog are never touched. - Monthly
pg_repack(first Sunday 4 AM) — reclaims disk pages on the heavy Dagster tables. The platform's Postgres image now shipspostgresql-17-repackand the extension is registered on first init.
All four jobs (backup, restore, cleanup, repack) carry a postgres-mutex=true tag. The Dagster run coordinator caps
concurrency for that tag at one, so the jobs serialise without blocking other instance traffic.
Configuration
Backup retention is set via BACKUP_RETENTION_DAYS and BACKUP_MINIMUM_KEEP. Maintenance is configured via
DAGSTER_DEBUG_LOG_RETENTION_DAYS (7), DAGSTER_INFO_LOG_RETENTION_DAYS (60), DAGSTER_WARNING_LOG_RETENTION_DAYS
(60), DAGSTER_UNIMPORTANT_EVENT_RETENTION_DAYS (30), DAGSTER_CLEANUP_BATCH_LIMIT (1,000,000 — per-DELETE row cap),
and MAINTENANCE_DISABLED (false — kill switch).
DocumentDB Catalog Maintenance
The PostgreSQL handler hardcodes DocumentDB extension catalog tables and sequences in _DOCUMENTDB_CATALOG_TABLES and
_DOCUMENTDB_CATALOG_SEQUENCES (services/postgres.py). After upgrading the DocumentDB extension, verify the list is
still complete:
SELECT c.relname, c.relkind FROM pg_class c
JOIN pg_depend d ON c.oid = d.objid
JOIN pg_extension e ON d.refobjid = e.oid
WHERE e.extname = 'documentdb' AND c.relkind IN ('r', 'S')
ORDER BY c.relkind, c.relname;
Run this against the postgres database on postgres-ferretdb. relkind = 'r' = tables, 'S' = sequences.
License
Copyright (C) 2024-2026 bbv Software Services AG.
AGPL-3.0-or-later — see packages/backup/LICENSE. For the full per-package matrix (root, AGPL, and proprietary packages), see LICENSES.md.
2 maintainers
Metadata
2 maintainers
Release files for swiss-ai-hub-backup 0.291.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| swiss_ai_hub_backup-0.291.5.tar.gz | 49.6 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| swiss_ai_hub_backup-0.291.5-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 123.3 kB
Release files / swiss_ai_hub_backup-0.291.5.tar.gz
| Download URL | swiss_ai_hub_backup-0.291.5.tar.gz |
|---|---|
| Size | 49.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
3e17b083cabadbee8a13be46aea6e2e6d8eb8607ee957ab8f033f5fa00a574dc
|
|
BLAKE2b-256 checksum How to use checksums |
9b01b4f4709d9dd7c1bde451db58964cf7394cf886476fc60f666056be38da17
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.10.4 {"installer":{"name":"uv","version":"0.10.4","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"22.04","id":"jammy","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / swiss_ai_hub_backup-0.291.5-py3-none-any.whl
| Download URL | swiss_ai_hub_backup-0.291.5-py3-none-any.whl |
|---|---|
| Size | 73.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e7a806d41d64216b8a4f68713832a1dbe7a915d1c4c7fba0a1f582692bd6568b
|
|
BLAKE2b-256 checksum How to use checksums |
606e8e9aae0cad0930198ef173b3afb0f5a095effc1f0ee448ad599c5e07e770
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.10.4 {"installer":{"name":"uv","version":"0.10.4","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"22.04","id":"jammy","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
