Skip to main content

swiss-ai-hub-backup

The centralized backup, restore, and PostgreSQL-maintenance service for Swiss AI Hub — a self-contained Dagster instance that snapshots every stateful service to S3.

PyPI Python License


What is Swiss AI Hub?

Swiss AI Hub is an open-source, self-hosted AI platform for enterprises. One docker compose up starts ~30 integrated containers across several stateful stores — PostgreSQL, FerretDB, Milvus, Neo4j, ClickHouse, Valkey, and NATS JetStream. This package keeps that data safe.

What is this package?

swiss-ai-hub-backup is the platform's backup/restore and database-maintenance plane. It runs as its own independent Dagster instance (separate from the data pipelines, with its own SQLite storage) and:

  • Backs up PostgreSQL (×2), Milvus, Neo4j, ClickHouse, Valkey, and NATS JetStream to S3 (SeaweedFS) on a schedule — gracefully stopping and restarting the managed containers around each run for consistent snapshots.
  • Restores any service from a chosen backup timestamp.
  • Maintains the platform PostgreSQL online: prunes verbose Dagster event_logs, tunes autovacuum, and runs pg_repack — so deployments stay bounded over time without downtime. A one-off job recompresses FerretDB's existing rows with lz4.

Each stateful service has a BackupHandler (postgres, milvus, neo4j, clickhouse, valkey, nats); the whole thing is wired into a Dagster asset graph by backup_definitions(). Because it operates on the storage layer and needs to stop containers, it needs the Docker containers API (DOCKER_HOST; in the platform compose a socket proxy that allows /containers/... only), which it uses to discover platform containers via their com.docker.compose.project label.

Unlike the other Swiss AI Hub packages, this is an operational service, not a library you build agents/APIs on. It is licensed AGPL-3.0-or-later (the rest of the SDK is Apache-2.0).

Should you use this package?

Most operators don't install it directly — it ships with the platform as the backup-* containers (a gRPC code server, a daemon, and a webserver UI on :3004). You'd reach for this PyPI package to run the backup plane standalone, embed its logic, or extend it — for example, adding a BackupHandler for a stateful service of your own.

What it does

Job Schedule Stops containers?
Full backup (all services → S3) daily Yes (consistent snapshots)
Restore (service ← chosen timestamp) on demand Yes
event_logs cleanup + autovacuum tuning weekly No (online-safe)
pg_repack (reclaim disk) monthly No (online-safe)
FerretDB rows pglz → lz4 by hand No (online-safe)

Installation

pip install swiss-ai-hub-backup
# or
uv add swiss-ai-hub-backup

Requires Python 3.13.


Quick start

The backup plane is a Dagster code location built by backup_definitions():

# my_backup/__init__.py
from swiss_ai_hub.backup.dagster.definitions import backup_definitions

defs = backup_definitions()  # 28 assets, 5 jobs: backup, restore, cleanup, repack, lz4 rewrite

Inspect and run it with the Dagster UI (it keeps its own state in DAGSTER_HOME):

export DAGSTER_HOME=/tmp/backup-dagster && mkdir -p "$DAGSTER_HOME"
set -a && source .env && set +a          # S3 + DB credentials, BACKUP_* settings
dagster dev -m my_backup                 # http://localhost:3000

From the UI you can materialize the online-safe maintenance jobs (cleanup, pg_repack, the lz4 rewrite) against a running stack without disruption. The full backup/restore jobs stop and restart containers, so run those deliberately — and note they need access to the Docker socket and to all the stateful services. dagster definitions validate -m my_backup loads the whole code location without running anything (a fast CI/sanity check).

Settings are not auto-loaded from the environment. Connection and BACKUP_* settings are read only when constructed, so export them in the process that runs Dagster (set -a && source .env && set +a).


How it's deployed

In production the backup plane runs as three containers from one image, forming a self-contained Dagster instance:

Container Role Notes
backup-code Dagster gRPC code server (dagster api grpc … :4266) Docker via backup-docker-proxy; gRPC bound to backup only; on data + storage
backup-daemon Dagster daemon runs the schedules and sensors; on backup
backup-webserver Dagster UI (:3004) inspect runs, trigger restores; on backup + backup-ui (with its oauth2-proxy)

Because it needs the Docker containers API and the platform's stateful services, the canonical deployment is the platform's own backup compose. See infra/deployment and the documentation for the full container setup, retention config, and the BACKUP_* environment variables. If you run your own variant, mirror that three-container shape and give the code server a DOCKER_HOST that allows the /containers/... API (including POST).

Extending — add a service to back up

Implement the BackupHandler ABC for your service in services/, then register it in HANDLER_FACTORIES — the Dagster asset wiring picks it up automatically (handlers are synchronous by design):

from swiss_ai_hub.backup.services.base import BackupHandler


class MyServiceHandler(BackupHandler):
    def backup(self, context) -> ...: ...  # dump your service's state to S3
    def restore(self, context) -> ...: ...  # restore it from a backup

If the handler needs Docker access, type-hint a DockerManager parameter in __init__ and the factory injects it. The maintenance subsystem follows the same pattern (MaintenanceHandler + CLEANUP_HANDLER_NAMES). See the documentation for the full handler contract.


License

AGPL-3.0-or-later — see packages/backup/LICENSE. Note this differs from the Apache-2.0 SDK packages; for the full per-package license matrix, see LICENSES.md.


Part of Swiss AI Hub. Built in Switzerland by bbv Software Services.

Metadata

Release files for swiss-ai-hub-backup 0.325.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for swiss-ai-hub-backup 0.325.2
File Size Uploaded Provenance Actions
swiss_ai_hub_backup-0.325.2.tar.gz 62.7 kB Without provenance Details for swiss_ai_hub_backup-0.325.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for swiss-ai-hub-backup 0.325.2
File Interpreter ABI Platform Actions
swiss_ai_hub_backup-0.325.2-py3-none-any.whl Python 3 none any Details for swiss_ai_hub_backup-0.325.2-py3-none-any.whl

Total release size: 147.0 kB

Release files / swiss_ai_hub_backup-0.325.2.tar.gz

Download URL swiss_ai_hub_backup-0.325.2.tar.gz
Size 62.7 kB
Tags Source
SHA-256 checksum
How to use checksums
606cc1dcd5aef460e247455c8b96866a3dafe4440f7abf941f8ecd9a759762a8
BLAKE2b-256 checksum
How to use checksums
2e7e01c694d5bb15fee171e5b08edf27590215b2f6e8192a6ead9219e171664d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / swiss_ai_hub_backup-0.325.2-py3-none-any.whl

Download URL swiss_ai_hub_backup-0.325.2-py3-none-any.whl
Size 84.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
b0e69be1f9b8ab7e9dd2006c78dddc86ece4621e3f7218e7b8c3659929945b26
BLAKE2b-256 checksum
How to use checksums
3efc4d386485507b4ebe636245fa151e9c8b41ea3a59630020590a3cecbe0986
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for swiss-ai-hub-backup 0.325.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.325.2 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page