Omnibackup - Universal Database Backup and Restore System

Production-ready Python library for database backup operations

---

Features

• Multi-Database Support: PostgreSQL, MySQL/MariaDB, MongoDB
• Async Task Processing: Celery + Redis for background operations
• Scheduling: Daily, weekly, hourly, or custom cron-like schedules
• Storage Backends: Local filesystem, S3, GCS (extensible)
• Compression: Gzip compression for all backups
• Retention Policies: Automatic cleanup of old backups
• Metadata Tracking: SQLite/JSON metadata storage
• CLI Interface: Easy-to-use command-line tool
• Unified API: Simple Python API for programmatic use

---

Quick Start

Installation

pip install omnibackup

Configuration

Create a .env file or set environment variables:

# Storage
OMNIBACKUP_STORAGE_PATH=/var/backups/omnibackup
OMNIBACKUP_METADATA_BACKEND=sqlite

# Celery
CELERY_BROKER_URL=redis://localhost:6379/0
CELERY_RESULT_BACKEND=redis://localhost:6379/0

Or use a YAML config file (omnibackup.yaml):

storage:
  path: /var/backups/omnibackup
  metadata_backend: sqlite

celery:
  broker_url: redis://localhost:6379/0
  result_backend: redis://localhost:6379/0
  timezone: UTC

logging:
  level: INFO

---

Usage

Python API

from omnibackup import BackupManager

# Initialize manager
backup = BackupManager(storage_path="/backups")

# Run a backup
result = backup.run(
    database="postgres",
    uri="postgresql://user:pass@localhost/mydb",
    destination="/backups/postgres",
    compress=True,
    retention_days=30,
)

print(f"Backup ID: {result.backup_id}")
print(f"Size: {result.size_bytes} bytes")
print(f"Path: {result.path}")

# Restore from backup
restore_result = backup.restore(
    backup_id=result.backup_id,
    target_uri="postgresql://user:pass@localhost/mydb_restore",
)

CLI Usage

Run a backup

# PostgreSQL backup
omnibackup run postgres postgresql://user:pass@localhost/db --retention 30

# MySQL backup with async processing
omnibackup run mysql mysql://user:pass@localhost/db --async --wait

# MongoDB backup to S3 (placeholder)
omnibackup run mongodb mongodb://localhost:27017/mydb --storage s3

Restore from backup

omnibackup restore <backup-id> postgresql://user:pass@localhost/newdb

List backups

omnibackup list-backups
omnibackup list-backups --database postgres --status success

Schedule backups

# Daily backup
omnibackup schedule "daily-postgres" postgres postgresql://user:pass@localhost/db --frequency daily

# Weekly backup on Sundays
omnibackup schedule "weekly-mysql" mysql mysql://user:pass@localhost/db --frequency weekly

# Custom cron expression (every 6 hours)
omnibackup schedule "frequent-backup" postgres postgresql://user:pass@localhost/db \
    --frequency custom --cron "0 */6 * * *"

List schedules

omnibackup list-schedules
omnibackup list-schedules --enabled-only

Check task status

omnibackup status --task <task-id>

Start Celery worker

omnibackup worker --concurrency 4

---

Running Celery Workers

Start Redis:

redis-server

Start Celery worker:

omnibackup worker --loglevel info --concurrency 4

Or using Celery directly:

celery -A omnibackup.tasks worker --loglevel=info

---

Architecture

omnibackup/
├── core/           # Core abstractions and logic
│   ├── base.py     # BaseBackupAdapter, StorageBackend
│   ├── manager.py  # BackupManager
│   ├── metadata.py # Metadata tracking
│   └── task_manager.py  # Celery task management
├── adapters/       # Database adapters
│   ├── postgres.py # PostgreSQL adapter (pg_dump/pg_restore)
│   ├── mysql.py    # MySQL adapter (mysqldump/mysql)
│   └── mongodb.py  # MongoDB adapter (mongodump/mongorestore)
├── storage/        # Storage backends
│   ├── local.py    # Local filesystem
│   └── cloud.py    # S3/GCS placeholders
├── tasks/          # Celery tasks
│   └── backup.py   # Async backup/restore tasks
├── scheduler/      # Scheduling system
│   └── scheduler.py  # Cron-like scheduling
├── cli/            # Command-line interface
│   └── main.py     # Typer CLI
├── config/         # Configuration management
│   ├── settings.py # Settings from env/YAML
│   └── celery_config.py  # Celery configuration
└── utils/          # Utility functions
    └── helpers.py  # Helper functions

---

Database URI Formats

PostgreSQL

postgresql://user:password@host:port/database
postgres://user:password@host:port/database

MySQL

mysql://user:password@host:port/database
mariadb://user:password@host:port/database

MongoDB

mongodb://user:password@host:port/database
mongodb://user:password@host:port/database?authSource=admin&replicaSet=rs0

---

Storage Backends

Local (Default)

backup.run(..., storage_backend="local")

S3 (Placeholder)

backup.run(..., storage_backend="s3")

Full S3 implementation requires boto3 and AWS credentials.

GCS (Placeholder)

backup.run(..., storage_backend="gcs")

Full GCS implementation requires google-cloud-storage.

---

Retention Policies

# Keep backups for 30 days
backup.run(..., retention_days=30)

# CLI
omnibackup run postgres <uri> --retention 30

Old backups are automatically cleaned up after the retention period.

---

Async Operations

# Run backup asynchronously
result = backup.run(..., async_task=True)
print(f"Task ID: {result.id}")

# Check status
status = backup.task_manager.get_task_status(result.id)
print(status)

---

Development

Setup

git clone https://github.com/yourusername/omnibackup.git
cd omnibackup
pip install -e .

Running Tests

pytest

Code Style

black omnibackup/
isort omnibackup/
flake8 omnibackup/

---

Environment Variables

Variable	Description	Default
OMNIBACKUP_STORAGE_PATH	Base storage path	/tmp/omnibackup
OMNIBACKUP_METADATA_BACKEND	Metadata backend (sqlite/json)	sqlite
OMNIBACKUP_LOG_LEVEL	Logging level	INFO
CELERY_BROKER_URL	Celery broker URL	redis://localhost:6379/0
CELERY_RESULT_BACKEND	Celery result backend	redis://localhost:6379/0
OMNIBACKUP_S3_BUCKET	S3 bucket name	-
OMNIBACKUP_S3_REGION	S3 region	us-east-1
OMNIBACKUP_S3_ACCESS_KEY	AWS access key	-
OMNIBACKUP_S3_SECRET_KEY	AWS secret key	-
OMNIBACKUP_GCS_BUCKET	GCS bucket name	-
OMNIBACKUP_GCS_PROJECT	GCP project ID	-
OMNIBACKUP_GCS_CREDENTIALS	Path to service account JSON	-

---

License

MIT License - See LICENSE file for details.

---

Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests
5. Submit a pull request

---

Roadmap

• Full S3/GCS implementation
• Web dashboard
• Email notifications
• Backup encryption
• Incremental backups
• Backup verification
• Multi-region replication

---

Support

For issues and questions:

• GitHub Issues: https://github.com/yourusername/omnibackup/issues
• Documentation: https://omnibackup.readthedocs.io