celery-multibeat-scheduler

A Celery Beat scheduler with automatic failover, backed by Redis leader election for multi-server deployments.

Features

• Automatic failover across servers with atomic leader election.
• Configurable lease and failover timing.
• Watchdog detection of stalled schedulers.
• Redis TLS and Sentinel support.
• Extensive test coverage.

Requirements

• Python 3.10+
• Celery 5.5+
• django-celery-beat 2.8.1+
• redis-py 6.4.0+

All Beat instances need access to the same Redis instance and Django database.

Quick start

pip install celery-multibeat-scheduler

In Django settings, with Celery’s namespace="CELERY":

INSTALLED_APPS += ["django_celery_beat"]

CELERY_BEAT_SCHEDULER = "multibeat.django.MultiBeatDatabaseScheduler"
CELERY_MULTIBEAT_URL = "redis://redis:6379/0"
CELERY_MULTIBEAT_LOCK_KEY = "my-service:beat:lock"

Apply migrations:

python manage.py migrate

Start Beat on each server, replacing your_project with your Celery application:

celery -A your_project beat

All instances must use the same schedule database, Redis database, and lock key.

How it works

Leader election

Each Beat instance competes for the same Redis lock. The lock grants a time-limited lease to one instance, which becomes the active scheduler.

Lock acquisition and renewal use an atomic Redis Lua script, so ownership checks and lease updates happen as one operation.

The active scheduler renews its lease in a background thread. Standby instances keep checking for an opportunity to take over.

Automatic failover

When the active scheduler releases the lock or its lease expires, a standby can acquire it.

Before dispatching tasks, the new leader reloads the schedule and execution history from django-celery-beat. Each takeover uses the shared database, rather than the standby’s cached state.

Lease checks prevent an instance with expired leadership from starting a new dispatch.

Watchdog

The lock manager monitors scheduler heartbeats. If the scheduler stalls beyond the watchdog deadline, the process exits with status 1.

Configuration

Settings below use Celery’s CELERY namespace.

Use a different lock key for each independent service or environment.

Lease timing

By default, the active scheduler renews its 60-second lease every 45 seconds. Standby instances attempt acquisition every 30 seconds.

A shorter lease reduces failover time but leaves less room for network delays and process pauses. Refresh and standby intervals scale with the lease, subject to configured minimums.

Redis connections

For TLS, use a rediss:// URL. Additional connection options belong in CELERY_MULTIBEAT_TRANSPORT_OPTIONS.

For Sentinel, list the nodes and set the master name:

CELERY_MULTIBEAT_URL = "sentinel://first:26379/0;sentinel://second:26379/0"
CELERY_MULTIBEAT_TRANSPORT_OPTIONS = {
    "master_name": "beat-master",
}

See the configuration reference for all settings, timing constraints, TLS certificates, Sentinel credentials, and retry policies.

Development

The suite covers leadership changes, dispatch safety, watchdog failures, and failover between live scheduler processes.

From a local checkout:

uv sync
uv run pytest
uv build

CI tests installed wheels on Python 3.10–3.14 against the minimum and latest supported redis-py versions.

File scheduler

The package also includes MultiBeatPersistentScheduler, an optional file scheduler for development and testing:

celery -A your_project beat \
    --scheduler multibeat.schedulers.MultiBeatPersistentScheduler \
    --schedule=/path/to/local.schedule

It uses Redis for leader election but stores the schedule and execution history locally. Give each instance its own file. Without shared history, it is unsuitable for multi-server deployments.