celery-redis-statedb 0.2.0

Redis-based statedb persistence for Celery workers

Celery Redis StateDB

Redis-based state persistence for Celery workers in containerized environments.

This package provides a Redis-based backend for Celery's worker state persistence, replacing the default filesystem-based shelve storage. It's designed to:

• ✅ Persist revoked task state across container restarts
• ✅ Isolate state per worker using unique key prefixes
• ✅ Eliminate concurrency issues through per-worker key isolation
• ✅ Handle Redis connection failures gracefully
• ✅ Integrate seamlessly with existing Celery applications

Features

• Distributed State Persistence: Store revoked tasks in Redis instead of local files
• Per-Worker Isolation: Each worker maintains its own state using unique key prefixes based on hostname
• Concurrency-Safe: No race conditions since workers never write to the same keys
• Easy Integration: Simple Celery bootstep that works with existing applications
• Migration Support: Migrate existing file-based statedb to Redis with automatic backup

Installation

# Using pip
pip install celery-redis-statedb

# Using uv
uv add celery-redis-statedb

Quick Start

Method 1: Command Line (Easiest)

The easiest way to use Redis state persistence is via the --redis-statedb command line option:

celery -A myapp worker --redis-statedb=redis://localhost:6379/0

Method 2: Programmatic Installation (Recommended)

Use the install_redis_statedb() convenience function in your app:

Basic usage (with defaults):

from celery import Celery
from celery_redis_statedb import install_redis_statedb

app = Celery('myapp')

# Install Redis state persistence
install_redis_statedb(app)

# Start worker with Redis statedb
# celery -A myapp worker --redis-statedb=redis://localhost:6379/0

# Or migrate from existing file-based statedb
# celery -A myapp worker --redis-statedb=redis://localhost:6379/0 --migrate-statedb=/path/to/worker.db

With custom configuration:

from celery import Celery
from celery_redis_statedb import install_redis_statedb

app = Celery('myapp')

# Optional: Configure via app.conf
app.conf.redis_state_key_prefix = 'myapp:worker:state:'

# Install bootstep
install_redis_statedb(app)

# Start worker
# celery -A myapp worker --redis-statedb=redis://localhost:6379/0

Method 3: Environment Variables

Configure additional settings via environment variables:

# Optional: Custom key prefix (default: celery:worker:state:)
export CELERY_REDIS_STATE_KEY_PREFIX=myapp:worker:state:

# Start worker
celery -A myapp worker --redis-statedb=redis://localhost:6379/0 --loglevel=info

Migrating from File-Based StateDB

If you have an existing Celery worker using file-based state persistence (--statedb), you can migrate to Redis:

celery -A myapp worker \
  --redis-statedb=redis://localhost:6379/0 \
  --migrate-statedb=/path/to/worker.db

What happens during migration:

1. Creates a timestamped backup: backup-worker.TIMESTAMP.db
2. Migrates revoked tasks and clock value to Redis
3. Renames original file to: worker.db.migrated.TIMESTAMP

The migration:

• ✅ Supports Celery statedb formats v1, v2, and v3
• ✅ Preserves logical clock synchronization
• ✅ Creates double redundancy (backup + renamed file)
• ✅ Fails safely if file cannot be renamed (prevents stale data overwrite)
• ⚠️ Linux-only: Assumes single-file shelve database (gdbm). On systems using ndbm/dumbdbm (BSD, macOS), multiple files (.dir, .dat) may not be fully handled.

After successful migration:

• Backup file: /path/to/backup-worker.TIMESTAMP.db
• Renamed original: /path/to/worker.db.migrated.TIMESTAMP
• Original path: /path/to/worker.db (removed)

⚠️ IMPORTANT: Remove --migrate-statedb After First Run

After the migration completes successfully, remove the --migrate-statedb flag from your worker command:

# After migration, restart with only --redis-statedb
celery -A myapp worker --redis-statedb=redis://localhost:6379/0

Keeping --migrate-statedb in your command can cause:

• Unnecessary migration attempts on every restart
• Potential data races if the file still exists
• Risk of corruption if multiple workers attempt migration simultaneously

The migration should only be run once during the transition from file-based to Redis-based state persistence.

Configuration Options

Configuration can be set via Celery app configuration (app.conf) or environment variables:

Configuration Key	Environment Variable	Default	Description
redis_state_key_prefix	CELERY_REDIS_STATE_KEY_PREFIX	celery:worker:state:	Base prefix for Redis keys (worker hostname is appended)

Example using app.conf:

app.conf.redis_state_key_prefix = 'myapp:worker:state:'

Example using environment variables:

export CELERY_REDIS_STATE_KEY_PREFIX=myapp:worker:state:

Revoked Task Persistence

Revoked tasks persist indefinitely in Redis (matching Celery's default behavior with shelve). This ensures that once a task is revoked, it remains revoked even across worker restarts.

Celery's LimitedSet automatically purges expired items based on the maxlen parameter (default: 10000 items). When the set reaches this limit, the oldest items are automatically removed to maintain the size constraint.

Architecture

Per-Worker Key Isolation

Each worker maintains its own state using a unique key prefix based on the worker's hostname:

celery:worker:state:<worker-hostname>:zrevoked
celery:worker:state:<worker-hostname>:clock

This design eliminates concurrency issues since:

• Workers never write to the same Redis keys
• No locks or atomic operations needed
• Simple, fast, and scalable
• Each worker independently manages its own state persistence

When a worker restarts, it loads its own state from its dedicated keys in Redis.

Worker Hostname Configuration

Default Hostname Behavior

By default, Celery generates worker hostnames in the format celery@<system-hostname>. In containerized environments:

• Docker: Uses the container ID (e.g., celery@f9207bc94b22)
• Kubernetes: Uses the pod name (e.g., celery@worker-deployment-7d9f8b6c-xk2p5)
• ECS: Uses the task ID or container hostname

Each container automatically gets a unique hostname, ensuring per-worker key isolation works correctly.

Customizing Worker Hostnames

For better visibility and debugging, you can set custom worker hostnames using the -n or --hostname option:

# Static name with system hostname
celery -A myapp worker -n worker1@%h --redis-statedb=redis://localhost:6379/0

# Using environment variable for replica number (Kubernetes)
celery -A myapp worker -n worker-$REPLICA_ID@%h --redis-statedb=redis://localhost:6379/0

# Custom name with domain
celery -A myapp worker -n [e-mail] --redis-statedb=redis://localhost:6379/0

Hostname Variables:

• %h - Full hostname with domain (e.g., george.example.com)
• %n - Hostname only (e.g., george)
• %d - Domain only (e.g., example.com)

Note: In Supervisor config files, escape % as %% (e.g., %%h).

Multiple Workers on Same Host

When running multiple workers on the same machine, ensure each has a unique hostname:

celery -A myapp worker -n worker1@%h --redis-statedb=redis://localhost:6379/0
celery -A myapp worker -n worker2@%h --redis-statedb=redis://localhost:6379/0
celery -A myapp worker -n worker3@%h --redis-statedb=redis://localhost:6379/0

⚠️ Important: If multiple workers share the same hostname, they will overwrite each other's state in Redis. Celery will also show a DuplicateNodenameWarning. Always ensure unique worker hostnames in production.

Container Deployments

AWS ECS Deployment

Example ECS task definition with custom worker hostname:

{
  "family": "celery-worker",
  "containerDefinitions": [
    {
      "name": "celery-worker",
      "image": "myapp:latest",
      "command": [
        "celery",
        "-A",
        "myapp",
        "worker",
        "-n", "worker@%h",
        "--redis-statedb=redis://redis.cluster.local:6379/0",
        "--loglevel=info"
      ]
    }
  ]
}

Best Practices for ECS:

• Set stopTimeout to allow graceful shutdown (e.g., 120 seconds for 2-minute tasks)

Troubleshooting

Verify Worker Hostnames

To check that each worker has a unique hostname, inspect active workers:

# List all active workers
celery -A myapp inspect active

# Check worker stats (shows hostname for each worker)
celery -A myapp inspect stats

Check Redis Keys

To verify per-worker keys are being created correctly:

# Connect to Redis
redis-cli

# List all worker state keys
KEYS celery:worker:state:*:zrevoked

# Example output:
# 1) "celery:worker:state:worker1@host1:zrevoked"
# 2) "celery:worker:state:worker2@host2:zrevoked"
# 3) "celery:worker:state:worker3@host3:zrevoked"

# Check revoked tasks for a specific worker
GET celery:worker:state:worker1@host1:zrevoked

Common Issues

Issue: Workers overwriting each other's state

• Cause: Multiple workers using the same hostname
• Solution: Set unique hostnames with -n option (e.g., -n worker1@%h, -n worker2@%h)

Issue: DuplicateNodenameWarning on worker startup

• Cause: Another worker with the same nodename is already registered
• Solution: Ensure each worker has a unique hostname

Issue: State not persisting across restarts

• Cause: Worker hostname changes between restarts
• Solution: In containers, use stable hostnames or set custom static names

Testing

# Quick check - run all quality checks at once
make check-all

# Individual commands
make test        # Run tests with coverage
make lint        # Run ruff linter
make typecheck   # Run mypy and ty type checkers
make format      # Format code with ruff
make clean       # Clean up temporary files

# Or run commands directly
uv run pytest --cov=celery_redis_statedb
uv run ruff check celery_redis_statedb/
uv run mypy celery_redis_statedb/
uv run ty check celery_redis_statedb/

Documentation

For detailed information about Celery's state persistence format and how this package implements it:

• Celery Shelve Format Documentation - Comprehensive guide to what Celery stores in its state database