Django-bgt

Django-orchestrated background threads for Python, built on bgt (a great way to reliably run background threads).

Define background tasks for your Django app.
Use the service decorator to configure how often and where they should run.
Run django runservices in as many places as makes sense.

Job done.

— It's pronounced "Django Beergut"
— It is now!

What is Django-bgt for?

Django apps usually have tasks that should run periodically or constantly in the background. Sometimes it's also important that a service is only run on a single host at a time. django-bgt simplifies these situations, and also ensures that if things go wrong your job is restarted, with exponential back-off between retries.

Install

1. Install the django-bgt package

Install django-bgt:

pip install 'django-bgt'

2. Add to INSTALLED_APPS

    INSTALLED_APPS = [
        ...,
        "django_bgt",
    ]

3. Create the database table

Run python manage.py migrate to create the Lease table.

4. Declare Services

Create a services.py file in an installed app folder, and define the jobs that you want to be run as functions. Use the service decorator to define how often you want the job to be run. Use leader=True if you only want the job running in a single place.

# Only one process at a time runs this.
@service(name="ticker", interval=1, leader=True, lease_ttl=15)
def tick() -> bool:


    return False

5. Run the service process

Use the runservices management command using one of the following approaches:

./manage.py runservices           # Run all services, until SIGINT/SIGTERM
./manage.py runservices ticker    # Run the "ticker" service, until SIGINT/SIGTERM
./manage.py runservices --list    # List the

Run one runservices process per host. Leader-elected services synchronise through the Lease table. Others run everywhere.

Example: Clean up session data

In a long-running Django application, you should periodically clear your session data from your database. Many developers don't do this, and their session data never gets cleared up!

The traditional way to clear sessions is to run the clearsessions management command from some kind of scheduler like cron. But configuring cron gives you another system to manage. django-bgt allows you to manage these kinds of tasks with Python code.

A function to call that management command from looks a bit like this:

from django.core.management import call_command

def clear_expired_sessions():
    call_command("clearsessions")

You can configure django-bgt to run this as a regular background task with a few lines of code.

Step One: Install django-bgt

Here are the instructions to install django-bgt. Come back here when you're ready to define your background services.

Step Two: Create a services.py file in one of your Django apps.

Add the above code to services.py and decorate it with the service decorator:

from django.core.management import call_command

from django_bgt import service

ONE_MINUTE = 60             # in seconds.
ONE_HOUR = 60 * ONE_MINUTE  # The number of seconds in an hour.

@service(interval=ONE_HOUR)
def clear_expired_sessions():
    """ Clear expired sessions, once every hour. """
    call_command("clearsessions")

Now you need to run the runservices management command somewhere. A background thread will be created for each service you've defined.

# Run all services defined in the Django project:
python ./manage.py runservices

Thanks to the magic of bgt, Your service will be immediately restarted if your service function returns True. (If you return False it will wait interval seconds.) If your service function raises an exception, the service will be restarted, but with exponential back-off, so that it doesn't overwhelm your system if there's a persistent problem.

If you're running in an environment with any sort of redundancy, you should probably run runservices on multiple hosts. In this case, the clear_expired_sessions service will run once every hour everywhere runservices is being run. This may not be desirable, so in this case pass leader=True to the service decorator, and only one host will run this service at any time. If that host fails or is simply shut down, another will spawn a new copy of the service.

Other Examples

The following example is similar to the example above, but manually clears all expired Session objects from the database. This will only be useful if you're running database-backed sessions.

from django_bgt import service

# Define a periodic service to clean up old sessions.
# Will run every 60 seconds.
# Service name defaults to the function name.
# Returning False means nothing left to process. Will wait for next wakeup.
@service(interval=60)
def cleanup() -> bool:
    Session.objects.filter(expire_date__lt=timezone.now()).delete()
    return False

The following defines a service that will be run every second, only in one single process in the Django cluster.

# Services can by synchronised by declaring leader=True.
# Only one process at a time runs this.
@service(name="ticker", interval=1, leader=True, lease_ttl=15)
def tick() -> bool:
    ...
    return False

The lease_ttl parameter determines how quickly failover to other service instances may occur. Long running work units will have their lease periodically renewed, and so can outlive the lease_ttl. As per bgt's notes on thread cancellation, it is advisable to keep individual work units bounded in their runtime. For bigger workloads, try to batch the work, returning True to be run again immediately, unless the supervisor has been stopped.