django-quick-jobs 0.2.2

A method of quickly and simply running background jobs in a Django project

Django Quick Jobs

A way of running simple periodic tasks without the use of Cron, Celery, RabbitMQ, Redis, or any external services.

Why was this created

I have a need to run some periodic jobs on the DigitalOcean App Platform, which doesn't have any scheduled job runners and my use cases were too simple to bother with Celery and such. This package gives a simple way to have a jobs.py file in your Django app(s) and then decorating each job with @job_runner.register_job(interval, variance, timeout). These jobs will then all be run via python manage.py run_jobs. Each job will be repeated every interval with an additional random delay between 0 and variance. The variance option is to reduce the impact of any "thundering herds". If timeout is specified then the job runner will be stopped if the job takes longer than the timeout to finish. The only required parameter to register_job is the interval. All times (interval, variance, and timeout) can be integers, floats, or timedeltas.

Jobs are not coordinated across multiple instances of run_jobs - the individual jobs need to be designed to handle concurrency on their own. Strategies for this would be to use select_for_update, a serializable isolation level, or some external locking mechanics.

This library is best used for smaller-scale sites where Celery and the like is overkill. Once you are worrying about large numbers of jobs or the performance of querying the database for any ready work, it is probably time to move to a more robust tool.

Sample use cases

Recalculating data

We might have some model that sets a needs_recalculation field. We could have a periodic job that queries everything that has needs_recalculation set to true and perform some calculation that takes a long time - such as updating other related data models. The models we are updating should use select_for_update so that multiple instances of the job runner don't try to recalculate the same objects at the same time

Sending emails

We might have a process that inserts outgoing email records into a database table. We could have a job that queries for all unsent email (again with select_for_update) and sends them, then marking them as sent in the database.

Example usage

settings.py:

INSTALLED_APPS = [
    ...
    'your_great_app',
    'job_runner',
]

your_great_app/jobs.py:

from datetime import datetime

from job_runner.registration import register_job
from job_runner.environment import RunEnv

# Run this job periodically - at most every 10 seconds and at least every 60 seconds
@register_job(10, 50)
def my_great_job(env: RunEnv):
    print(f"My great job is getting called at {datetime.now()}")

Start the job runner: python manage.py run_jobs

> python manage.py run_jobs

My great job is getting called at 2021-12-02 19:24:11.139457
My great job is getting called at 2021-12-02 19:24:27.777766
My great job is getting called at 2021-12-02 19:25:21.121113

Command line options

For most use cases no additional command line flags need to be set.

• --include-job: The full path to a registered job that should be run in this instance of the job runner. This flag can be repeated to run multiple jobs, and if it is included no jobs excepted the listed jobs will be executed. Included jobs do not have to be in a jobs.py file and can be anywhere that can be imported from Python. Jobs must use the @registered_job decorator even if they are not in jobs.py
• --exclude-job: The full path of a registered job that should be excluded from being executed. All jobs not excluded will be run and this option is mutually exclusive with --include-jobs
• --stop-after: Stop the job runner after some amount of time, listed in seconds. Useful to temporarily fix a resource leak by stopping the job runner periodically and then letting your execution environment start it again. By default the job runner does not shut itself down.
• --stop-variance: A random delay to add to the --stop-after parameter in order to prevent thundering herds if you have multiple job runner instances.
• --stop-timeout: When stopping, how long before the job runner forces an exit if the individual jobs are not shutting down cleanly. Defaults to 5 seconds.
• --trial-run: Just make sure all the included or excluded jobs can be found. The logger will emit a job list at the info level that can be used to verify what would be run. If there are no jobs to run, the job runner with exit with an error even if the --trial-run flag is set.

The job run environment

Every job that is being run will be passed an instance of job_runner.environments.RunEnv. This environment gives the job instance the ability to interact with the job runner in limited ways.

The following functions and properties are exposed for use in the run environment:

• is_stopping: This allows the script to check if it has been requested to stop. It is recommended to check this as often as possible and exit cleanly if a stop is requested.
• request_rerun(): This allows the job being executed to request that it gets executed immediately again. A sample use case might be a work queue where the queue tries to get a job from the database. You may want to request that it be rerun until there are no jobs left and then let the scheduler delay execution until the next time when the queue is empty.
• request_stop(): Request that the entire job runner shut down. Useful if running in Kubernetes or another system that will restart the job runner and the job has gotten into a situation that requires a restart to fix. Note that the entire runner and thus all jobs will exit.
• request_fatal_errors(): A shortcut to indicate that any thrown errors should be propagated and the job runner shut down if an error occurs. Effectively triggers request_stop() on an exception.
• sleep(timeout): Delay execution of the job for some amount of time. Will throw an exception if the runtime environment has requested that the system shut down. Use this instead of time.sleep to be a well behaved job that exits when it is asked to.