django-cronman

Overview

Django app to define and manage periodic tasks at Python level.

Installation

django-cronman can be installed directly from PyPI using pip:

pip install django-cronman

You can also install it with additional dependencies to be able to use Cron Remote Manager.

pip install django-cronman[redis]

Define a new cron job

Cron job definition is inspired by Django Admin configuration. To add a new job, you have to create cron_job.py file inside an app, create BaseCronJob subclass inside and register it:

from cronman.job import BaseCronJob, cron_job_registry

class HelloWorld(BaseCronJob):
    """Demo Cron Job class"""

    def run(self):
        """Main logic"""
        pass

cron_job_registry.register(HelloWorld)

Cron job classes are registered (and referred to) by name, which may be customized on registration:

cron_job_registry.register(HelloWorld, name='Hello')

It's also possible to retrieve or unregister a class (e.g. while testing):

cron_job_registry.get('HelloWorld')
cron_job_registry.unregister('HelloWorld')

If there is more than 1 cron job in given app, it's recommended to create a package instead of single cron_jobs module, create one submodule per class and do the imports and registration in package's __init__.py.

Configure cron jobs

To ensure that a cron job is executed periodically, you have add an entry to CRON_JOBS:

CRON_JOBS = (
    ...
    # (<time spec>, <job spec>)
    # 'HelloWorld' will be executed a 5:15AM every day:
    ('   15   5   *   *   *', 'HelloWorld'),
)

Set CRONMAN_JOBS_MODULE to the dotted path name of the module where cron jobs are specified. Remember, this module MUST have a CRON_JOBS attribute. CRONMAN_JOBS_MODULE is None by default. For example:

# settings_local.py

CRONMAN_JOBS_MODULE = 'app.cron_jobs.name'

Run the scheduler

Cron jobs defined in settings.CRONMAN_JOBS_MODULE are started by cron_scheduler command from cron app. This command constructs a list of jobs that should be executed in current period (now +/- 1 minute) and creates a new subprocess for each job.

python manage.py cron_scheduler run

This command should be added to system's crontab on server responsible for running periodic tasks and executed every 2 minutes.

Run single cron job

Command cron_worker run <job spec> is responsible for executing cron jobs:

python manage.py cron_worker run HelloWorld

Cron job parameters

Cron job classes can accept parameters which are passed to run method as positional or named arguments:

class HelloWorld(BaseCronJob):
    """Demo Cron Job class"""

    def run(self, what, sleep=None):
        """Main logic"""
        print "Hello {}".format(what)
        if sleep:
            time.sleep(int(sleep))
    ...

python manage.py cron_worker run HelloWorld:world,sleep=5

Parameters are passed as string values, any type casts should be made in run method. Quoted string with spaces are supported, but comma can be used only as argument separator:

python manage.py cron_worker run HelloWorld:"big world",sleep=5

There are utility functions for extracting lists and boolean values in cronman.utils module.

Configure Cronitor support

cron_worker command can notify Cronitor when a job is started, finished or it has failed. To enable this you have to:

1. Enable Cronitor support in settings CRONMAN_CRONITOR_ENABLED = True
2. Configure you cron job class:

class HelloWorld(BaseCronJob):
    """Demo Cron Job class"""
    cronitor_id = 'ab12z'  # ID is assigned in Cronitor's dashboard

We may disable sending optional "RUN" and "FAIL" pings to Cronitor when cron job starts by setting cronitor_ping_run = False or cronitor_ping_fail = False but this doesn't seem to be necessary.

Important note: When adding a new monitor in Cronitor dashboard, please use type heartbeat. Avoid using cron job monitors, as they're sending false-positive alerts "Has not run on schedule".

Configure lock

Tasks can acquire locks to prevent concurrent calls. Locks have form of PIDfiles located in settings.CRONMAN_DATA_DIR. To modify lock behavior for given cron job class you can set lock_type attribute:

from cronman.taxonomies import LockType

class HelloWorld(BaseCronJob):
    """Demo Cron Job class"""
    lock_type = LockType.PARAMS

The following values are supported:

• None - no lock, concurrency is allowed
• LockType.CLASS (default) - only one instance of given cron job class can be run at the same time (e.g. Foo:p=1 and Foo:p=2 can't work concurrently)
• LockType.PARAMS - only one combination of class and params can be run at the same time (e.g. Foo:p=1 and Foo:p=2 can work concurrently, but another call to Foo:p=1 will be prohibited) Locks acquired/released by cron_worker command.

We can configure a shared lock for several cron job classes to make sure only one of them is running:

from cronman.taxonomies import LockType

class HelloWorld1(BaseCronJob):
    """Demo Cron Job class (1)"""
    lock_type = LockType.CLASS
    lock_name = 'HelloWorld'

class HelloWorld2(BaseCronJob):
    """Demo Cron Job class (2)"""
    lock_type = LockType.CLASS
    lock_name = 'HelloWorld'

Configure CPU and IO priority

We can assign CPU priority (nice) to a cron job class by using worker_cpu_priority attribute:

from cronman.taxonomies import CPUPriority

class NiceHellowWorld(BaseCronJob):
    """Hello World running through `nice -n 19`"""
    worker_cpu_priority = CPUPriority.LOWEST

We can also customize IO priority (ionice) by assigning one of values from cronman.taxonomies.IOPriority to worker_io_priority attribute, but this is not necessary in most cases, as nice changes IO priority as well.

Commands cron_scheduler run, cron_worker resume, and cron job RunCronTasks will spawn worker processes with respect to CPU and IO priorities assigned to cron job classes. These settings are not enforced when running cron_worker run so you have to prepend nice/ionice to such calls manually.

List and kill running cron jobs

Command cron_worker status shows currently running cron jobs - PIDfile name, PID and status (ALIVE, DEAD). Search results can be limited by job spec (cron job name, parameters):

python manage.py cron_worker status
python manage.py cron_worker status Foo
python manage.py cron_worker status Foo:bar=1

Command cron_worker kill kills active cron jobs, gracefully (SIGTERM) or by force when process refuses to die (SIGKILL). List of tasks can be limited by job spec:

python manage.py cron_worker kill
python manage.py cron_worker kill Foo
python manage.py cron_worker kill Foo:bar=1

Single process can be killed also using PID:

python manage.py cron_worker kill 39078

Resuming cron jobs

Subset of cron jobs can be resumed after being killed:

class ResumableHelloWorld(BaseCronJob):
    """Demo Cron Job class"""
    can_resume = True

Command cron_worker resume starts all killed cron jobs with can_resume capability:

python manage.py cron_worker resume

To remove all entries about dead cron jobs and make sure they won't be resumed we can run cron_worker clean command:

python manage.py cron_worker clean

Command cron_worker suspend cleans all previous entries about dead cron jobs and then kills all running ones to make sure that next resume will raise only recently killed jobs:

python manage.py cron_worker suspend

List available cron jobs

Command cron_worker info shows list of all available cron jobs:

python manage.py cron_worker info

When cron job name is passed to this command, system displays docstring and parameters of given cron job:

python manage.py cron_worker info Foo

Disable the scheduler

Scheduler command can be disabled temporarily:

python manage.py cron_scheduler disable

and re-enabled later:

python manage.py cron_scheduler enable

Calls to cron_scheduler run will not spawn worker processes while scheduler is disabled.

Send errors to sentry

Errors in cron job classes are intercepted by cron_worker and sent to Sentry using the same config as other Django commands (settings.RAVEN_MANAGEMENT_COMMAND_CONFIG). If settings.CRONMAN_RAVEN_CMD is defined, the scheduler will use it as execution script for worker processes, e.g. python manage.py cron_worker run Foo:bar=1 will be converted to {CRONMAN_RAVEN_CMD} -c "python manage.py cron_worker run Foo:bar=1"

Cron Tasks - running cron jobs from Admin area

Some cron jobs can be requested to start from Admin area: Admin > Cron > Cron Tasks To add a cron job class to the list in Admin we need to set ALLOWED_CRON_TASKS setting:

ALLOWED_CRON_TASKS = (
    'HelloWorld',
)

To request another run of given cron job we can just create a new CronTask record in Admin. Cron job RunCronTasks started every 4 minutes by the scheduler will spawn a separate worker process for each pending Cron Task.

Changelog

2021-01-20 - 3.0.0 Remove Python 2 checks. Fix issue with suspend all. 2020-05-25 - 2.1.0 Python 3 + Django 2 compatibility. 2020-04-30 - 2.0.1 Fix for sentry-sdk. 2020-04-23 - 2.0.0 Replace raven with sentry-sdk. 2020-01-09 - 1.2.0 Django 2 compatibility. 2019-04-30 - 1.1.1 Pre-commit.com hooks support. Docs update 2019-03-13 - 1.1.0 Add support for cronitor ping for cron_scheduler 2019-02-25 - 1.0.0 Initial version released