RDBBeat

rdbbeat is a SQLAlchemy-based scheduler for celery-beat that persists periodic tasks in a relational database.

Table Of Contents

• Installation
• Celery Configuration
• Usage
• Deployment
• Flask Service Example
• Contribution
  • Setup
  • Version Control
  • Workflows
• License
• Acknowledgements

NOTE: We assume prior knowledge of session management in SQLAlchemy and migrations in Alembic. If you are not familiar with these concepts, please read the documentation for SQLAlchemy and Alembic.

Installation

The library is available for download from the official pypi website:

pip install rdbbeat

Or you can build the library for the source code hosted publicly on GitHub.

Celery Configuration

The library makes use of the parent service's database and scope management mechanism. You can configure SQLAlchemy session_scope when you configure celery, for example as:

from celery import Celery

# this comes from parent service
from server.db import session_scope

celery = Celery('tasks')

celery.conf.update(
    {'session_scope': session_scope}
)

Usage

Creating crontab-based periodic task

A crontab schedule has the fields: minute, hour, day_of_week, day_of_month and month_of_year, so if you want the equivalent of a 30 * * * * (execute every 30 minutes) crontab entry, you specify:

from rdbbeat.controller import schedule_task
from rdbbeat.data_models import ScheduledTask

scheduled_task = {
    "name": "task_1",
    "task": "echo",
    "schedule": {
        "minute": "30",
        "hour": "*",
        "day_of_week": "*",
        "day_of_month": "*",
        "month_of_year": "*",
        "timezone": "UTC",
    },
}

with session_scope() as session:
    task = schedule_task(session, ScheduledTask.parse_ob(scheduled_task))

The crontab schedule is linked to a specific timezone using the timezone input parameter.

Run the migrations

rdbbeat includes a migration script that is required to create the database tables in the scheduler schema. Run the following command to run the migrations:

python -m alembic -n scheduler upgrade head

Deployment

The periodic tasks still need 'workers' to execute them. So make sure the default Celery package is installed. (If not installed, please follow the installation instructions here: https://github.com/celery/celery)

Both the worker and beat services need to be running at the same time.

1. Start a Celery worker service (specify your project name):

   $ celery -A [project-name] worker --loglevel=info
   

2. As a separate process, start the beat service (specify the scheduler):

   $ celery -A [project-name] beat -l info --scheduler rdbbeat.schedulers:DatabaseScheduler
   

Flask Service Example

Check out the rdbbeat-flask-example repo for a simple Flask service that uses rdbbeat to schedule periodic tasks.

Contribution

Setup

This project is setup to use editorconfig. Most editors work but some require a plugin like VSCode

It's advisable to create a virtual environment for this project to keep packages separate.

NOTE: Using pyenv, you can run pyenv virtualenv 3.10.<latest> celery-sqlalchemy-scheduler

After creating a virtual environment, install the required dependencies.

just setup-dev

Version Control

This repo follows the SemVer 2 version format.

Given a version number MAJOR.MINOR.PATCH, increment the:

• MAJOR version when you make incompatible API changes,
• MINOR version when you add functionality in a backwards compatible manner, and
• PATCH version when you make backwards compatible bug fixes.

Workflows

The repository has a number of github workflows defined in the the .github/workflows folder.

Lint & Test Code

• Tests the code for linting issues
• Tests the requirements file for any changes
• Runs the provided unit tests to ensure code quality

Release

• Pushes the library to Pypi

License

MIT licensed. See the bundled LICENSE file for more details.

Acknowledgements

• django-celery-beat
• celerybeatredis
• celery
• celery-sqlalchemy-scheduler