Skip to main content

A distributed declarative task scheduler built on Celery

Project description

thescheduler: A Declarative Task Scheduling Engine built on Celery

https://img.shields.io/pypi/v/thescheduler.svg https://img.shields.io/travis/ehm-are/thescheduler.svg Documentation Status

This project is in pre-alpha, early development, and subject to fast (read: breaking) changes

A proof-of-concept Distributed Task/Job Scheduler, designed largely for batch-driven environments.

thescheduler exposes built-in functionality of Celery through “tags”, declarative syntax used to create job definitions.

It also implements additional features (e.g. centralized logging, log rotation) to cover extended use-cases.

Features

  • Write job definitions using a simple yaml-based syntax.
  • Definitions are file based, so that they can be incorporated into existing development workflows (CI/CD). Write/commit/deploy your job definitions alongside your code.
  • thescheduler will read and parse valid definitions, add them to the schedule, and send to the pool of workers for execution.
  • Build workflows/pipelines by linking jobs using the dependency tag.

External Dependencies

  • redis

Installation

  • Development version:

    $ git clone git@github.com:ehm-are/thescheduler.git
    $ cd thescheduler
    $ pip install -e .
    
  • or direct from the repository:

    pip install git+https://github.com/ehm-are/thescheduler.git
    

Docker

TODO

Usage

  1. Configure settings

    TODO

  2. Start thescheduler

    TODO

  3. Start the workers

    TODO

  4. Create a job definition in your configured job path ex: /jobs/test/job_3
    runcmd:      python3 hello_world.py
    startat:     13:00 US/Pacific
    failat:      13:05 US/Pacific
    runday:      mon, wed, fri
    pool:        us-west-1-dev
    env:         PATH="/home/user/.local/bin"
    dependon:    /test/job_1, /test/job_2
    

Credits

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.



History

0.1.0 (2020-01-01)

  • First release on PyPI.

Project details


Release history Release notifications

This version

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for thescheduler, version 0.1.0
Filename, size File type Python version Upload date Hashes
Filename, size thescheduler-0.1.0-py2.py3-none-any.whl (8.8 kB) File type Wheel Python version py2.py3 Upload date Hashes View hashes
Filename, size thescheduler-0.1.0.tar.gz (12.8 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page