Skip to main content

A distributed declarative task scheduler built on Celery

Project description

thescheduler: A Declarative Task Scheduling Engine built on Celery

https://img.shields.io/pypi/v/thescheduler.svg https://img.shields.io/travis/ehm-are/thescheduler.svg Documentation Status

This project is in pre-alpha, early development, and subject to fast (read: breaking) changes

A proof-of-concept Distributed Task/Job Scheduler, designed largely for batch-driven environments.

thescheduler exposes built-in functionality of Celery through “tags”, declarative syntax used to create job definitions.

It also implements additional features (e.g. centralized logging, log rotation) to cover extended use-cases.

Features

  • Write job definitions using a simple yaml-based syntax.

  • Definitions are file based, so that they can be incorporated into existing development workflows (CI/CD). Write/commit/deploy your job definitions alongside your code.

  • thescheduler will read and parse valid definitions, add them to the schedule, and send to the pool of workers for execution.

  • Build workflows/pipelines by linking jobs using the dependency tag.

External Dependencies

  • redis

Installation

  • Development version:

    $ git clone git@github.com:ehm-are/thescheduler.git
    $ cd thescheduler
    $ pip install -e .
  • or direct from the repository:

    pip install git+https://github.com/ehm-are/thescheduler.git

Docker

TODO

Usage

  1. Configure settings

    TODO

  2. Start thescheduler

    TODO

  3. Start the workers

    TODO

  4. Create a job definition in your configured job path ex: /jobs/test/job_3
    runcmd:      python3 hello_world.py
    startat:     13:00 US/Pacific
    failat:      13:05 US/Pacific
    runday:      mon, wed, fri
    pool:        us-west-1-dev
    env:         PATH="/home/user/.local/bin"
    dependon:    /test/job_1, /test/job_2

Credits

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.



History

0.1.0 (2020-01-01)

  • First release on PyPI.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

thescheduler-0.1.0.tar.gz (12.8 kB view details)

Uploaded Source

Built Distribution

thescheduler-0.1.0-py2.py3-none-any.whl (8.8 kB view details)

Uploaded Python 2Python 3

File details

Details for the file thescheduler-0.1.0.tar.gz.

File metadata

  • Download URL: thescheduler-0.1.0.tar.gz
  • Upload date:
  • Size: 12.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.15.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/43.0.0 requests-toolbelt/0.9.1 tqdm/4.41.1 CPython/2.7.16

File hashes

Hashes for thescheduler-0.1.0.tar.gz
Algorithm Hash digest
SHA256 ecda97541c588610a089abd9f8fc13cf08970c4cb7171c7fd6c97430a3eb69f2
MD5 d090458eaea57df7ff604c179b3b0328
BLAKE2b-256 b4124babf44ccb8c70b9d6d1ae266706783f456a25255fc59e4bb7610dc5ee76

See more details on using hashes here.

File details

Details for the file thescheduler-0.1.0-py2.py3-none-any.whl.

File metadata

  • Download URL: thescheduler-0.1.0-py2.py3-none-any.whl
  • Upload date:
  • Size: 8.8 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.15.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/43.0.0 requests-toolbelt/0.9.1 tqdm/4.41.1 CPython/2.7.16

File hashes

Hashes for thescheduler-0.1.0-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 08430e45dc16104d4b51cb89b8d407ea66065fadaa1ff9b4fb726d330e3e2962
MD5 80a04a226e78de6d0f95f5865bf2ef3d
BLAKE2b-256 7aa4afff0e568e24e057589589339c7aaaa49e9e36869055aaf2bb24b32b8065

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page