A distributed declarative task scheduler built on Celery
Project description
thescheduler: A Declarative Task Scheduling Engine built on Celery
This project is in pre-alpha, early development, and subject to fast (read: breaking) changes
A proof-of-concept Distributed Task/Job Scheduler, designed largely for batch-driven environments.
thescheduler exposes built-in functionality of Celery through “tags”, declarative syntax used to create job definitions.
It also implements additional features (e.g. centralized logging, log rotation) to cover extended use-cases.
Features
Write job definitions using a simple yaml-based syntax.
Definitions are file based, so that they can be incorporated into existing development workflows (CI/CD). Write/commit/deploy your job definitions alongside your code.
thescheduler will read and parse valid definitions, add them to the schedule, and send to the pool of workers for execution.
Build workflows/pipelines by linking jobs using the dependency tag.
External Dependencies
redis
Installation
Development version:
$ git clone git@github.com:ehm-are/thescheduler.git $ cd thescheduler $ pip install -e .
or direct from the repository:
pip install git+https://github.com/ehm-are/thescheduler.git
Docker
TODO
Usage
- Configure settings
TODO
- Start thescheduler
TODO
- Start the workers
TODO
- Create a job definition in your configured job path ex: /jobs/test/job_3
runcmd: python3 hello_world.py startat: 13:00 US/Pacific failat: 13:05 US/Pacific runday: mon, wed, fri pool: us-west-1-dev env: PATH="/home/user/.local/bin" dependon: /test/job_1, /test/job_2
Credits
This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.
Free software: Mozilla Public License 2.0 (MPL 2.0)
Documentation: https://thescheduler.readthedocs.io.
History
0.1.0 (2020-01-01)
First release on PyPI.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for thescheduler-0.1.0-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 08430e45dc16104d4b51cb89b8d407ea66065fadaa1ff9b4fb726d330e3e2962 |
|
MD5 | 80a04a226e78de6d0f95f5865bf2ef3d |
|
BLAKE2b-256 | 7aa4afff0e568e24e057589589339c7aaaa49e9e36869055aaf2bb24b32b8065 |