A distributed declarative task scheduler built on Celery
Project description
thescheduler: A Declarative Task Scheduling Engine built on Celery
This project is in pre-alpha, early development, and subject to fast (read: breaking) changes
A proof-of-concept Distributed Task/Job Scheduler, designed largely for batch-driven environments.
thescheduler exposes built-in functionality of Celery through “tags”, declarative syntax used to create job definitions.
It also implements additional features (e.g. centralized logging, log rotation) to cover extended use-cases.
Features
Write job definitions using a simple yaml-based syntax.
Definitions are file based, so that they can be incorporated into existing development workflows (CI/CD). Write/commit/deploy your job definitions alongside your code.
thescheduler will read and parse valid definitions, add them to the schedule, and send to the pool of workers for execution.
Build workflows/pipelines by linking jobs using the dependency tag.
External Dependencies
redis
Installation
Development version:
$ git clone git@github.com:ehm-are/thescheduler.git $ cd thescheduler $ pip install -e .
or direct from the repository:
pip install git+https://github.com/ehm-are/thescheduler.git
Docker
TODO
Usage
- Configure settings
TODO
- Start thescheduler
TODO
- Start the workers
TODO
- Create a job definition in your configured job path ex: /jobs/test/job_3
runcmd: python3 hello_world.py startat: 13:00 US/Pacific failat: 13:05 US/Pacific runday: mon, wed, fri pool: us-west-1-dev env: PATH="/home/user/.local/bin" dependon: /test/job_1, /test/job_2
Credits
This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.
Free software: Mozilla Public License 2.0 (MPL 2.0)
Documentation: https://thescheduler.readthedocs.io.
History
0.1.0 (2020-01-01)
First release on PyPI.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file thescheduler-0.1.0.tar.gz
.
File metadata
- Download URL: thescheduler-0.1.0.tar.gz
- Upload date:
- Size: 12.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.15.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/43.0.0 requests-toolbelt/0.9.1 tqdm/4.41.1 CPython/2.7.16
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 |
ecda97541c588610a089abd9f8fc13cf08970c4cb7171c7fd6c97430a3eb69f2
|
|
MD5 |
d090458eaea57df7ff604c179b3b0328
|
|
BLAKE2b-256 |
b4124babf44ccb8c70b9d6d1ae266706783f456a25255fc59e4bb7610dc5ee76
|
File details
Details for the file thescheduler-0.1.0-py2.py3-none-any.whl
.
File metadata
- Download URL: thescheduler-0.1.0-py2.py3-none-any.whl
- Upload date:
- Size: 8.8 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.15.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/43.0.0 requests-toolbelt/0.9.1 tqdm/4.41.1 CPython/2.7.16
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 |
08430e45dc16104d4b51cb89b8d407ea66065fadaa1ff9b4fb726d330e3e2962
|
|
MD5 |
80a04a226e78de6d0f95f5865bf2ef3d
|
|
BLAKE2b-256 |
7aa4afff0e568e24e057589589339c7aaaa49e9e36869055aaf2bb24b32b8065
|