Skip to main content

thescheduler: A Declarative Task Scheduling Engine built on Celery

https://img.shields.io/pypi/v/thescheduler.svg https://img.shields.io/travis/ehm-are/thescheduler.svg Documentation Status

This project is in pre-alpha, early development, and subject to fast (read: breaking) changes

A proof-of-concept Distributed Task/Job Scheduler, designed largely for batch-driven environments.

thescheduler exposes built-in functionality of Celery through “tags”, declarative syntax used to create job definitions.

It also implements additional features (e.g. centralized logging, log rotation) to cover extended use-cases.

Features

  • Write job definitions using a simple yaml-based syntax.

  • Definitions are file based, so that they can be incorporated into existing development workflows (CI/CD). Write/commit/deploy your job definitions alongside your code.

  • thescheduler will read and parse valid definitions, add them to the schedule, and send to the pool of workers for execution.

  • Build workflows/pipelines by linking jobs using the dependency tag.

External Dependencies

  • redis

Installation

  • Development version:

    $ git clone git@github.com:ehm-are/thescheduler.git
    $ cd thescheduler
    $ pip install -e .
  • or direct from the repository:

    pip install git+https://github.com/ehm-are/thescheduler.git

Docker

TODO

Usage

  1. Configure settings

    TODO

  2. Start thescheduler

    TODO

  3. Start the workers

    TODO

  4. Create a job definition in your configured job path ex: /jobs/test/job_3
    runcmd:      python3 hello_world.py
    startat:     13:00 US/Pacific
    failat:      13:05 US/Pacific
    runday:      mon, wed, fri
    pool:        us-west-1-dev
    env:         PATH="/home/user/.local/bin"
    dependon:    /test/job_1, /test/job_2

Credits

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.



History

0.1.0 (2020-01-01)

  • First release on PyPI.

Metadata

Release files for thescheduler 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for thescheduler 0.1.0
File Size Uploaded
thescheduler-0.1.0.tar.gz 12.8 kB Details

Built distribution (wheel)

Table of built distributions (wheels) for thescheduler 0.1.0
File Interpreter ABI Platform
thescheduler-0.1.0-py2.py3-none-any.whl Python 3, Python 2 none any Details

Total release size: 21.6 kB

Release files / thescheduler-0.1.0.tar.gz

Download URL thescheduler-0.1.0.tar.gz
Size 12.8 kB
Tags Source
SHA-256 checksum
How to use checksums
ecda97541c588610a089abd9f8fc13cf08970c4cb7171c7fd6c97430a3eb69f2
BLAKE2b-256 checksum
How to use checksums
b4124babf44ccb8c70b9d6d1ae266706783f456a25255fc59e4bb7610dc5ee76
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/1.15.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/43.0.0 requests-toolbelt/0.9.1 tqdm/4.41.1 CPython/2.7.16

Release files / thescheduler-0.1.0-py2.py3-none-any.whl

Download URL thescheduler-0.1.0-py2.py3-none-any.whl
Size 8.8 kB
Tags Python 2 Python 3
SHA-256 checksum
How to use checksums
08430e45dc16104d4b51cb89b8d407ea66065fadaa1ff9b4fb726d330e3e2962
BLAKE2b-256 checksum
How to use checksums
7aa4afff0e568e24e057589589339c7aaaa49e9e36869055aaf2bb24b32b8065
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/1.15.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/43.0.0 requests-toolbelt/0.9.1 tqdm/4.41.1 CPython/2.7.16

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page