Dequindre /de-KWIN-der/ (n.): A minimalist scheduler.
Project description
Dequindre /de-KWIN-der/ (n.): A minimalist scheduler.
Vision: Simplify Workflow Automation
dequindre
aims to simplify workflow automation. It is part of a larger vision to teach the fundamentals and best practices to practicing and aspiring data scientists and data engineers.
What is dequindre?
dequindre
is a minimalist scheduler you can use to:
- quickly configure python workflows at home or at work,
- run dependent tasks in separate python environments, and
- learn the fundamentals and best practices of scheduling workflows.
Features
- Automated workflow scheduling
- Pure Python: Relies entirely on Python built-ins
- This reduces bugs, complexity, and prevents dependency hell
- Cross-Python compatible: Supports Python 2 and Python 3
- Cross-platform: Windows and Unix style OS environments
- Run your Python tasks in any pre-defined environments
dequindre
facilitates virtualenv, conda, and pipenv environments
- Supports dynamic workflow configuration also seen in Airflow
- Documented examples and configuration
Basic Example
First, install dequindre
with pip install dequindre
. Then, in the
REPL or in a schedule.py
file,
from dequindre import Task, DAG, Dequindre
## define tasks and environments
pour_water = Task('./tea-tasks/pour_water.py')
boil_water = Task('./tea-tasks/boil_water.py')
prep_infuser = Task('./tea-tasks/prep_infuser.py')
steep_tea = Task('./tea-tasks/steep_tea.py')
## define runtime dependencies
make_tea = DAG(dependencies={
boil_water: {pour_water},
steep_tea: {boil_water, prep_infuser}
})
## run tasks
dq = Dequindre(make_tea)
dq.get_schedules()
# defaultdict(<class 'set'>, {
# 1: {Task(prep_infuser.py), Task(pour_water.py)},
# 2: {Task(boil_water.py)},
# 3: {Task(steep_tea.py)}})
## dq.run_tasks() can run the files if they exist.
Getting Started
Dequindre is has two requirements: conda and python.
Installation
$ pip install dequindre
Contribute
If you're interested in contributing to Dequindre, raise an issue, make a pull request to dev
, and reach out to the author, vogt4nick.
Please read contributing.md for details on our code of conduct, and the process for submitting pull requests to us.
Versioning
We use SemVer for versioning. For the versions available, see the tags on this repository.
License
This project is licensed under the MIT License - see the license.md file for details.
Acknowledgements
Thank you, Dynatrace, for facilitating the early development of Dequindre during Innovation Day, February 2019.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for dequindre-0.8.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7e9c7270c7edf98bae284ce29213ab1f18057a60966334c8ef24fea992dea618 |
|
MD5 | 9583390a9b7f5c276e6f870fa1cbadfd |
|
BLAKE2b-256 | 480c5a2a03151c251f4351248bd6644f1fe62fb589afc5af860cf181b2aabaec |