Dequindre /de-KWIN-der/ (n.): A minimalist scheduler.
Project description
Vision: Simplify Workflow Automation
dequindre aims to simplify workflow automation. It is part of a larger vision to teach the fundamentals and best practices to practicing and aspiring data scientists and data engineers.
What is dequindre?
dequindre is a minimalist scheduler you can use to:
quickly configure python workflows at home or at work,
run dependent tasks in separate python environments, and
learn the fundamentals and best practices of scheduling workflows.
Features
Automated workflow scheduling
- Pure Python: Relies entirely on Python built-ins
This reduces bugs, complexity, and prevents dependency hell
Cross-Python compatible: Supports Python 2 and Python 3
Cross-platform: Windows and Unix style OS environments
- Run your Python tasks in any pre-defined environments
dequindre facilitates virtualenv, conda, and pipenv environments
Supports dynamic workflow configuration also seen in Airflow
Documented examples and configuration
Getting Started
Installation
Install dequindre from PyPI with
$ pip install dequindre
Conda users can use pip too. Since dequindre has no third-party dependencies, conda doesn’t benefit from tracking dequindre’s dependencies.
Basic Example
In the REPL or in a schedule.py file,
>>> from dequindre import Task, DAG, Dequindre
>>> ## define tasks and environments
>>> pour_water = Task('./tea-tasks/pour_water.py')
>>> boil_water = Task('./tea-tasks/boil_water.py')
>>> prep_infuser = Task('./tea-tasks/prep_infuser.py')
>>> steep_tea = Task('./tea-tasks/steep_tea.py')
>>> ## define runtime dependencies
>>> make_tea = DAG(dependencies={
... boil_water: {pour_water},
... steep_tea: {boil_water, prep_infuser}
... })
>>> ## run tasks
>>> dq = Dequindre(make_tea)
>>> dq.get_schedules()
defaultdict(<class 'set'>, {
1: {Task(prep_infuser.py), Task(pour_water.py)},
2: {Task(boil_water.py)},
3: {Task(steep_tea.py)}})
>>> ## dq.run_tasks() can run the files if they exist.
License
This project is licensed under the MIT License - see the LICENSE file for details.
Versioning
We use SemVer for versioning. For the versions available, see the tags on this repository.
Contribute
If you’re interested in contributing to Dequindre, raise an issue, make a pull request to dev, and reach out to the author, vogt4nick.
Please read our contribution guidelines for details on our code of conduct, and the process for submitting pull requests to us.
Acknowledgements
Thank you, Dynatrace, for facilitating the early development of Dequindre during Innovation Day, February 2019.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for dequindre-0.9.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 643c13d5eec1c6b666bcd62a437255bbc9a3d54343b84f17a47b693d8d3f2be4 |
|
MD5 | 57389e994bc72955b074c2298cfe91dd |
|
BLAKE2b-256 | 3b4aba4db643fe8d6608a883012299152c75cc1dfa2be2ec42615f07d19e9221 |