dequindre 0.8.5

Dequindre /de-KWIN-der/ (n.): A minimalist scheduler.

Vision: Simplify Workflow Automation

dequindre aims to simplify workflow automation. It is part of a larger vision to teach the fundamentals and best practices to practicing and aspiring data scientists and data engineers.

What is dequindre?

dequindre is a minimalist scheduler you can use to:

• quickly configure python workflows at home or at work,

• run dependent tasks in separate python environments, and

• learn the fundamentals and best practices of scheduling workflows.

Features

• Automated workflow scheduling

• Pure Python: Relies entirely on Python built-ins

  • This reduces bugs, complexity, and prevents dependency hell

• Cross-Python compatible: Supports Python 2 and Python 3

• Cross-platform: Windows and Unix style OS environments

• Run your Python tasks in any pre-defined environments

  • dequindre facilitates virtualenv, conda, and pipenv environments

• Supports dynamic workflow configuration also seen in Airflow

• Documented examples and configuration

Getting Started

Installation

Install dequindre from PyPI with

$ pip install dequindre

Conda users can use pip too. Since dequindre has no third-party dependencies, conda doesn’t benefit from tracking dequindre’s dependencies.

Basic Example

In the REPL or in a schedule.py file,

>>> from dequindre import Task, DAG, Dequindre

>>> ## define tasks and environments
>>> pour_water = Task('./tea-tasks/pour_water.py')
>>> boil_water = Task('./tea-tasks/boil_water.py')
>>> prep_infuser = Task('./tea-tasks/prep_infuser.py')
>>> steep_tea = Task('./tea-tasks/steep_tea.py')

>>> ## define runtime dependencies
>>> make_tea = DAG(dependencies={
...     boil_water: {pour_water},
...     steep_tea: {boil_water, prep_infuser}
... })

>>> ## run tasks
>>> dq = Dequindre(make_tea)
>>> dq.get_schedules()
defaultdict(<class 'set'>, {
    1: {Task(prep_infuser.py), Task(pour_water.py)},
    2: {Task(boil_water.py)},
    3: {Task(steep_tea.py)}})

>>> ## dq.run_tasks() can run the files if they exist.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Versioning

We use SemVer for versioning. For the versions available, see the tags on this repository.

Contribute

If you’re interested in contributing to Dequindre, raise an issue, make a pull request to dev, and reach out to the author, vogt4nick.

Please read our contribution guidelines for details on our code of conduct, and the process for submitting pull requests to us.

Acknowledgements

Thank you, Dynatrace, for facilitating the early development of Dequindre during Innovation Day, February 2019.