zag 0.2.12

Distributed workflow processing.

A library to do [jobs, tasks, flows] in a highly available, easy to understand and declarative manner (and more!) to be used with OpenStack and other projects.

• Free software: Apache license

• Documentation: https://docs.openstack.org/zag/latest/

• Source: https://git.openstack.org/cgit/openstack/zag

• Bugs: https://bugs.launchpad.net/zag/

• Release notes: https://docs.openstack.org/releasenotes/zag/

Why Fork?

Zag is a fork of OpenStack TaskFlow. Josh Harlow and others did an amazing job creating and maintaining TaskfFlow for many years, but it has languished for years with few updates. The worker-based engine and job board pieces of TaskFlow never got wide usage, so they remained stuck in a rather buggy, somewhat difficult-to-use state. The goals of Zag will be to focus on those pieces. Also, rather than trying to support a myriad of technologies that sort of fit the bill, it will focus on optimizing with the right technologies. So, to that end, the aims of Zag will be to do the following:

• Focus on Zookeeper for distributed coordination. Support for others will be provided by the tooz library, but Zookeeper is really the best technology available for this purpose, so some features might not work with others.

• Focus on RabbitMQ or other AMQP providers for worker communication. Support for others will be available via kombu, but some features will likely not work without the ability to use dead-letter queues to delay task execution or retries.

• Reduce the cognitive load required to get Zag up and running. Simply posting a job in the job board in TaskFlow requires something like 50 lines of code and a rather in-depth understanding of how TaskFlow works under the covers.

• Make writing flows simpler and more enjoyable. Adding a declarative syntax for building flows and simplifying how arguments are passed to tasks.

To accomplish those goals, some of the TaskFlow APIs will need to be refactored, and that would require breaking upstream users. So in the end, I opted to fork and change the name so we can push forward at a more rapid pace. This will be a work in progress for some time, so the initial releases will mostly keep things as-is. Over time, we’ll morph a few key pieces.

Testing and requirements

Requirements

Because this project has many optional (pluggable) parts like persistence backends and engines, we decided to split our requirements into two parts: - things that are absolutely required (you can’t use the project without them) are put into requirements.txt. The requirements that are required by some optional part of this project (you can use the project without them) are put into our test-requirements.txt file (so that we can still test the optional functionality works as expected). If you want to use the feature in question (eventlet or the worker based engine that uses kombu or the sqlalchemy persistence backend or jobboards which have an implementation built using kazoo …), you should add that requirement(s) to your project or environment.

Tox.ini

Our tox.ini file describes several test environments that allow to test Zag with different python versions and sets of requirements installed. Please refer to the tox documentation to understand how to make these test environments work for you.

Developer documentation

We also have sphinx documentation in docs/source.

To build it, run:

$ python setup.py build_sphinx