Why Fork?

Zag is a fork of OpenStack TaskFlow. Josh Harlow and others did an amazing job creating and maintaining TaskfFlow for many years, but it has languished for years with few updates. The worker-based engine and job board pieces of TaskFlow never got wide usage, so they remained stuck in a rather buggy, somewhat difficult-to-use state. The goals of Zag will be to focus on those pieces. Also, rather than trying to support a myriad of technologies that sort of fit the bill, it will focus on optimizing with the right technologies. So, to that end, the aims of Zag will be to do the following:

• Focus on Zookeeper for distributed coordination. Support for others will be provided by the tooz library, but Zookeeper is really the best technology available for this purpose, so some features might not work with others.

• Focus on RabbitMQ or other AMQP providers for worker communication. Support for others will be available via kombu, but some features will likely not work without the ability to use dead-letter queues to delay task execution or retries.

• Reduce the cognitive load required to get Zag up and running. Simply posting a job in the job board in TaskFlow requires something like 50 lines of code and a rather in-depth understanding of how TaskFlow works under the covers.

• Make writing flows simpler and more enjoyable. Adding a declarative syntax for building flows and simplifying how arguments are passed to tasks.

To accomplish those goals, some of the TaskFlow APIs will need to be refactored, and that would require breaking upstream users. So in the end, I opted to fork and change the name so we can push forward at a more rapid pace. This will be a work in progress for some time, so the initial releases will mostly keep things as-is. Over time, we’ll morph a few key pieces.

Testing and requirements

Developer documentation

We also have sphinx documentation in docs/source.

To build it, run:

$ python setup.py build_sphinx