Python library for dataflow programming with Amazon SWF
Project description
Simpleflow
Simpleflow is a Python library that provides abstractions to write programs in the distributed dataflow paradigm. It coordinates the execution of distributed tasks with Amazon SWF.
It relies on futures to describe the dependencies between tasks. A Future
object
models the asynchronous execution of a computation that may end. It tries to mimic
the interface of the Python concurrent.futures library.
Features
- Provides a
Future
abstraction to define dependencies between tasks. - Define asynchronous tasks from callables.
- Handle workflows with Amazon SWF.
- Implement replay behavior like the Amazon Flow framework.
- Handle retry of tasks that failed.
- Automatically register decorated tasks.
- Encodes/decodes large fields to S3 objects transparently (aka "jumbo fields").
- Handle the completion of a decision with more than 100 tasks.
- Provides a local executor to check a workflow without Amazon SWF (see
simpleflow --local
command). - Provides decider and activity worker process for execution with Amazon SWF.
- Ships with the
simpleflow
command.simpleflow --help
for more information about the commands it supports.
You can read more in the Features section of the documentation.
Overview
Please read and even run the demo
script to have a quick glance of
simpleflow
commands. To run the demo
you will need to start decider
and activity worker processes.
Start a decider with:
$ PYTHONPATH=$PWD simpleflow decider.start --domain TestDomain --task-list test examples.basic.BasicWorkflow
Start an activity worker with:
$ simpleflow worker.start --domain TestDomain --task-list quickstart
Then execute ./extras/demo
.
More information
Read the main documentation at https://botify-labs.github.io/simpleflow/.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for simpleflow-0.33.1rc2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5e29243f34354d8d47ed1fa512520aaa0fb37c2974fd890ebf5ca7bbf7211b73 |
|
MD5 | a9574dd9fa49d2d83c798195e99c18c8 |
|
BLAKE2b-256 | 9e213252b404b32bdbff226230d4cc719bfe90743fe437824f56c886bdf01796 |