Skip to main content
Help us improve Python packaging – donate today!

Python library for dataflow programming with Amazon SWF

Project Description

https://badge.fury.io/py/simpleflow.png https://travis-ci.org/botify-labs/simpleflow.png?branch=master https://pypip.in/d/simpleflow/badge.png

Simple Flow is a Python library that provides abstractions to write programs in the distributed dataflow paradigm. It relies on futures to describe the dependencies between tasks. It coordinates the execution of distributed tasks with Amazon SWF.

A Future object models the asynchronous execution of a computation that may end.

It tries to mimics the interface of the Python concurrent.futures library.

Features

  • Provides a Future abstraction to define dependencies between tasks.
  • Define asynchronous tasks from callables.
  • Handle workflows with Amazon SWF.
  • Implement replay behavior like the Amazon Flow framework.
  • Handle retry of tasks that failed.
  • Automatically register decorated tasks.
  • Handle the completion of a decision with more than 100 tasks.
  • Provides a local executor to check a workflow without Amazon SWF (see simpleflow --local command).

Quickstart

Let’s take a simple example that computes the result of (x + 1) * 2.

We need to declare the functions as activities to make them available:

from simpleflow import activity

@activity.with_attributes(task_list='quickstart')
def increment(x):
    return x + 1

@activity.with_attributes(task_list='quickstart')
def double(x):
    return x * 2

And then define the workflow itself in a example.py file:

from simpleflow import Workflow

class SimpleComputation(Workflow):
    def run(self, x):
        y = self.submit(increment, x)
        z = self.submit(double, y)
        return z.result

Now check that the workflow works locally:

$ simpleflow --local -w example.SimpleComputation -i example/input.json

The file example/input.json contains the input passed to the workflow. It should have the format:

{"args": [1],
 "kwargs": {}
}

which is equivalent to:

{"kwargs": {"x": 1}}

Documentation

Full documentation is available at https://simpleflow.readthedocs.org/.

Requirements

  • Python >= 2.6 or >= 3.3

License

MIT licensed. See the bundled LICENSE file for more details.

Changelog

0.1.0 (2014-02-19)

  • First release.

Release history Release notifications

History Node

0.21.0

History Node

0.20.8

History Node

0.20.7

History Node

0.20.6

History Node

0.20.5

History Node

0.20.4

History Node

0.20.3

History Node

0.20.2

History Node

0.20.1

History Node

0.20.0

History Node

0.19.2

History Node

0.19.1

History Node

0.19.0

History Node

0.18.15

History Node

0.18.14

History Node

0.18.13

History Node

0.18.12

History Node

0.18.11

History Node

0.18.10

History Node

0.18.9

History Node

0.18.8

History Node

0.18.7

History Node

0.18.4

History Node

0.18.3

History Node

0.18.2

History Node

0.18.1

History Node

0.18.0

History Node

0.17.0

History Node

0.16.0

History Node

0.15.7

History Node

0.15.6

History Node

0.15.5

History Node

0.15.4

History Node

0.15.3

History Node

0.15.2

History Node

0.15.1

History Node

0.15.0

History Node

0.14.2

History Node

0.14.1

History Node

0.14.0

History Node

0.13.4

History Node

0.13.3

History Node

0.13.2

History Node

0.13.1

History Node

0.13.0

History Node

0.12.7

History Node

0.12.6

History Node

0.12.5

History Node

0.12.4

History Node

0.12.3

History Node

0.12.2

History Node

0.12.1

History Node

0.12.0

History Node

0.11.17

History Node

0.11.16

History Node

0.11.15

History Node

0.11.14

History Node

0.11.13

History Node

0.11.12

History Node

0.11.11

History Node

0.11.10

History Node

0.11.9

History Node

0.11.8

History Node

0.11.7

History Node

0.11.6

History Node

0.11.5

History Node

0.11.4

History Node

0.11.3

History Node

0.11.2

History Node

0.11.1

History Node

0.11.0

History Node

0.10.3

History Node

0.10.2

History Node

0.10.1

History Node

0.10.0

History Node

0.9.7

History Node

0.9.6

History Node

0.9.5

History Node

0.9.4

History Node

0.9.3

History Node

0.9.2

History Node

0.9.1

History Node

0.9.0

History Node

0.8.0

History Node

0.7.0

History Node

0.6.4

History Node

0.6.3

History Node

0.6.2

History Node

0.6.1

History Node

0.6.0

History Node

0.5.3

History Node

0.5.2

History Node

0.5.1

History Node

0.5.0

History Node

0.4.0

History Node

0.3.2

History Node

0.3.1

History Node

0.3.0

History Node

0.2.3

History Node

0.2.2

History Node

0.2.1

History Node

0.2.0

This version
History Node

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
simpleflow-0.1.0-py2.py3-none-any.whl (6.4 kB) Copy SHA256 hash SHA256 Wheel 2.7 Feb 19, 2014
simpleflow-0.1.0.tar.gz (5.6 kB) Copy SHA256 hash SHA256 Source None Feb 19, 2014

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging CloudAMQP CloudAMQP RabbitMQ AWS AWS Cloud computing Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page