Skip to main content
This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (
Help us improve Python packaging - Donate today!

Flow based programming library for Python

Project Description

Pipedream is a lightweight library that helps organize Python software around concepts inspired by flow based programming. It allows you to define the dependencies between a number of loosely related resources declaratively and then uses this information to coordinate the control flow.

Apart from being an interesting and productive way to structure a program, this can also greatly simplify concurrency, testing and modularity.


Run pip install pipedream to install it quickly, or you can checkout the source at


With Pipedream, you define a dispatcher object to keep track of the resources you want to use:

from pipedream import Dispatcher
dispatcher = Dispatcher()

You can add functions to the dispatcher with a simple decorator:

def who():
    return 'World'

You define dependencies by specifying the name of the function you need as an argument:

def greet(who):
    return 'Hello {0}!'.format(who)

You can then ask the dispatcher to run any of its functions by name:

assert'greet') == 'Hello World!'

You can also specify keyword arguments at call time. These will be used even if the dispatcher has a function available that matches that name:

assert'greet', who='Everyone') == 'Hello Everyone!'


In a typical web application, a view function will make a number of database queries or other remote API calls and return the results rendered in a template. All of this work is usually done in a serial fashion, resulting in a lot of time spent waiting on network latency.

Since many of these steps are often independent of each other, doing work concurrently can cut the total response time dramatically. Coordinating this manually can get complicated very quickly, especially when certain steps depend on the results of other steps.

Once the dispatcher knows about the dependencies between various parts of a program, it is able to use an optional async back end to automatically coordinate the flow and make as concurrent as possible.

Currently the only async back end supported is powered by gevent. To use it, you will need to install the gevent <> and ProxyTypes <> libraries and then use the following code to setup your dispatcher:

from pipedream import Dispatcher
from pipedream.async.gevent import GeventPool
dispatcher = Dispatcher(async_pool=GeventPool())

All functions will automatically run in separate greenlets and the dispatcher will take care of coordinating their progress.

Advanced Decorating

There are a few more flexible options available in addition to the simple form of function decorating. If you want to manually specify the name of a function, you can do so as the first argument of the decorator:

def bar():
    return 'A Bar named Foo.'

If you want to specify the dependencies and use different names within the function, you can also do so in the decorator. The function will receive them in the order specified:

def medium(long):
    return '{0} is really big!'.format(long)
Release History

Release History

This version
History Node


History Node


Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
pipedream-0.1.1-alpha.tar.gz (5.4 kB) Copy SHA256 Checksum SHA256 Source Jun 28, 2013

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting