This is a pre-production deployment of Warehouse, however changes made here WILL affect the production instance of PyPI.
Latest Version Dependencies status unknown Test status unknown Test coverage unknown
Project Description

Pipedream is a lightweight library that helps organize Python software around concepts inspired by flow based programming. It allows you to define the dependencies between a number of loosely related resources declaratively and then uses this information to coordinate the control flow.

Apart from being an interesting and productive way to structure a program, this can also greatly simplify concurrency, testing and modularity.

Installing

Run pip install pipedream to install it quickly, or you can checkout the source at http://github.com/tgecho/pipedream/

Quickstart

With Pipedream, you define a dispatcher object to keep track of the resources you want to use:

from pipedream import Dispatcher
dispatcher = Dispatcher()

You can add functions to the dispatcher with a simple decorator:

@dispatcher.add
def who():
    return 'World'

You define dependencies by specifying the name of the function you need as an argument:

@dispatcher.add
def greet(who):
    return 'Hello {0}!'.format(who)

You can then ask the dispatcher to run any of its functions by name:

assert dispatcher.call('greet') == 'Hello World!'

You can also specify keyword arguments at call time. These will be used even if the dispatcher has a function available that matches that name:

assert dispatcher.call('greet', who='Everyone') == 'Hello Everyone!'

Concurrency

In a typical web application, a view function will make a number of database queries or other remote API calls and return the results rendered in a template. All of this work is usually done in a serial fashion, resulting in a lot of time spent waiting on network latency.

Since many of these steps are often independent of each other, doing work concurrently can cut the total response time dramatically. Coordinating this manually can get complicated very quickly, especially when certain steps depend on the results of other steps.

Once the dispatcher knows about the dependencies between various parts of a program, it is able to use an optional async back end to automatically coordinate the flow and make as concurrent as possible.

Currently the only async back end supported is powered by gevent. To use it, you will need to install the gevent <https://pypi.python.org/pypi/gevent> and ProxyTypes <https://pypi.python.org/pypi/ProxyTypes> libraries and then use the following code to setup your dispatcher:

from pipedream import Dispatcher
from pipedream.async.gevent import GeventPool
dispatcher = Dispatcher(async_pool=GeventPool())

All functions will automatically run in separate greenlets and the dispatcher will take care of coordinating their progress.

Advanced Decorating

There are a few more flexible options available in addition to the simple form of function decorating. If you want to manually specify the name of a function, you can do so as the first argument of the decorator:

@dispatcher.add('foo')
def bar():
    return 'A Bar named Foo.'

If you want to specify the dependencies and use different names within the function, you can also do so in the decorator. The function will receive them in the order specified:

@dispatcher.add(requires=['a_really_long_name'])
def medium(long):
    return '{0} is really big!'.format(long)
Release History

Release History

0.1.1-alpha

This version

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.1.0-alpha

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

Download Files

Download Files

TODO: Brief introduction on what you do with files - including link to relevant help section.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
pipedream-0.1.1-alpha.tar.gz (5.4 kB) Copy SHA256 Checksum SHA256 Source Jun 28, 2013

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS HPE HPE Development Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting