Skip to main content

Coroutines for dataflow pipelines

Project description

==> COP
coroutines for dataflow pipelines

Cop is designed to make it easy
to write readable dataflow programs.
Speed is important, but probably
isn't there yet.

This library is a learning exercise.
At this stage, it should really be
seen as a source for you to see
how to implement your own coroutines.

The primary motivation for cop is
to facilitate readable and flexible
dataflow programming. Once the
project is more mature, its focus
could shift to enhancing performance.

There are three basic terms that are
used within cop: sources, steps and

/sources/ Feed data to steps.
(cop.file, cop.web)

/steps/ Process data, then
feed it to a target.

/sinks/ Process data.

Using cop is straight forward. Examples
generally follow the following pattern:

from cop.sources import source
from cop.steps import step1, step2
from cop.sinks import sink



>>> from cop.sources import web
>>> from cop.steps import delimit
>>> from cop.stops import grep
>>> from cop.steps import transform
>>> from cop.sinks import printer

>>> url = ''
>>> web(url,
... delimit('\n',
... transform(lambda l: l.lower(),
... grep('monty',
... printer())))

In this case, nothing comes out. That's
basically because there's very little
about Monty Python at

/non-linear pipelines/
Sometimes, a linear pathway isn't enough.
Perhaps we would like to do something
with intermediate results.

Enter `cop.flow`. It provides `broadcast`
for sending data to multiple steps, and
`loadbalance`, which implements a round-
robin load balancing system between steps.

The start of this whole thing is from
code written by David Beazley, especially
his tutorial on coroutines[0]. That
work is excellent and should basically
be considered the manual for this library.

I read the PDF of the tutorial in the
morning. By the end of the evening I had
a few dozen workable functions that could
be strung together in an almost arbitary

I would really like to see interacting
with multiple cores and hosts become
trivally easy.

If you would like to contribute to cop,
that's great. It's easy to do and will
help people. At the moment, test
coverage is at a healthy 0%.

The plan is to use heavy use of doctests.
`cop.printer` should be quite useful:

>> source(sink(printer()))

I use >> as prompt when I don't want
doctest to claim it. I find this
preferable to using the skip directive.

Apart from code written by David Beazley,
which is used with with permission, the
code the copyright of Tim McNamara[1].

The code is released under the Apache 2

All documentation is the copyright of
Tim McNamara. All docs are released under
the "Creative Commons Attribution 3.0
New Zealand (CC BY 3.0)" licence[3], in
addition to the Apache 2 licence where it

/trade marks/
"cop" is an unregistered trade mark of
Tim McNamara under New Zealand law.

/Consumer Guarantees Act 1993/
If you use this software for personal use,
you have certain rights under the Consumer
Guarantees Act 1993. They are explained by
the Ministry of Consumer Affairs[4].

Generally speaking, cop must be of
"acceptable" quality and be fit for the
purposes described.

However, when judging acceptable quality,
please consider that you are downloading
free software from the Internet.

I'm not entirely certain whether
software code counts as a good or a
service for the purpose of the Act

No, we can't opt out of these provisions.
Irrespective of the terms of the licence.


"Structure and Interpretation of Computer Programs"*
Chaper 5: Sequences and Coroutines (esp. 5.3 "Coroutines")

This is not the original SICP, written in Scheme, but
a version which is Python specific.


[1] @timClicks

Project details

Release history Release notifications

This version


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for cop, version 0.2
Filename, size File type Python version Upload date Hashes
Filename, size cop-0.2.tar.gz (6.2 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page