tinyflow 0.1

Experimental in-memory data flow pipelines.

Experiments in data flow programming.

After some experimentation, Apache Beam’s Python SDK got the API right. Use that instead.

Standard Word Count Example

Grab the 5 most common words in LICENSE.txt

from collections import Counter

from tinyflow.serial import ops, Pipeline


pipe = Pipeline() \
    | "Split line into words" >> ops.flatmap(lambda x: x.lower().split()) \
    | "Remove empty lines" >> ops.filter(bool) \
    | "Produce the 5 most common words" >> ops.counter(5) \
    | "Sort by frequency desc" >> ops.sort(key=lambda x: x[1], reverse=True)

with open('LICENSE.txt') as f:
    results = dict(pipe(f))

Using only Python’s builtins:

from collections import Counter
import itertools as it

with open('LICENSE.txt') as f:
    lines = (line.lower().split() for line in f)
    words = it.chain.from_iterable(lines)
    count = Counter(words)
    results = dict(count.most_common(10))

Developing

$ git clone https://github.com/geowurster/tinyflow.git
$ cd tinyflow
$ pip install -e .\[all\]
$ pytest --cov tinyflow --cov-report term-missing

License

See LICENSE.txt

Changelog

See CHANGES.md