larzpipe

Lazy, composable data pipelines. Pure Python, zero dependencies.

Instead of nesting map(filter(...)) inside out, or building intermediate lists, chain the operations left-to-right and evaluate lazily. Nothing runs until you ask for a result, so infinite and huge sources are fine.

from larzpipe import pipe

(pipe(range(1_000_000))
    .filter(lambda x: x % 2 == 0)
    .map(lambda x: x * x)
    .take(5)
    .to_list())                 # [0, 4, 16, 36, 64]

pipe(words).map(str.lower).distinct().group_by(len)

Why

• Reads top-to-bottom. The order you write is the order data flows - no more reading map(filter(map(...))) from the inside out.
• Lazy. Everything is generator-based; take(5) off an infinite source does five items of work. Re-iterable when the source is (lists, ranges).
• Batteries. map filter flat_map take/skip take_while/drop_while distinct(key) batch window enumerate tap sort zip chain, plus terminals to_list/to_dict/reduce/group_by/count/first/sum/min/ max/any/all.
• Zero dependencies. Just a friendly face over itertools.

Install

pip install larzpipe

Usage

from larzpipe import pipe

pipe(data).filter(pred).map(fn).batch(100).for_each(save)
pipe(logs).map(parse).distinct(key=lambda e: e.id).group_by(lambda e: e.level)
pipe(nums).take_while(lambda x: x < 100).reduce(lambda a, b: a + b)

Tests

python -m unittest discover -s tests -v   # 20 tests incl. laziness + re-iterability

The Larz stack

One of 30+ pure-Python, zero-dependency libraries at github.com/larz-scripter.

License

MIT (c) larz-scripter