Genpipes
Typed, lazy Python data pipelines using decorators and generators. Bind source arguments, compose stream processors, and consume the result when needed.
Genpipes requires Python 3.12+ and has no runtime dependencies. It provides synchronous stream composition; scheduling, persistence, retries, and distributed execution are outside its scope.
Installation
This branch prepares 3.0.0, with breaking changes from 2.x. To try v3 before it is published, install from a checkout of this branch:
python -m pip install .
After the 3.x release is published to PyPI:
python -m pip install "genpipes>=3,<4"
Existing users should read the v2 to v3 migration guide. The CI matrix covers CPython 3.12–3.15 on Linux.
Quick start
from collections.abc import Iterable, Iterator
from genpipes import processor, source
@source
def numbers(stop: int) -> Iterator[int]:
yield from range(stop)
@processor
def multiply(stream: Iterable[int], factor: int) -> Iterator[int]:
for value in stream:
yield value * factor
pipeline = numbers(3).then(multiply(10))
assert list(pipeline) == [0, 10, 20]
assert pipeline.run() == 20
assert list(pipeline) == [0, 10, 20] # A fresh traversal.
Calling a decorated source binds its arguments and returns a Pipeline[T].
Calling a decorated processor binds every argument after its upstream iterable;
pass the resulting callable to .then(...). The type checker tracks each
stage's item type and rejects incompatible connections.
Constructing, extending, or obtaining an iterator from a pipeline does not invoke its source or processors. Execution starts when that iterator is consumed. Each traversal invokes each reached stage once; processors decide how much of their upstream stream to consume.
Items and batches
Sources return iterators. Use yield for one item or batch and yield from
for multiple items. Genpipes does not guess whether an iterable is a batch.
from collections.abc import Iterator
from genpipes import source
@source
def batches() -> Iterator[list[int]]:
yield [1, 2, 3]
assert list(batches()) == [[1, 2, 3]]
The same rule applies to strings, dictionaries, and pandas DataFrames. For DataFrame pipelines, install pandas separately and explicitly yield each frame:
from collections.abc import Iterable, Iterator
import pandas as pd
from genpipes import processor, source
@source
def frames() -> Iterator[pd.DataFrame]:
yield pd.DataFrame({"id": [1, 1, 2]})
@processor
def deduplicate(
stream: Iterable[pd.DataFrame], subset: list[str]
) -> Iterator[pd.DataFrame]:
for frame in stream:
yield frame.drop_duplicates(subset=subset)
result = frames().then(deduplicate(["id"])).run()
assert result is not None
assert result["id"].tolist() == [1, 2]
Composition without decorators
Decorators are argument-binding helpers. A plain source factory and processor work directly:
from collections.abc import Iterable, Iterator
from genpipes import Pipeline
def load() -> Iterator[int]:
return iter([1, 2, 3])
def render(stream: Iterable[int]) -> Iterator[str]:
return (f"item:{value}" for value in stream)
original = Pipeline(load)
rendered = original.then(render)
assert list(original) == [1, 2, 3]
assert list(rendered) == ["item:1", "item:2", "item:3"]
.then(...) returns a new pipeline, so a prefix can be reused in multiple
branches. To combine existing pipelines, yield from them in a new source:
def combined() -> Iterator[int]:
yield from original
yield from original
assert list(Pipeline(combined)) == [1, 2, 3, 1, 2, 3]
Execution contract
iter(pipeline)creates an independent lazy traversal;list(pipeline)collects all emitted items.run()consumes a traversal and returns its last item, orNoneif empty. An empty stream and a last item ofNoneintentionally have the same result.- A source must create a fresh iterator per call. Reusing a captured, exhausted iterator does not make its data replayable.
- Pipeline execution state is local to each traversal. Bound arguments and emitted objects are not copied; their mutability and external side effects remain the caller's responsibility.
- Exceptions propagate to the consumer. A later traversal starts fresh; it does not resume or retry the failed traversal.
- Laziness follows the processors you write. A processor that materializes its entire input will do so when reached. Manage external resource lifetimes in your source/processor code, especially when stopping consumption early.
The public interface is Pipeline, source, and processor, imported from
genpipes. The distribution includes py.typed for downstream type checking.
Development
Install uv and work from the repository root:
uv sync --locked
uv run ruff check .
uv run ruff format --check .
uv run mypy
uv run pytest --cov --cov-report=term-missing
uv run python examples/quickstart.py
uv build --no-sources
uv run twine check --strict dist/*
Use uv run ruff format . to apply formatting locally. Commit uv.lock when
changing development dependencies. CI uses the same locked environment, checks
types and formatting, runs the Python matrix, and tests the built wheel outside
the checkout.
See release instructions for the GitHub OIDC/PyPI Trusted Publishing setup and CHANGELOG.md for v3 changes.
License
GNU GPL version 3. The license file is unchanged by v3.
Metadata
Release files for genpipes 3.0.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| genpipes-3.0.0.tar.gz | 95.2 kB | With provenance | Details for genpipes-3.0.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| genpipes-3.0.0-py3-none-any.whl | Python 3 | none | any | Details for genpipes-3.0.0-py3-none-any.whl |
Total release size: 113.2 kB
Release files / genpipes-3.0.0.tar.gz
| Download URL | genpipes-3.0.0.tar.gz |
|---|---|
| Size | 95.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
dba2b9559f121afbda3466584503ba7aca772cd9b6e8a7185d4e617a804eb323
|
|
BLAKE2b-256 checksum How to use checksums |
1017bbc36c6186e4bd6f54df762b60be998e3942ceb5f1d5078074539306e28e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 1, 2026.
Transparency logRelease files / genpipes-3.0.0-py3-none-any.whl
| Download URL | genpipes-3.0.0-py3-none-any.whl |
|---|---|
| Size | 18.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
016449a3646156b99377460f6f1c99bb4650be50b9a30feae27ed3d0623624dc
|
|
BLAKE2b-256 checksum How to use checksums |
4142c818a4de69e683670bbf08388f7dcd1d137557f76a05c74f924d043a6773
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 1, 2026.
Transparency logSecurity for genpipes 3.0.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for genpipes-3.0.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 1, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.