Skip to main content

Genpipes

Typed, lazy Python data pipelines using decorators and generators. Bind source arguments, compose stream processors, and consume the result when needed.

Genpipes requires Python 3.12+ and has no runtime dependencies. It provides synchronous stream composition; scheduling, persistence, retries, and distributed execution are outside its scope.

Installation

This branch prepares 3.0.0, with breaking changes from 2.x. To try v3 before it is published, install from a checkout of this branch:

python -m pip install .

After the 3.x release is published to PyPI:

python -m pip install "genpipes>=3,<4"

Existing users should read the v2 to v3 migration guide. The CI matrix covers CPython 3.12–3.15 on Linux.

Quick start

from collections.abc import Iterable, Iterator

from genpipes import processor, source


@source
def numbers(stop: int) -> Iterator[int]:
    yield from range(stop)


@processor
def multiply(stream: Iterable[int], factor: int) -> Iterator[int]:
    for value in stream:
        yield value * factor


pipeline = numbers(3).then(multiply(10))

assert list(pipeline) == [0, 10, 20]
assert pipeline.run() == 20
assert list(pipeline) == [0, 10, 20]  # A fresh traversal.

Calling a decorated source binds its arguments and returns a Pipeline[T]. Calling a decorated processor binds every argument after its upstream iterable; pass the resulting callable to .then(...). The type checker tracks each stage's item type and rejects incompatible connections.

Constructing, extending, or obtaining an iterator from a pipeline does not invoke its source or processors. Execution starts when that iterator is consumed. Each traversal invokes each reached stage once; processors decide how much of their upstream stream to consume.

Items and batches

Sources return iterators. Use yield for one item or batch and yield from for multiple items. Genpipes does not guess whether an iterable is a batch.

from collections.abc import Iterator

from genpipes import source


@source
def batches() -> Iterator[list[int]]:
    yield [1, 2, 3]


assert list(batches()) == [[1, 2, 3]]

The same rule applies to strings, dictionaries, and pandas DataFrames. For DataFrame pipelines, install pandas separately and explicitly yield each frame:

from collections.abc import Iterable, Iterator

import pandas as pd

from genpipes import processor, source


@source
def frames() -> Iterator[pd.DataFrame]:
    yield pd.DataFrame({"id": [1, 1, 2]})


@processor
def deduplicate(
    stream: Iterable[pd.DataFrame], subset: list[str]
) -> Iterator[pd.DataFrame]:
    for frame in stream:
        yield frame.drop_duplicates(subset=subset)


result = frames().then(deduplicate(["id"])).run()
assert result is not None
assert result["id"].tolist() == [1, 2]

Composition without decorators

Decorators are argument-binding helpers. A plain source factory and processor work directly:

from collections.abc import Iterable, Iterator

from genpipes import Pipeline


def load() -> Iterator[int]:
    return iter([1, 2, 3])


def render(stream: Iterable[int]) -> Iterator[str]:
    return (f"item:{value}" for value in stream)


original = Pipeline(load)
rendered = original.then(render)

assert list(original) == [1, 2, 3]
assert list(rendered) == ["item:1", "item:2", "item:3"]

.then(...) returns a new pipeline, so a prefix can be reused in multiple branches. To combine existing pipelines, yield from them in a new source:

def combined() -> Iterator[int]:
    yield from original
    yield from original


assert list(Pipeline(combined)) == [1, 2, 3, 1, 2, 3]

Execution contract

  • iter(pipeline) creates an independent lazy traversal; list(pipeline) collects all emitted items.
  • run() consumes a traversal and returns its last item, or None if empty. An empty stream and a last item of None intentionally have the same result.
  • A source must create a fresh iterator per call. Reusing a captured, exhausted iterator does not make its data replayable.
  • Pipeline execution state is local to each traversal. Bound arguments and emitted objects are not copied; their mutability and external side effects remain the caller's responsibility.
  • Exceptions propagate to the consumer. A later traversal starts fresh; it does not resume or retry the failed traversal.
  • Laziness follows the processors you write. A processor that materializes its entire input will do so when reached. Manage external resource lifetimes in your source/processor code, especially when stopping consumption early.

The public interface is Pipeline, source, and processor, imported from genpipes. The distribution includes py.typed for downstream type checking.

Development

Install uv and work from the repository root:

uv sync --locked
uv run ruff check .
uv run ruff format --check .
uv run mypy
uv run pytest --cov --cov-report=term-missing
uv run python examples/quickstart.py
uv build --no-sources
uv run twine check --strict dist/*

Use uv run ruff format . to apply formatting locally. Commit uv.lock when changing development dependencies. CI uses the same locked environment, checks types and formatting, runs the Python matrix, and tests the built wheel outside the checkout.

See release instructions for the GitHub OIDC/PyPI Trusted Publishing setup and CHANGELOG.md for v3 changes.

License

GNU GPL version 3. The license file is unchanged by v3.

Metadata

Release files for genpipes 3.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for genpipes 3.0.0
File Size Uploaded Provenance Actions
genpipes-3.0.0.tar.gz 95.2 kB With provenance Details for genpipes-3.0.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for genpipes 3.0.0
File Interpreter ABI Platform Actions
genpipes-3.0.0-py3-none-any.whl Python 3 none any Details for genpipes-3.0.0-py3-none-any.whl

Total release size: 113.2 kB

Release files / genpipes-3.0.0.tar.gz

Download URL genpipes-3.0.0.tar.gz
Size 95.2 kB
Tags Source
SHA-256 checksum
How to use checksums
dba2b9559f121afbda3466584503ba7aca772cd9b6e8a7185d4e617a804eb323
BLAKE2b-256 checksum
How to use checksums
1017bbc36c6186e4bd6f54df762b60be998e3942ceb5f1d5078074539306e28e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 1, 2026.

Transparency log

Release files / genpipes-3.0.0-py3-none-any.whl

Download URL genpipes-3.0.0-py3-none-any.whl
Size 18.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
016449a3646156b99377460f6f1c99bb4650be50b9a30feae27ed3d0623624dc
BLAKE2b-256 checksum
How to use checksums
4142c818a4de69e683670bbf08388f7dcd1d137557f76a05c74f924d043a6773
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 1, 2026.

Transparency log

Security for genpipes 3.0.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for genpipes-3.0.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 1, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page