Skip to main content

plumbum

CI PyPI Python 3.12+ License: MIT Code style: ruff Coverage

A plumbing syntax for Python that provides a clear distinction between data and operators, enabling composable function pipelines.

Overview

plumbum is a library for threading data through function calls using intuitive pipe operators. Inspired by Pipe, it offers a redesigned approach with a separation between operator construction and execution.

This is primarily a syntax library focused on making data transformations more readable and composable. It is not optimized for performance-critical applications, but rather caters to the aesthetics of @habemus-papadum -- so, your mileage may vary...

Installation

pip install habemus-papadum-plumbum

Usage

# Basics
5 > add(1) | mul(2)  # 12

# Iterators
[1, 2, 3] > select(mul(3) | add(1)) | where(even) | list # [4, 10]

# Async
await (5 > async_add(1) | async_mul(2))  # 12
await ([1, 2, 3] > aiter | aselect(async_add(1)) | alist) # [2, 3, 4]

# Pipelines are reusable values
op = add(1) | multiply(2)
op2 = add(3) | multiply(12)
6 > op | op2  # 204

Mental Model

  • Compose first, execute later. The | operator chains together @pb/@apb operators (and plain callables) without running them. The pipeline becomes a first-class value that you can store, combine, or pass around.
  • Thread data explicitly. The > operator injects the left-hand value into the pipeline’s first argument. That works for numbers, strings, dicts, iterables, custom classes—any Python object.
  • Async is contagious by design. Introducing an async operator (created with @apb, an async iterator helper, or any awaitable) upgrades the whole pipeline so value > pipeline returns a coroutine. Call it with await.
  • Iterables stay lazy. Synchronous helpers like select, where, take, or chain return iterators; async counterparts (aselect, awhere, aiter, …) yield async iterators. Add a materializer (list, tuple, alist) when you actually need the concrete collection.

Features

  • Compatible with arbitrary data types; the library focuses on syntax and composability rather than constraining what flows through the pipes.
  • Seamless sync/async mixing: pipelines promote themselves to async when an async stage is present, including adapters for awaitables returned by sync operators.
  • Rich iterable toolkits for sync (pdum.plumbum.iterops) and async (pdum.plumbum.aiterops) flows, covering batching, zipping, traversal, networking, and more.(Modelled after the operators fromPipe)
  • jq-inspired JSON utilities (pdum.plumbum.jq) for parsing dotted paths, querying nested data, and performing immutable transformations—plus async counterparts.

Style Guide

  • Favor | for composition. Build operators with | so you can refactor pipelines into reusable pieces. Use > sparingly—ideally once—when you finally thread data through the pipeline. Remember that | binds more tightly than >, so value > a | b means value > (a | b); add parentheses only when you truly need different grouping.
  • Iterators remain lazy by default. Iterator pipelines usually yield another iterator. In tests or scripts, append a materializer such as | list when you need concrete values. Plain callables are auto-wrapped; no need for pb(list).
  • Sync to async promotion is automatic. Pipelines built from @pb operators seamlessly adopt async semantics when an @apb stage (or any awaitable) appears. Keep composing with |; the result becomes awaitable.
  • Await async chains. Any pipeline that includes async operators returns a coroutine. Execute it with await (value > pipeline) (or equivalently await pipeline(value)).
  • Collect async results with alist / acollect. Finish async iterator pipelines with | alist (alias | acollect) when you need a list in memory.
  • Avoid chained > comparisons. Python treats x > a > b as a chained comparison, which is incompatible with plumbum operators. Prefer x > a | b; only use (x > a) > b when you intentionally need two separate execution steps.

Tutorial Notebook

Learn the details of synchronous and asynchronous pipelines by following along step-by-step in the Tutorial, which hosts the full quick-start tour covering core operators, partial application, iterable helpers, and async variants.

Acknowledgements

  • plumbum draws direct inspiration from Julien Palard’s Pipe; several iterable operators (for example select, where, and dedup) started life there before being reworked for plumbum’s APIs.
  • Key differences from Pipe:
    • Operators are inert values: | builds pipelines while > triggers execution, making composition and reuse explicit.
    • Async is a core capability: sync and async operators coexist in the same pipeline, and iterator helpers come in both synchronous and asynchronous flavors.
    • The bundled pdum.plumbum.jq module adds a jq-inspired path/query layer, expanding the original operator toolbox into structured-data transformations.
  • The codebase was authored primarily with AI assistance—roughly 95% by OpenAI Codex and 5% by Claude Code—with artistic direction and stewardship by @habemus-papadum.

Development

This project uses UV for dependency management.

Setup

# Install UV if you haven't already
curl -LsSf https://astral.sh/uv/install.sh | sh

# Clone the repository
git clone https://github.com/habemus-papadum/pdum_plumbum.git
cd pdum_plumbum

# Provision the entire toolchain (uv sync, pnpm install, widget build, pre-commit hooks)
./scripts/setup.sh

Important for Development:

  • ./scripts/setup.sh is idempotent—rerun it after pulling dependency changes
  • Use uv sync --frozen to ensure the lockfile is respected when installing Python deps

Running Tests

# Run all tests
uv run pytest

# Run a specific test file
uv run pytest tests/test_example.py

# Run a specific test function
uv run pytest tests/test_example.py::test_version

# Run tests with coverage
uv run pytest --cov=src/pdum/plumbum --cov-report=xml --cov-report=term

Code Quality

# Check code with ruff
uv run ruff check .

# Format code with ruff
uv run ruff format .

# Fix auto-fixable issues
uv run ruff check --fix .

Building

# Build Python
./scripts/build.sh

# Or build just the Python distribution artifacts
uv build

Releasing

Releases run from the release GitHub Actions workflow, which calls ship:

# Plan first (the default), then release for real
gh workflow run release.yml -f bump=minor -f dry_run=true
gh workflow run release.yml -f bump=minor -f dry_run=false

Each release is tagged habemus-papadum-plumbum-v<version> on the commit it was built from. Between releases pyproject.toml carries <last release>+dev; ship sets the release version, builds, tests, publishes to PyPI, tags, and restores the development marker. To see the plan locally:

uv run ship release habemus-papadum-plumbum --bump minor --dry-run

Install the git hooks (including the ship-check hook) with uv run pre-commit install; ./scripts/setup.sh does this.

Automation scripts

  • ./scripts/setup.sh – bootstrap uv and pre-commit hooks
  • ./scripts/build.sh – sync the environment the way CI does
  • ./scripts/test_notebooks.sh – execute demo notebooks (uses ./scripts/nb.sh under the hood)
  • ./scripts/setup-visual-tests.sh – install Playwright browsers for visual tests

License

MIT License - see LICENSE file for details.

Metadata

Release files for habemus-papadum-plumbum 0.6.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for habemus-papadum-plumbum 0.6.1
File Size Uploaded Provenance Actions
habemus_papadum_plumbum-0.6.1.tar.gz 166.6 kB Without provenance Details for habemus_papadum_plumbum-0.6.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for habemus-papadum-plumbum 0.6.1
File Interpreter ABI Platform Actions
habemus_papadum_plumbum-0.6.1-py3-none-any.whl Python 3 none any Details for habemus_papadum_plumbum-0.6.1-py3-none-any.whl

Total release size: 191.0 kB

Release files / habemus_papadum_plumbum-0.6.1.tar.gz

Download URL habemus_papadum_plumbum-0.6.1.tar.gz
Size 166.6 kB
Tags Source
SHA-256 checksum
How to use checksums
7ee6f091e2bba7a1ec6ce1a83929b5d8b1ed8c31b9550d568b9da0f1501a078c
BLAKE2b-256 checksum
How to use checksums
189d9ec2aa29524f031d403c5769803615a5567750654e60f207a74d6f2be5fb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.12.17 {"installer":{"name":"uv","version":"0.12.17","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / habemus_papadum_plumbum-0.6.1-py3-none-any.whl

Download URL habemus_papadum_plumbum-0.6.1-py3-none-any.whl
Size 24.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
6e5f9bf185a3d2d3d585ccdd7ad83ecf530b4f0e468ac84ea4c60fe906b1262a
BLAKE2b-256 checksum
How to use checksums
4868f110d740d7843d1bb00c3d12c4f8d34d0198d5d3c8ff86707acd1ebcf7cb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.12.17 {"installer":{"name":"uv","version":"0.12.17","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for habemus-papadum-plumbum 0.6.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page