Skip to main content

pipedata

Chained operations in Python, applied to data processing.

Installation

To install with all optional dependencies:

pip install pipedata[ops]

If you only want the core functionality (building pipelines), and not the data processing applications, then:

pip install pipedata

Examples

Chaining Data Operations

pipedata.ops provides some operations for streaming data through memory.

import json
import zipfile

import pyarrow.parquet as pq

from pipedata.core import Stream
from pipedata.ops import json_records, parquet_writer, zipped_files


data1 = [
    {"col1": 1, "col2": "Hello"},
    {"col1": 2, "col2": "world"},
]
data2 = [
    {"col1": 3, "col2": "!"},
]

with zipfile.ZipFile("test_input.json.zip", "w") as zipped:
    zipped.writestr("file1.json", json.dumps(data1))
    zipped.writestr("file2.json", json.dumps(data2))

result = (
    Stream(["test_input.json.zip"])
    .then(zipped_files)
    .then(json_records())
    .then(parquet_writer("test_output.parquet"))
    .to_list()
)

table = pq.read_table("test_output.parquet")
print(table.to_pydict())
#> {'col1': [1, 2, 3], 'col2': ['Hello', 'world', '!']}

Alternatively, you can construct the pipeline as a chain:

import pyarrow.parquet as pq

from pipedata.core import Chain, Stream
from pipedata.ops import json_records, parquet_writer, zipped_files

# Running this after input file created in above example
chain = (
    Chain()
    .then(zipped_files)
    .then(json_records())
    .then(parquet_writer("test_output_2.parquet"))
)
result = Stream(["test_input.json.zip"]).then(chain).to_list()
table = pq.read_table("test_output_2.parquet")
print(table.to_pydict())
#> {'col1': [1, 2, 3], 'col2': ['Hello', 'world', '!']}

Core Framework

The core framework provides the building blocks for chaining operations.

Running a stream:

from pipedata.core import Stream, ops


result = (
    Stream(range(10))
    .then(ops.filtering(lambda x: x % 2 == 0))
    .then(ops.mapping(lambda x: x ^ 2))
    .then(ops.batched(lambda x: x, 2))
    .to_list()
)
print(result)
#> [(2, 0), (6, 4), (10,)]

Creating a chain and then using it, this time using the pipe notation:

import json
from pipedata.core import Chain, Stream, ops


chain = (
    Chain()
    | ops.filtering(lambda x: x % 2 == 0)
    | ops.mapping(lambda x: x ^ 2)
    | ops.batched(lambda x: sum(x), 2)
)
print(Stream(range(10)).then(chain).to_list())
#> [2, 10, 10]
print(json.dumps(chain.get_counts(), indent=4))
#> [
#>     {
#>         "name": "_identity",
#>         "inputs": 10,
#>         "outputs": 10
#>     },
#>     {
#>         "name": "<lambda>",
#>         "inputs": 10,
#>         "outputs": 5
#>     },
#>     {
#>         "name": "<lambda>",
#>         "inputs": 5,
#>         "outputs": 5
#>     },
#>     {
#>         "name": "<lambda>",
#>         "inputs": 5,
#>         "outputs": 3
#>     }
#> ]

Similar Functionality

Metadata

Release files for pipedata 0.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pipedata 0.4
File Size Uploaded Provenance Actions
pipedata-0.4.tar.gz 8.0 kB With provenance Details for pipedata-0.4.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pipedata 0.4
File Interpreter ABI Platform Actions
pipedata-0.4-py3-none-any.whl Python 3 none any Details for pipedata-0.4-py3-none-any.whl

Total release size: 17.8 kB

Release files / pipedata-0.4.tar.gz

Download URL pipedata-0.4.tar.gz
Size 8.0 kB
Tags Source
SHA-256 checksum
How to use checksums
93960218c0f6b3afa576993475d8feac5216135af901a871566faed439154866
BLAKE2b-256 checksum
How to use checksums
3d6d62edee1d846942036f1447ebe1b72bde84332f0613ea9c1b52a4d145fd83
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.12.8

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Feb 6, 2025.

Transparency log

Release files / pipedata-0.4-py3-none-any.whl

Download URL pipedata-0.4-py3-none-any.whl
Size 9.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
1fd25a2d99a89818855baf625cfbbbc8675ee429e6ca2d02d24fd06b8b8c70e6
BLAKE2b-256 checksum
How to use checksums
7e809f211cc5001ce7e6b797f975e410d75133230f59846f196efa78b6a17f8e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.12.8

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Feb 6, 2025.

Transparency log

Security for pipedata 0.4

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for pipedata-0.4.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Feb 6, 2025.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page