Skip to main content

Overview

Datapiper provides a flexible easy-to-use library for constructing and running simple data batch processing pipelines.

Give Datapiper your list of data processing callables and it will construct a runnable data pipeline for you.

If you instantiate the pipe with a (iterable) data source, you get a generator that reads from a source and outputs processed data for you:

>>> operations = [lambda context, data: data+1]
>>> datasource = [1,2,3]
>>> p = Piper(operations, source=datasource)
>>> print p
pipe: source > <lambda>
>>> [r for r in p]
[2,3,4]

If you instead instantiate it with a (callable) data sink, you get a coroutine that accepts data from a producer and delivers processed data to a sink:

>>> operations = [lambda context, data: data+1]
>>> results = []
>>> def datasink(data):
...    results.append(data)
>>> p = Piper(operations, sink=datasink)
>>> print p
pipe: <lambda> > sink
>>> for v in (1,2,3):
...    p.send(v)
...
>>> results
[2,3,4]

The context parameter passed to the data operations callables is meant for sharing state between them. It can be initialized to desired value(s) by passing it to the Piper class as a (optional) keyword argument. The context parameter can be anything; a dictionary is recommended.

Please see the tests for more examples.

History

0.1.0 (2017-10-31)

  • First release on PyPI.

Metadata

Release files for datapiper 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for datapiper 0.1.0
File Size Uploaded Provenance Actions
datapiper-0.1.0.zip 23.7 kB Without provenance Details for datapiper-0.1.0.zip

Release files / datapiper-0.1.0.zip

Download URL datapiper-0.1.0.zip
Size 23.7 kB
Tags Source
SHA-256 checksum
How to use checksums
362afe8010344ab845ab4cf12b2c615631464a3fa6c1055e07a1bac996be26f3
BLAKE2b-256 checksum
How to use checksums
3c246251fe73e543b8d49deb3f3c750b417f84d087c7a6da5c2943aaf57aa3e8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Security for datapiper 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

1 release file No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page