Overview
Datapiper provides a flexible easy-to-use library for constructing and running simple data batch processing pipelines.
Give Datapiper your list of data processing callables and it will construct a runnable data pipeline for you.
If you instantiate the pipe with a (iterable) data source, you get a generator that reads from a source and outputs processed data for you:
>>> operations = [lambda context, data: data+1]
>>> datasource = [1,2,3]
>>> p = Piper(operations, source=datasource)
>>> print p
pipe: source > <lambda>
>>> [r for r in p]
[2,3,4]
If you instead instantiate it with a (callable) data sink, you get a coroutine that accepts data from a producer and delivers processed data to a sink:
>>> operations = [lambda context, data: data+1]
>>> results = []
>>> def datasink(data):
... results.append(data)
>>> p = Piper(operations, sink=datasink)
>>> print p
pipe: <lambda> > sink
>>> for v in (1,2,3):
... p.send(v)
...
>>> results
[2,3,4]
The context parameter passed to the data operations callables is meant for sharing state between them. It can be initialized to desired value(s) by passing it to the Piper class as a (optional) keyword argument. The context parameter can be anything; a dictionary is recommended.
Please see the tests for more examples.
History
0.1.0 (2017-10-31)
First release on PyPI.
Metadata
Release files for datapiper 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| datapiper-0.1.0.zip | 23.7 kB | Without provenance | Details for datapiper-0.1.0.zip |
Release files / datapiper-0.1.0.zip
| Download URL | datapiper-0.1.0.zip |
|---|---|
| Size | 23.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
362afe8010344ab845ab4cf12b2c615631464a3fa6c1055e07a1bac996be26f3
|
|
BLAKE2b-256 checksum How to use checksums |
3c246251fe73e543b8d49deb3f3c750b417f84d087c7a6da5c2943aaf57aa3e8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
Security for datapiper 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.