lightweight but versatile python-framework for multi-stage information processing
Project description
data-plumber
data-plumber
is a lightweight but versatile python-framework for multi-stage
information processing. It allows to construct processing pipelines from both
atomic building blocks and via recombination of existing pipelines. Forks
enable more complex (i.e. non-linear) orders of execution. Pipelines can also
be collected into arrays that can be executed at once with the same input
data.
Minimal usage example
Consider a scenario where the contents of a dictionary have to be validated and a suitable error message has to be generated. Specifically, a valid input- dictionary is expected to have a key "data" with the respective value being a list of integer numbers. A suitable pipeline might look like this
>>> from data_plumber import Stage, Pipeline, Previous
>>> pipeline = Pipeline(
Stage(
primer=lambda **kwargs: "data" in kwargs,
status=lambda primer, **kwargs: 0 if primer else 1,
message=lambda primer, **kwargs: "" if primer else "missing key"
),
Stage(
requires={Previous: 0},
primer=lambda data, **kwargs: isinstance(data, list),
status=lambda primer, **kwargs: 0 if primer else 1,
message=lambda primer, **kwargs: "" if primer else "bad type"
),
Stage(
requires={Previous: 0},
primer=lambda data, **kwargs: all(isinstance(i, int) for i in data),
status=lambda primer, **kwargs: 0 if primer else 1,
message=lambda primer, **kwargs: "validation success" if primer else "bad type in data"
),
exit_on_status=1
)
>>> pipeline.run(**{}).stages
[('missing key', 1)]
>>> pipeline.run(**{"data": 1}).stages
[('', 0), ('bad type', 1)]
>>> pipeline.run(**{"data": [1, "2", 3]}).stages
[('', 0), ('', 0), ('bad type in data', 1)]
>>> pipeline.run(**{"data": [1, 2, 3]}).stages
[('', 0), ('', 0), ('validation success', 0)]
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for data_plumber-1.4.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dc7be4428f805af5c9bb4466d57baf8ff1a7ddc0f51dbb6aa22fa876b929c47c |
|
MD5 | 0c38fedd9c4c64333e19048e64d2aedc |
|
BLAKE2b-256 | 2b1f589ec22b9eaf096cbf4bd54dfba3810234a87c478e1161f059cb25155e34 |