lightweight but versatile python-framework for multi-stage information processing
Project description
data-plumber
data-plumber
is a lightweight but versatile python-framework for multi-stage
information processing. It allows to construct processing pipelines from both
atomic building blocks and via recombination of existing pipelines. Forks
enable more complex (i.e. non-linear) orders of execution. Pipelines can also
be collected into arrays that can be executed at once with the same input
data.
Minimal usage example
Consider a scenario where the contents of a dictionary have to be validated and a suitable error message has to be generated. Specifically, a valid input- dictionary is expected to have a key "list" with the respective value being a list of integer numbers. A suitable pipeline might look like this
>>> from data_plumber import Stage, Pipeline, Previous
>>> pipeline = Pipeline(
Stage(
primer=lambda in_, **kwargs: "list" in in_,
status=lambda primer, **kwargs: 0 if primer else 1,
message=lambda primer, **kwargs: "" if primer else "missing key"
),
Stage(
requires={Previous: 0},
primer=lambda in_, **kwargs: isinstance(in_["list"], list),
status=lambda primer, **kwargs: 0 if primer else 1,
message=lambda primer, **kwargs: "" if primer else "bad type"
),
Stage(
requires={Previous: 0},
primer=lambda in_, **kwargs: all(isinstance(i, int) for i in in_["list"]),
status=lambda primer, **kwargs: 0 if primer else 1,
message=lambda primer, **kwargs: "validation success" if primer else "bad type in list"
),
exit_on_status=1
)
>>> pipeline.run(**{}).stages
[('missing key', 1)]
>>> pipeline.run(**{"list": 1}).stages
[('', 0), ('bad type', 1)]
>>> pipeline.run(**{"list": [1, 2, 3]}).stages
[('', 0), ('', 0), ('validation success', 0)]
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for data_plumber-0.1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cb15a90c46f4412070ce9f2039e96be251dce3f5dac3afcae35ad491ef3b8e33 |
|
MD5 | 1125c36c182c9ef9c95410c2dc726c11 |
|
BLAKE2b-256 | 1073882547bef6aef6883b7a3cbe67279e04e8ce0ae513e4aec96a68e6e9fcd5 |