A Python multithreading library for data processing pipelines, data streaming, etc.
Project description
multiflow
About
multiflow
is a Python multithreading library for data processing pipelines/workflows, streaming, etc. It extends concurrent.futures
by allowing the input and output to be generator objects. And, it makes it easy to string together multiple thread pools together to create a multithreaded pipeline.
Additionally, multiflow
comes with periodic logging, automatic retries, error handling, and argument expansion.
Why?
The ability to accept an input generator object while yielding an output generator object makes it ideal for concurrently doing multiple jobs where the output of the first job is the input of the second job. This means that it can start doing work on the second job before the first job completes; thus, completing the total work faster.
A great use case for this is streaming data. For example, with multiflow
and smart_open
, you could stream images from S3 and process them in a multithreaded environment before exporting them elsewhere.
Quickstart
from multiflow import MultithreadedFlow
image_paths = [] # list of images
def transform(image_path):
# do some work
return new_path
with MultithreadedFlow() as flow:
flow.consume(image_paths) # can accept generator object or iterable item
flow.add_function(transform)
for output in flow:
if output: # if successful
print(output) # new_path
else:
e = output.get_exception()
success = flow.get_successful_job_count()
failed = flow.get_failed_job_count()
Install
pip install multiflow
Documentation
The documentation is still a work in progress, but for the most up to date documentation, please see this page.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for multiflow-1.0.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 139d0c17705f6d3517d1b67cf0cbb9cf1ed32f0177ffd5fd009b39c521c0efb7 |
|
MD5 | 4e59cf9a7e806da53755c1fb853df450 |
|
BLAKE2b-256 | 406611d817919af0db7b275d77c54058b0c334799d88d0b7b067d6d0e97cc464 |