A collection of patterns to use over top of the built in multiprocessing package
Reason this release was yanked:
bug in calling functions
Project description
mp_boilerplate
A collection of patterns to use over top of the built in multiprocessing package.
Install
pip install mp_boilerplate
Use Case: (E)xtract (P)arallel (T)ransform (S)ave
The extract parallel transform save (EPTS) use case is as follows:
- A producer(single)/consumer(multiple) that applies a transform
- A producer(multiple)/consumer(single) that saves the transform
This usually means: read a file/folder, do something to each item, save the result.
import mp_boilerplate as mpb
import typing as t
from typeguard import typechecked
def extract() -> t.Iterator:
for i in range(100):
yield i
def transform(i):
return i + 1
def save(items: t.Iterator):
for item in items:
print(item)
if __name__ == '__main__':
worker = mpb.EPTS(extract, transform, save)
worker.start()
worker.join()
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
mp_boilerplate-0.3.0.tar.gz
(3.4 kB
view hashes)
Built Distribution
Close
Hashes for mp_boilerplate-0.3.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 902902a56070a567785b104505c6f58b11ed553ab251c87c4f1e6616d8782c09 |
|
MD5 | b2b5851495df272a77c2ce4f31f98c63 |
|
BLAKE2b-256 | afaef3e463ddd2559c8579b2705812ae3835e592b20b3b9ab5825feb539a91c6 |