A collection of patterns to use over top of the built in multiprocessing package
Reason this release was yanked:
bug in transform_init
Project description
mp_boilerplate
A collection of patterns to use over top of the built in multiprocessing package.
Install
pip install mp_boilerplate
Use Case: (E)xtract (P)arallel (T)ransform (S)ave
The extract parallel transform save (EPTS) use case is as follows:
- A producer(single)/consumer(multiple) that applies a transform
- A producer(multiple)/consumer(single) that saves the transform
This usually means: read a file/folder, do something to each item, save the result.
import mp_boilerplate as mpb
import typing as t
from typeguard import typechecked
def extract() -> t.Iterator:
for i in range(100):
yield i
def transform(i):
return i + 1
def save(items: t.Iterator):
for item in items:
print(item)
if __name__ == '__main__':
worker = mpb.EPTS(extract, transform, save)
worker.start()
worker.join()
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
mp_boilerplate-0.3.2.tar.gz
(3.8 kB
view hashes)
Built Distribution
Close
Hashes for mp_boilerplate-0.3.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 158d545d1251e2afb8aef8345c8fd54cf65ef42c08cd6e6eb3078e608688900c |
|
MD5 | c26584e29e8ed501e247a847ef5bb7b7 |
|
BLAKE2b-256 | 9d94745104c5d844c06123ec8e9d604f321337c0e7547e38e4ac50d192f0428a |