A collection of patterns to use over top of the built in multiprocessing package
Project description
mp_boilerplate
A collection of patterns to use over top of the built in multiprocessing package.
Install
pip install mp_boilerplate
Use Case: (E)xtract (P)arallel (T)ransform (S)ave
The extract parallel transform save (EPTS) use case is as follows:
- A producer(single)/consumer(multiple) that applies a transform
- A producer(multiple)/consumer(single) that saves the transform
This usually means: read a file/folder, do something to each item, save the result.
import mp_boilerplate as mpb
import typing as t
from typeguard import typechecked
def extract() -> t.Iterator:
for i in range(100):
yield i
def transform(i):
return i + 1
def save(items: t.Iterator):
for item in items:
print(item)
if __name__ == '__main__':
worker = mpb.EPTS(extract, transform, save)
worker.start()
worker.join()
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
mp_boilerplate-0.3.3.tar.gz
(3.9 kB
view hashes)
Built Distribution
Close
Hashes for mp_boilerplate-0.3.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7fe5c4afa2ee673f2d65407388589c4c7daaafd2ee716ae42e0d538c0ec9529b |
|
MD5 | 2ad006f74c1e736eaa83f5f0a3133569 |
|
BLAKE2b-256 | ed16719da3ffaa59ccc21ed95c37f103d1c5c9e91c056a5086aaf4bbe4f650c6 |