tqdm_pathos
Project description
tqdm_pathos
Progress bars for multiprocessing with pathos
Wrappers based on parmap for multiprocessing with pathos and progress bar completion with tqdm. Following parmap, multiprocessing is extended to functions of multiple iterables, arguments, and keyword arguments.
While parmap includes these extensions and a progress bar, it is built on the default multiprocessing library. The multiprocessing/pools module in pathos includes enhanced serialization to allow multiprocessing of, e.g., lambda functions, class methods, etc.
Usage
A pool with an automatically detected number of cores is set up by default. To choose the number of cores, use the n_cpus
kwarg.
Alternatively, an existing pool can be used by passing it to the pool
kwarg.
Function of a single iterable:
f = lambda x: x**2
iterable = [1, 2, 3]
# Serial
y = [f(x) for x in iterable]
# Parallel
y = tqdm_pathos.map(f, iterable)
Function of a single iterable, with non-iterable args and kwargs:
def f(x, a, b=0):
return x**2 * a + b
iterable = [1, 2, 3]
a = 1
b = 0
# Serial
y = [f(x, a, b=b) for x in iterable]
# Parallel
y = tqdm_pathos.map(f, iterable, a, b=b)
Function of multiple iterables:
f = lambda x, y: x * y
iterable1 = [1, 2, 3]
iterable2 = [4, 5, 6]
# Serial
z = [f(x, y) for x, y in zip(iterable1, iterable2)]
# Parallel
z = tqdm_pathos.starmap(f, zip(iterable1, iterable2))
Function of multiple iterables, with non-iterable args and kwargs:
def f(x, y, a, b=0):
return x * y * a + b
iterable1 = [1, 2, 3]
iterable2 = [4, 5, 6]
a = 1
b = 0
# Serial
z = [f(x, y, a, b=b) for x, y in zip(iterable1, iterable2)]
# Parallel
z = tqdm_pathos.starmap(f, zip(iterable1, iterable2), a, b=b)
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for tqdm_pathos-0.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7d643f23e6a7b194b2e6fc3e5a583699d2dcdc3d096916083c013cceff155e64 |
|
MD5 | 7dbfd81c4df4805a71526099519ea3d7 |
|
BLAKE2b-256 | 22c55b5d16384d039cd184fefda4962c345fe51b7322d38ed461c85422e7ca3b |