map and starmap implementations passing additional arguments and parallelizing if possible
Project description
This small python module implements two functions: map and starmap.
What does parmap offer?
Provide an easy to use syntax for both map and starmap.
Parallelize transparently whenever possible.
Handle multiple (positional -for now-) arguments as needed.
Installation:
pip install parmap
Usage:
Here are some examples with some unparallelized code parallelized with parmap:
import parmap # You want to do: y = [myfunction(x, argument1, argument2) for x in mylist] # In parallel: y = parmap.map(myfunction, mylist, argument1, argument2) # You want to do: z = [myfunction(x, y, argument1, argument2) for (x,y) in mylist] # In parallel: z = parmap.starmap(myfunction, mylist, argument1, argument2) # You want to do: listx = [1, 2, 3, 4, 5, 6] listy = [2, 3, 4, 5, 6, 7] param = 3.14 param2 = 42 listz = [] for (x, y) in zip(listx, listy): listz.append(myfunction(x, y, param1, param2)) # In parallel: listz = parmap.starmap(myfunction, zip(listx, listy), param1, param2)
map (and starmap on python 3.3) already exist. Why reinvent the wheel?
Please correct me if I am wrong, but from my point of view, existing functions have some usability limitations:
The built-in python function map [1] is not able to parallelize.
multiprocessing.Pool().starmap [2] is only available in python-3.3 and later versions.
multiprocessing.Pool().map [3] does not allow any additional argument to the mapped function.
multiprocessing.Pool().starmap allows passing multiple arguments, but in order to pass a constant argument to the mapped function you will need to convert it to an iterator using itertools.repeat(your_parameter) [4]
parmap aims to overcome this limitations in the simplest possible way.
Additional features in parmap:
Create a pool for parallel computation automatically if possible.
parmap.map(..., ..., parallel=False) # disables parallelization
parmap.map(..., ..., chunksize=3) # size of chunks (see multiprocessing.Pool().map)
parmap.map(..., ..., pool=multiprocessing.Pool()) # use an existing pool, in this case parmap will not close the pool.
To do:
Pull requests and suggestions are welcome.
Pass keyword arguments to functions?
Acknowledgments:
The original idea for this implementation was given by J.F. Sebastian. I just provided an alternative answer implementing it in a package.
Known works using parmap
Davide Gerosa, Michael Kesden, “PRECESSION. Dynamics of spinning black-hole binaries with python.” arXiv:1605.01067, 2016
References
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for parmap-1.3.0-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9d34552df102d08a99a37ea2afe5aaabafd988db5bde5cb7b975a0531ea8d477 |
|
MD5 | 4a0783a420e11d5dc1bc840974c48edc |
|
BLAKE2b-256 | 1eee4b87cbd23ae86e80e524b159baa13826d37b751c78c1fbe4dc1093cc7ed7 |