Asynchronous Computing Made Easy
ACME: Asynchronous Computing Made Easy
The objective of ACME (pronounced "ak-mee") is to provide easy-to-use wrappers for calling Python functions in parallel ("embarassingly parallel workloads"). ACME is developed at the Ernst Strüngmann Institute (ESI) gGmbH for Neuroscience in Cooperation with Max Planck Society and released free of charge under the BSD 3-Clause "New" or "Revised" License. ACME relies on the concurrent processing library Dask and was primarily designed to facilitate the use of SLURM on the ESI HPC cluster. However, local multi-processing hardware (i.e., multi-core CPUs) is fully supported as well. ACME is based on the parallelization engine used in SyNCoPy and is itself part of the SyNCoPy package.
ACME can be installed with pip
pip install esi-acme
To get the latest development version, simply clone our GitHub repository:
git clone https://github.com/esi-neuroscience/acme.git
Simplest use, everything is done automatically.
from acme import ParallelMap def f(x, y, z=3): return (x + y) * z with ParallelMap(f, [2, 4, 6, 8], 4) as pmap: pmap.compute()
Set number of function calls via
import numpy as np from acme import ParallelMap def f(x, y, z=3, w=np.zeros((3, 1)), **kwargs): return (sum(x) + y) * z * w.max() pmap = ParallelMap(f, [2, 4, 6, 8], [2, 2], z=np.array([1, 2]), w=np.ones((8, 1)), n_inputs=2) with pmap as p: p.compute()
client object and recycle it for several computations
import numpy as np from acme import ParallelMap, esi_cluster_setup def f(x, y, z=3, w=np.zeros((3, 1)), **kwargs): return (sum(x) + y) * z * w.max() def g(x, y, z=3, w=np.zeros((3, 1)), **kwargs): return (max(x) + y) * z * w.sum() n_jobs = 200 client = esi_cluster_setup(partition="8GBXS", n_jobs=n_jobs) x = [2, 4, 6, 8] z = range(n_jobs) w = np.ones((8, 1)) pmap = ParallelMap(f, x, np.random.rand(n_jobs), z=z, w=w, n_inputs=n_jobs) with pmap as p: p.compute() pmap = ParallelMap(g, x, np.random.rand(n_jobs), z=z, w=w, n_inputs=n_jobs) with pmap as p: p.compute()
Load results from files
The results are saved to disk in HDF5 format and the filenames are returned as a list of strings.
with ParallelMap(f, [2, 4, 6, 8], 4) as pmap: filenames = pmap.compute()
Example loading code:
out = np.zeros((4)) import h5py for ii, fname in enumerate(filenames): with h5py.File(fname, 'r') as f: out[ii] = np.array(f['result_0'])
Collect results in local memory
This is possible but not recommended.
with ParallelMap(f, [2, 4, 6, 8], 4, write_worker_results=False) as pmap: results = pmap.compute() out = np.array([xi for xi in results])
debug keyword to perform all function calls in the local thread of
the active Python interpreter
with ParallelMap(f, [2, 4, 6, 8], 4, z=None) as pmap: results = pmap.compute(debug=True)
This way tools like
%debug IPython magics can be used.
Documentation and Contact
Release history Release notifications | RSS feed
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size esi_acme-0.1b1.dev2-py3-none-any.whl (42.2 kB)||File type Wheel||Python version py3||Upload date||Hashes View|
|Filename, size esi-acme-0.1b1.dev2.tar.gz (36.7 kB)||File type Source||Python version None||Upload date||Hashes View|
Hashes for esi_acme-0.1b1.dev2-py3-none-any.whl