esi-acme 0.2a1.dev0

Asynchronous Computing Made Easy

ACME: Asynchronous Computing Made Easy

main: dev:

Summary

The objective of ACME (pronounced "ak-mee") is to provide easy-to-use wrappers for calling Python functions in parallel ("embarassingly parallel workloads"). ACME is developed at the Ernst Strüngmann Institute (ESI) gGmbH for Neuroscience in Cooperation with Max Planck Society and released free of charge under the BSD 3-Clause "New" or "Revised" License. ACME relies on the concurrent processing library Dask and was primarily designed to facilitate the use of SLURM on the ESI HPC cluster. However, local multi-processing hardware (i.e., multi-core CPUs) is fully supported as well. ACME is based on the parallelization engine used in SyNCoPy and is itself part of the SyNCoPy package.

Installation

ACME can be installed with pip

pip install esi-acme

To get the latest development version, simply clone our GitHub repository:

git clone https://github.com/esi-neuroscience/acme.git

Usage

Basic Examples

Simplest use, everything is done automatically.

from acme import ParallelMap

def f(x, y, z=3):
  return (x + y) * z

with ParallelMap(f, [2, 4, 6, 8], 4) as pmap:
  pmap.compute()

Intermediate Examples

Set number of function calls via n_inputs

import numpy as np
from acme import ParallelMap

def f(x, y, z=3, w=np.zeros((3, 1)), **kwargs):
    return (sum(x) + y) * z * w.max()

pmap = ParallelMap(f, [2, 4, 6, 8], [2, 2], z=np.array([1, 2]), w=np.ones((8, 1)), n_inputs=2)

with pmap as p:
  p.compute()

Advanced Use

Allocate custom client object and recycle it for several computations

import numpy as np
from acme import ParallelMap, esi_cluster_setup

def f(x, y, z=3, w=np.zeros((3, 1)), **kwargs):
    return (sum(x) + y) * z * w.max()

def g(x, y, z=3, w=np.zeros((3, 1)), **kwargs):
    return (max(x) + y) * z * w.sum()

n_jobs = 200
client = esi_cluster_setup(partition="8GBXS", n_jobs=n_jobs)

x = [2, 4, 6, 8]
z = range(n_jobs)
w = np.ones((8, 1))

pmap = ParallelMap(f, x, np.random.rand(n_jobs), z=z, w=w, n_inputs=n_jobs)
with pmap as p:
    p.compute()

pmap = ParallelMap(g, x, np.random.rand(n_jobs), z=z, w=w, n_inputs=n_jobs)
with pmap as p:
    p.compute()

Handling results

Load results from files

The results are saved to disk in HDF5 format and the filenames are returned as a list of strings.

with ParallelMap(f, [2, 4, 6, 8], 4) as pmap:
  filenames = pmap.compute()

Example loading code:

out = np.zeros((4))
import h5py
for ii, fname in enumerate(filenames):
    with h5py.File(fname, 'r') as f:
        out[ii] = np.array(f['result_0'])

Collect results in local memory

This is possible but not recommended.

with ParallelMap(f, [2, 4, 6, 8], 4, write_worker_results=False) as pmap:
  results = pmap.compute() # returns a list of outputs

Debugging

Use the debug keyword to perform all function calls in the local thread of the active Python interpreter

with ParallelMap(f, [2, 4, 6, 8], 4, z=None) as pmap:
    results = pmap.compute(debug=True)

This way tools like pdb or %debug IPython magics can be used.

Documentation and Contact

To report bugs or ask questions please use our GitHub issue tracker. More usage details and background information is available in our online documentation.