esi-acme 2022.11

Asynchronous Computing Made ESI

ACME: Asynchronous Computing Made ESI

main:

dev:

Summary

The objective of ACME (pronounced "ak-mee") is to provide easy-to-use wrappers for calling Python functions concurrently ("embarassingly parallel workloads"). ACME is developed at the Ernst Strüngmann Institute (ESI) gGmbH for Neuroscience in Cooperation with Max Planck Society and released free of charge under the BSD 3-Clause "New" or "Revised" License. ACME relies heavily on the concurrent processing library dask and was primarily designed to facilitate the use of SLURM on the ESI HPC cluster (although other HPC infrastructure running SLURM can be leveraged as well). Local multi-processing hardware (i.e., multi-core CPUs) is fully supported too. ACME is itself used as the parallelization engine of SyNCoPy.

Installation

ACME can be installed with pip

pip install esi-acme

or via conda

conda install -c conda-forge esi-acme

To get the latest development version, simply clone our GitHub repository:

git clone https://github.com/esi-neuroscience/acme.git
cd acme/
pip install -e .

Usage

Basic Examples

Simplest use, everything is done automatically.

from acme import ParallelMap

def f(x, y, z=3):
  return (x + y) * z

with ParallelMap(f, [2, 4, 6, 8], 4) as pmap:
  pmap.compute()

Intermediate Examples

Set number of function calls via n_inputs

import numpy as np
from acme import ParallelMap

def f(x, y, z=3, w=np.zeros((3, 1)), **kwargs):
    return (sum(x) + y) * z * w.max()

pmap = ParallelMap(f, [2, 4, 6, 8], [2, 2], z=np.array([1, 2]), w=np.ones((8, 1)), n_inputs=2)

with pmap as p:
  p.compute()

Advanced Use

Allocate custom client object and recycle it for several computations (use slurm_cluster_setup on non-ESI HPC infrastructure or local_cluster_setup when working on your local machine)

import numpy as np
from acme import ParallelMap, esi_cluster_setup

def f(x, y, z=3, w=np.zeros((3, 1)), **kwargs):
    return (sum(x) + y) * z * w.max()

def g(x, y, z=3, w=np.zeros((3, 1)), **kwargs):
    return (max(x) + y) * z * w.sum()

n_workers = 200
client = esi_cluster_setup(partition="8GBXS", n_workers=n_workers)

x = [2, 4, 6, 8]
z = range(n_workers)
w = np.ones((8, 1))

pmap = ParallelMap(f, x, np.random.rand(n_workers), z=z, w=w, n_inputs=n_workers)
with pmap as p:
    p.compute()

pmap = ParallelMap(g, x, np.random.rand(n_workers), z=z, w=w, n_inputs=n_workers)
with pmap as p:
    p.compute()

Handling results

Load results from files

By default, results are saved to disk in HDF5 format and can be accessed using the results_container attribute of ParallelMap:

with ParallelMap(f, [2, 4, 6, 8], 4) as pmap:
  filenames = pmap.compute()

Example loading code:

import h5py
import numpy as np
out = np.zeros((4))

with h5py.File(pmap.results_container, "r") as h5f:
  for k, key in enumerate(h5f.keys()):
    out[k] = h5f[key]["result_0"]

Collect results in local memory

This is possible but not recommended.

with ParallelMap(f, [2, 4, 6, 8], 4, write_worker_results=False) as pmap:
  results = pmap.compute() # returns a list of outputs

Debugging

Use the debug keyword to perform all function calls in the local thread of the active Python interpreter

with ParallelMap(f, [2, 4, 6, 8], 4, z=None) as pmap:
    results = pmap.compute(debug=True)

This way tools like pdb or %debug IPython magics can be used.

Documentation and Contact

To report bugs or ask questions please use our GitHub issue tracker. More usage details and background information is available in our online documentation.