Skip to main content

MPI Pool similar to multiprocessing

Project description

About

mpipool offers a Pool class similar multiprocessing.Pool from the standard library.

mpipool uses MPIPool implementation of schwimmbad library and circumvents some of its limitations:

  • A series of mpipool.Pool.map calls do not crash,
  • In case a worker raises an exception, the MPI tasks are shut down properly so that the full program halts and does not hang.

Example usage

from mpipool import Pool

def add(a, b):
    return a + b

p = Pool()

sums = p.map(add, [(ai, bi) for ai in range(10) for bi in range(10)])

assert len(sums) == 100
assert sums[0] == 0
assert sums[-1] == 18

sums = p.map(add, [(ai, bi) for ai in range(10) for bi in range(10)])

assert len(sums) == 100
assert sums[0] == 0
assert sums[-1] == 18

The program must be run on the commandline like:

$ mpirun -n 4 python example.py

Currently mpipool.Pool only implements a map method.

Contrary to the MPIPool implementation of schwimmbad the statements after from mpipool import Pool are only executed by the task with rank 0.

Credits

mpipool uses of the MPIPool implementation of schwimmbad library.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for mpipool, version 0.2.1
Filename, size File type Python version Upload date Hashes
Filename, size mpipool-0.2.1.tar.gz (4.9 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page