A Distributed List API over MPI
Project description
This package implements the DFM class.
The DFM is a useful abstraction for working with lists distributed over a set of MPI ranks. The acronym stands for distributed free monoid, which is just a fancy way to say it’s a list.
If you’re familiar with spark, it’s like an RDD, but only holds a list.
Quick Start
from mpi_list import Context, DFM
C = Context() # calls MPI_Init via mpi4py
# After each of the three lines below:
# 1. each rank now has 1000//C.procs consecutive numbers
# 2. each rank now has a list of strings
# 3. only numbers containing a '2' remain
dfm = C . iterates(1000) \
. map(lambda i: f"String {i}") \
. filter(lambda s: '2' in s)
if C.rank == 0:
# Caution! Uncommenting this will deadlock your program.
# Collective calls must be called by all ranks!
#print( dfm . head(10) )
pass
# This is OK, since all ranks now have 'ans'
ans = dfm.head(10)
if C.rank == 0:
print( ans )
ans = dfm . filter(lambda s: len(s) <= len("String nn")) \
. collect()
if ans is not None: # only rank 0 gets "collect"
print( ans )
Launch your program with mpirun python my_prog.py.
If you’re using a supercomputer, consider installing spindle, and then use spindle mpirun python my_prog.py.
Note
This project has been set up using PyScaffold 4.0.1. For details and usage information on PyScaffold see https://pyscaffold.org/.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
mpi_list-0.3.tar.gz
(23.6 kB
view hashes)
Built Distribution
mpi_list-0.3-py2.py3-none-any.whl
(14.3 kB
view hashes)
Close
Hashes for mpi_list-0.3-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 29b3a11c7d7022ebdc02c05a34da39f061acdc44fa8cf69d54564451a1bf862f |
|
MD5 | 67190863be4dde9fd634df8520f833cb |
|
BLAKE2b-256 | 079bee2d99393d8586152410e0e1c7b58d32523fcabefd87828a9067f6c417a8 |