Skip to main content

A Distributed List API over MPI

Project description

This package implements the DFM class.

The DFM is a useful abstraction for working with lists distributed over a set of MPI ranks. The acronym stands for distributed free monoid, which is just a fancy way to say it’s a list.

If you’re familiar with spark, it’s like an RDD, but only holds a list.

Quick Start

from mpi_list import Context, DFM

C = Context() # calls MPI_Init via mpi4py

# After each of the three lines below:
#  1. each rank now has 1000//C.procs consecutive numbers
#  2. each rank now has a list of strings
#  3. only numbers containing a '2' remain
dfm = C . iterates(1000) \
        . map(lambda i: f"String {i}") \
        . filter(lambda s: '2' in s)

if C.rank == 0:
    # Caution! Uncommenting this will deadlock your program.
    # Collective calls must be called by all ranks!
    #print( dfm . head(10) )

# This is OK, since all ranks now have 'ans'
ans = dfm.head(10)
if C.rank == 0:
    print( ans )

ans = dfm . filter(lambda s: len(s) <= len("String nn")) \
          . collect()
if ans is not None: # only rank 0 gets "collect"
    print( ans )

Launch your program with mpirun python

If you’re using a supercomputer, consider installing spindle, and then use spindle mpirun python


This project has been set up using PyScaffold 4.0.1. For details and usage information on PyScaffold see

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mpi_list-0.3.tar.gz (23.6 kB view hashes)

Uploaded Source

Built Distribution

mpi_list-0.3-py2.py3-none-any.whl (14.3 kB view hashes)

Uploaded Python 2 Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page