mr4mp 2.5.0

Thin MapReduce-like layer that wraps the Python multiprocessing library.

Thin MapReduce-like layer that wraps the Python multiprocessing library.

Purpose

This package provides a streamlined interface for the built-in Python multiprocessing library. The interface makes it possible to parallelize in a succinct way (sometimes using only one line of code) a data workflow that can be expressed in a MapReduce-like form. More background information about this package’s design and implementation, as well a detailed use case, can be found in a related article.

Installation and Usage

This library is available as a package on PyPI:

python -m pip install mr4mp

The library can be imported in the usual way:

import mr4mp

Word-Document Index Example

Suppose we have some functions that we can use to build an index of randomly generated words:

from random import choice
from string import ascii_lowercase

def word(): # Generate a random 7-letter "word".
    return ''.join(choice(ascii_lowercase) for _ in range(7))

def index(identifier): # Build an index mapping some random words to an identifier.
    return {w:{identifier} for w in {word() for _ in range(100)}}

def merge(i, j): # Merge two index dictionaries i and j.
    return {k:(i.get(k,set()) | j.get(k,set())) for k in i.keys() | j.keys()}

We can then construct an index in the following way:

from timeit import default_timer

start = default_timer()
pool = mr4mp.pool()
pool.mapreduce(index, merge, range(100))
pool.close()
print("Finished in " + str(default_timer()-start) + "s using " + str(len(pool)) + " process(es).")

The above might yield the following output:

Finished in 0.664681524217187s using 2 process(es).

Suppose that we instead explicitly specify that only one process can be used:

pool = mr4mp.pool(1)

After the above modification, we might see the following output from the code block:

Finished in 2.23329004518571s using 1 process(es).

Development

All installation and development dependencies are managed using setuptools and are fully specified in setup.py. The extras_require parameter is used to specify optional requirements for various development tasks. This makes it possible to specify additional options (such as docs, lint, and so on) when performing installation using pip:

python -m pip install .[docs,lint]

Documentation

The documentation can be generated automatically from the source files using Sphinx:

python -m pip install .[docs]
cd docs
sphinx-apidoc -f -E --templatedir=_templates -o _source .. ../setup.py && make html

Testing and Conventions

All unit tests are executed and their coverage is measured when using pytest (see setup.cfg for configuration details):

python -m pip install .[test]
python -m pytest

Some unit tests are included in the module itself and can be executed using doctest:

python mr4mp/mr4mp.py -v

Style conventions are enforced using Pylint:

python -m pip install .[lint]
python -m pylint mr4mp ./test/test_mr4mp.py

Contributions

In order to contribute to the source code, open an issue or submit a pull request on the GitHub page for this library.

Versioning

Beginning with version 0.1.0, the version number format for this library and the changes to the library associated with version number increments conform with Semantic Versioning 2.0.0.

Publishing

This library can be published as a package on PyPI by a package maintainer. First, install the dependencies required for packaging and publishing:

python -m pip install .[publish]

Remove any old build/distribution files. Then, package the source into a distribution archive using the wheel package:

rm -rf dist *.egg-info
python setup.py sdist bdist_wheel

Finally, upload the package distribution archive to PyPI using the twine package:

python -m twine upload dist/*