Skip to main content

Thin MapReduce-like layer that wraps the Python multiprocessing library.

Project description

Thin MapReduce-like layer that wraps the Python multiprocessing library.

PyPI version and link. Read the Docs documentation status. Travis CI build status. Coveralls test coverage summary.

Purpose

This package provides a streamlined interface for the built-in Python multiprocessing library. The interface makes it possible to parallelize in a succinct way (sometimes using only one line of code) a data workflow that can be expressed in a MapReduce-like form. More background information about this package’s design and implementation, as well a detailed use case, can be found in a related article.

Package Installation and Usage

The package is available on PyPI:

python -m pip install mr4mp

The library can be imported in the usual way:

import mr4mp

Word-Document Index Example

Suppose we have some functions that we can use to build an index of randomly generated words:

from random import choice
from string import ascii_lowercase

def word(): # Generate a random 7-letter "word".
    return ''.join(choice(ascii_lowercase) for _ in range(7))

def index(id): # Build an index mapping some random words to an identifier.
    return {w:{id} for w in {word() for _ in range(100)}}

def merge(i, j): # Merge two index dictionaries i and j.
    return {k:(i.get(k,set()) | j.get(k,set())) for k in i.keys() | j.keys()}

We can then construct an index in the following way:

from timeit import default_timer

start = default_timer()
pool = mr4mp.pool()
pool.mapreduce(index, merge, range(100))
pool.close()
print("Finished in " + str(default_timer()-start) + "s using " + str(len(pool)) + " process(es).")

The above might yield the following output:

Finished in 0.664681524217187s using 2 process(es).

Suppose that we instead explicitly specify that only one process can be used:

pool = mr4mp.pool(1)

After the above modification, we might see the following output from the code block:

Finished in 2.23329004518571s using 1 process(es).

Documentation

The documentation can be generated automatically from the source files using Sphinx:

python -m pip install sphinx sphinx-rtd-theme
cd docs
sphinx-apidoc -f -E --templatedir=_templates -o _source .. ../setup.py && make html

Testing and Conventions

All unit tests are executed and their coverage is measured when using nose (see setup.cfg for configuration details):

python -m pip install nose coverage
nosetests

Some unit tests are included in the module itself and can be executed using doctest:

python mr4mp/mr4mp.py -v

Style conventions are enforced using Pylint:

python -m pip install pylint
pylint mr4mp

Contributions

In order to contribute to the source code, open an issue or submit a pull request on the GitHub page for this library.

Versioning

Beginning with version 0.1.0, the version number format for this library and the changes to the library associated with version number increments conform with Semantic Versioning 2.0.0.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mr4mp-2.1.2.tar.gz (7.3 kB view details)

Uploaded Source

Built Distribution

mr4mp-2.1.2-py3-none-any.whl (6.3 kB view details)

Uploaded Python 3

File details

Details for the file mr4mp-2.1.2.tar.gz.

File metadata

  • Download URL: mr4mp-2.1.2.tar.gz
  • Upload date:
  • Size: 7.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.25.1 setuptools/42.0.1 requests-toolbelt/0.9.1 tqdm/4.61.2 CPython/3.8.0

File hashes

Hashes for mr4mp-2.1.2.tar.gz
Algorithm Hash digest
SHA256 6308158bbdcb2d34ffa0aa2ea1d44ee0060f9414b5a242fb98419944c68537d4
MD5 9e464d185d085b74878758a0d07bda1c
BLAKE2b-256 04286b9028eb15306c7fd7231e6eeb036b9d0e867319a82e8d936b6670cdd4a7

See more details on using hashes here.

Provenance

File details

Details for the file mr4mp-2.1.2-py3-none-any.whl.

File metadata

  • Download URL: mr4mp-2.1.2-py3-none-any.whl
  • Upload date:
  • Size: 6.3 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.25.1 setuptools/42.0.1 requests-toolbelt/0.9.1 tqdm/4.61.2 CPython/3.8.0

File hashes

Hashes for mr4mp-2.1.2-py3-none-any.whl
Algorithm Hash digest
SHA256 72769b61811927f7739ce791242a029f81e3dd7b2b906b6335874b343b52d25b
MD5 fc1bba4cdc2b8c769c4080f36621adda
BLAKE2b-256 a90927641fdbc3080996652dde4dd5f0de5466adb221092c12e6553a3f14ffa3

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page