Skip to main content

Thin MapReduce-like layer that wraps the Python multiprocessing library.

Project description

Thin MapReduce-like layer that wraps the Python multiprocessing library.

PyPI version and link. travis coveralls

Package Installation and Usage

The package is available on PyPI:

python -m pip install mr4mp

The library can be imported in the usual way:

import mr4mp

Examples

Word-Document Index

Suppose we have some functions that we can use to build an index of randomly generated words:

def word(): # Generate a random 7-letter "word".
    return ''.join(choice(ascii_lowercase) for _ in range(7))

def index(id): # Build an index mapping some random words to an identifier.
    return {w:{id} for w in {word() for _ in range(100)}}

def merge(i, j): # Merge two index dictionaries i and j.
    return {k:(i.get(k,set()) | j.get(k,set())) for k in i.keys() | j.keys()}

We can then construct an index in the following way:

from random import choice
from string import ascii_lowercase
from timeit import default_timer
start = default_timer()
pool = mr4mp.pool()
pool.mapreduce(index, merge, range(100))
print("Finished in " + str(default_timer()-start) + "s using " + str(len(pool)) + " process(es).")

The above might yield the following output:

Finished in 0.664681524217187s using 2 process(es).

Suppose we had instead explicitly specified that only one process can be used:

pool = mr4mp.pool(1)

After the above modification, we might see the following output from the code block:

Finished in 2.23329004518571s using 1 process(es).

Testing and Conventions

All unit tests are executed and their coverage is measured when using nose (see setup.cfg for configution details):

nosetests

Style conventions are enforced using Pylint:

pylint mr4mp

Contributions

In order to contribute to the source code, open an issue or submit a pull request on the GitHub page for this library.

Versioning

Beginning with version 0.1.0, the version number format for this library and the changes to the library associated with version number increments conform with Semantic Versioning 2.0.0.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mr4mp-0.1.0.tar.gz (4.5 kB view details)

Uploaded Source

Built Distribution

mr4mp-0.1.0-py3-none-any.whl (4.7 kB view details)

Uploaded Python 3

File details

Details for the file mr4mp-0.1.0.tar.gz.

File metadata

  • Download URL: mr4mp-0.1.0.tar.gz
  • Upload date:
  • Size: 4.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.1 requests-toolbelt/0.9.1 tqdm/4.39.0 CPython/3.8.0

File hashes

Hashes for mr4mp-0.1.0.tar.gz
Algorithm Hash digest
SHA256 f236e03f8642ee653fc2f77173d9d746d2541be63dc0f86e014c906e3e3746dd
MD5 9af5e3b5b44b545642a0d009953ed088
BLAKE2b-256 8930578821095c5b966d9758af9ae7bc4d3a04a9392f33b0babd422ad1d485fd

See more details on using hashes here.

Provenance

File details

Details for the file mr4mp-0.1.0-py3-none-any.whl.

File metadata

  • Download URL: mr4mp-0.1.0-py3-none-any.whl
  • Upload date:
  • Size: 4.7 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.1 requests-toolbelt/0.9.1 tqdm/4.39.0 CPython/3.8.0

File hashes

Hashes for mr4mp-0.1.0-py3-none-any.whl
Algorithm Hash digest
SHA256 9814ea715ca6baa1702f24575ed46ad1ad52dec7613e28f422c0b2ba2d7e6c85
MD5 dc76ac88b2fe68cd32d2b6d8bb416d4d
BLAKE2b-256 acb00f07fb39e5f0b06fec231a7b1a9b0f1f38a1787d8c8fe43fedb97c8ba619

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page