Thin MapReduce-like layer that wraps the Python multiprocessing library.
Project description
Thin MapReduce-like layer that wraps the Python multiprocessing library.
Package Installation and Usage
The package is available on PyPI:
python -m pip install mr4mp
The library can be imported in the usual way:
import mr4mp
Examples
Word-Document Index
Suppose we have some functions that we can use to build an index of randomly generated words:
def word(): # Generate a random 7-letter "word". return ''.join(choice(ascii_lowercase) for _ in range(7)) def index(id): # Build an index mapping some random words to an identifier. return {w:{id} for w in {word() for _ in range(100)}} def merge(i, j): # Merge two index dictionaries i and j. return {k:(i.get(k,set()) | j.get(k,set())) for k in i.keys() | j.keys()}
We can then construct an index in the following way:
from random import choice from string import ascii_lowercase from timeit import default_timer start = default_timer() pool = mr4mp.pool() pool.mapreduce(index, merge, range(100)) print("Finished in " + str(default_timer()-start) + "s using " + str(len(pool)) + " process(es).")
The above might yield the following output:
Finished in 0.664681524217187s using 2 process(es).
Suppose we had instead explicitly specified that only one process can be used:
pool = mr4mp.pool(1)
After the above modification, we might see the following output from the code block:
Finished in 2.23329004518571s using 1 process(es).
Testing and Conventions
All unit tests are executed and their coverage is measured when using nose (see setup.cfg for configution details):
nosetests
Style conventions are enforced using Pylint:
pylint mr4mp
Contributions
In order to contribute to the source code, open an issue or submit a pull request on the GitHub page for this library.
Versioning
Beginning with version 0.1.0, the version number format for this library and the changes to the library associated with version number increments conform with Semantic Versioning 2.0.0.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file mr4mp-0.1.0.tar.gz
.
File metadata
- Download URL: mr4mp-0.1.0.tar.gz
- Upload date:
- Size: 4.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.1 requests-toolbelt/0.9.1 tqdm/4.39.0 CPython/3.8.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | f236e03f8642ee653fc2f77173d9d746d2541be63dc0f86e014c906e3e3746dd |
|
MD5 | 9af5e3b5b44b545642a0d009953ed088 |
|
BLAKE2b-256 | 8930578821095c5b966d9758af9ae7bc4d3a04a9392f33b0babd422ad1d485fd |
Provenance
File details
Details for the file mr4mp-0.1.0-py3-none-any.whl
.
File metadata
- Download URL: mr4mp-0.1.0-py3-none-any.whl
- Upload date:
- Size: 4.7 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.1 requests-toolbelt/0.9.1 tqdm/4.39.0 CPython/3.8.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9814ea715ca6baa1702f24575ed46ad1ad52dec7613e28f422c0b2ba2d7e6c85 |
|
MD5 | dc76ac88b2fe68cd32d2b6d8bb416d4d |
|
BLAKE2b-256 | acb00f07fb39e5f0b06fec231a7b1a9b0f1f38a1787d8c8fe43fedb97c8ba619 |