Approachable map/reduce jobs
Project description
For all your medium data needs!
Mister attempts to make running a map/reduce job approachable.
When you’ve got data that isn’t really big and so you’re not quite ready to distribute the data across a gazillian machines and stuff but would still like an answer in a reasonable amount of time.
5 minute getting started
Mister needs you to define three methods: prepare (get the data ready to be run across multiple processes), map (actually do something with the chunks of data from prepare), and reduce (mash all the values returned from map together).
The reduce method
prepare(self, count, *args, **kwargs)
The count is the number of processes the job will be run across, and *args and **kwargs is whatever is passed into your child class’s __init__ method. The prepare method returns count rows containing a tuple ((), {}) of the arguments that will be passed to each map process.
The map method
map(self, *args, **kwargs)
The *args and **kwargs are whatever was returned from prepare. The map method returns whatever you want reduce to use to merge all the data together.
The reduce method
reduce(self, output, value)
The output is the global aggregation of all the value arguments the reduce method has seen. Basically, whatever you return from one reduce call will be passed back into the next reduce call as output. The value argument is whatever the recently finished map call returned.
Bringing it all together
So let’s bring it all together in our MrHelloWorld job, first let’s get the skeleton in place:
from mister import BaseMister
class MrHelloWorld(BaseMister):
def prepare(self, count, *args, **kwargs): pass
def map(self, *args, **kwargs): pass
def reduce(self, output, value): pass
Now let’s flesh out the prepare method:
def prepare(self, count, name):
# we're just going to return the number and the name we pass in
for x in range(count):
yield ([x, name], {})
And our map method:
def map(self, x, name):
return "Process {} says 'hello {}'".format(x, name)
Finally, our reduce method:
def reduce(self, output, value):
if output is None:
output = []
output.append(value)
return output
Running our job:
mr = MrHelloWorld("Alice")
output = mr.run()
print(output)
will result in:
[ "Process 1 says 'hello Alice'", "Process 0 says 'hello Alice'", "Process 2 says 'hello Alice'", "Process 3 says 'hello Alice'", "Process 4 says 'hello Alice'", "Process 5 says 'hello Alice'", "Process 6 says 'hello Alice'", "Process 7 says 'hello Alice'", "Process 8 says 'hello Alice'", "Process 9 says 'hello Alice'", "Process 10 says 'hello Alice'" ]
Congrats, you just ran a map/reduce job, you are now an AI and a ML engineer, remember me when you’re famous!
Another Example
I think word counting is the traditional map/reduce example? So here it is:
import os
import re
improt math
from collections import Counter
from mister import BaseMister
class MrWordCount(BaseMister):
def prepare(self, count, path):
"""prepare segments the data for the map() method"""
size = os.path.getsize(path)
length = int(math.ceil(size / count))
start = 0
for x in range(count):
kwargs = {}
kwargs["path"] = path
kwargs["start"] = start
kwargs["length"] = length
start += length
yield (), kwargs
def map(self, path, start, length):
"""all the magic happens right here"""
output = Counter()
with open(path) as fp:
fp.seek(start, 0)
words = fp.read(length)
# I don't compensate for word boundaries because example
for word in re.split(r"\s+", words):
output[word] += 1
return output
def reduce(self, output, count):
"""take all the return values from map() and aggregate them to the final value"""
if not output:
output = Counter()
output.update(count)
return output
# let's count the bible
path = "./testdata/bible-kjv.txt"
mr = MrWordCount(path)
wordcounts = mr.run()
print(wordcounts.most_common(10))
On my computer, the asynchronous code above runs about 3x faster than its syncronous equivalent below:
import re
from collections import Counter
path = "./testdata/bible-kjv.txt"
output = Counter()
with open(path) as fp:
words = fp.read()
for word in re.split(r"\s+", words):
output[word] += 1
print(wordcounts.most_common(10))
Installation
To install, use Pip:
$ pip install mister
Or, to grab the latest and greatest:
$ pip install --upgrade git+https://github.com/Jaymon/mister#egg=mister
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
File details
Details for the file mister-0.0.2.tar.gz
.
File metadata
- Download URL: mister-0.0.2.tar.gz
- Upload date:
- Size: 6.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: Python-urllib/2.7
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5b837b952920189e6d1e189f5a4a09d6d24596623a13c59519c142fe761f5afd |
|
MD5 | fa3344ff1b357e75b20c1945162589fe |
|
BLAKE2b-256 | 3873887e8787648bb26862e4c042472cb9046af0155e235fdf5c32a14438d61b |