Skip to main content

Approachable map/reduce jobs

Project description

For all your medium data needs!

Mister attempts to make running a map/reduce job approachable.

When you’ve got data that isn’t really big and so you’re not quite ready to distribute the data across a gazillian machines and stuff but would still like an answer in a reasonable amount of time.

5 minute getting started

Mister needs you to define three methods: prepare (get the data ready to be run across multiple processes), map (actually do something with the chunks of data from prepare), and reduce (mash all the values returned from map together).

The reduce method

prepare(self, count, *args, **kwargs)

The count is the number of processes the job will be run across, and *args and **kwargs is whatever is passed into your child class’s __init__ method. The prepare method returns count rows containing a tuple ((), {}) of the arguments that will be passed to each map process.

The map method

map(self, *args, **kwargs)

The *args and **kwargs are whatever was returned from prepare. The map method returns whatever you want reduce to use to merge all the data together.

The reduce method

reduce(self, output, value)

The output is the global aggregation of all the value arguments the reduce method has seen. Basically, whatever you return from one reduce call will be passed back into the next reduce call as output. The value argument is whatever the recently finished map call returned.

Bringing it all together

So let’s bring it all together in our MrHelloWorld job, first let’s get the skeleton in place:

from mister import BaseMister


class MrHelloWorld(BaseMister):
    def prepare(self, count, *args, **kwargs): pass
    def map(self, *args, **kwargs): pass
    def reduce(self, output, value): pass

Now let’s flesh out the prepare method:

def prepare(self, count, name):
    # we're just going to return the number and the name we pass in
    for x in range(count):
        yield ([x, name], {})

And our map method:

def map(self, x, name):
    return "Process {} says 'hello {}'".format(x, name)

Finally, our reduce method:

def reduce(self, output, value):
    if output is None:
        output = []
    output.append(value)
    return output

Running our job:

mr = MrHelloWorld("Alice")
output = mr.run()
print(output)

will result in:

[
    "Process 1 says 'hello Alice'",
    "Process 0 says 'hello Alice'",
    "Process 2 says 'hello Alice'",
    "Process 3 says 'hello Alice'",
    "Process 4 says 'hello Alice'",
    "Process 5 says 'hello Alice'",
    "Process 6 says 'hello Alice'",
    "Process 7 says 'hello Alice'",
    "Process 8 says 'hello Alice'",
    "Process 9 says 'hello Alice'",
    "Process 10 says 'hello Alice'"
]

Congrats, you just ran a map/reduce job, you are now an AI and a ML engineer, remember me when you’re famous!

Another Example

I think word counting is the traditional map/reduce example? So here it is:

import os
import re
improt math
from collections import Counter

from mister import BaseMister


class MrWordCount(BaseMister):
    def prepare(self, count, path):
        """prepare segments the data for the map() method"""
        size = os.path.getsize(path)
        length = int(math.ceil(size / count))
        start = 0
        for x in range(count):
            kwargs = {}
            kwargs["path"] = path
            kwargs["start"] = start
            kwargs["length"] = length
            start += length
            yield (), kwargs

    def map(self, path, start, length):
        """all the magic happens right here"""
        output = Counter()
        with open(path) as fp:
            fp.seek(start, 0)
            words = fp.read(length)

        # I don't compensate for word boundaries because example
        for word in re.split(r"\s+", words):
            output[word] += 1
        return output

    def reduce(self, output, count):
        """take all the return values from map() and aggregate them to the final value"""
        if not output:
            output = Counter()
        output.update(count)
        return output

# let's count the bible
path = "./testdata/bible-kjv.txt"
mr = MrWordCount(path)
wordcounts = mr.run()
print(wordcounts.most_common(10))

On my computer, the asynchronous code above runs about 3x faster than its syncronous equivalent below:

import re
from collections import Counter

path = "./testdata/bible-kjv.txt"

output = Counter()
with open(path) as fp:
    words = fp.read()

for word in re.split(r"\s+", words):
    output[word] += 1

print(wordcounts.most_common(10))

Installation

To install, use Pip:

$ pip install mister

Or, to grab the latest and greatest:

$ pip install --upgrade git+https://github.com/Jaymon/mister#egg=mister

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mister-0.0.2.tar.gz (6.5 kB view details)

Uploaded Source

File details

Details for the file mister-0.0.2.tar.gz.

File metadata

  • Download URL: mister-0.0.2.tar.gz
  • Upload date:
  • Size: 6.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: Python-urllib/2.7

File hashes

Hashes for mister-0.0.2.tar.gz
Algorithm Hash digest
SHA256 5b837b952920189e6d1e189f5a4a09d6d24596623a13c59519c142fe761f5afd
MD5 fa3344ff1b357e75b20c1945162589fe
BLAKE2b-256 3873887e8787648bb26862e4c042472cb9046af0155e235fdf5c32a14438d61b

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page