The reduce method

prepare(self, count, *args, **kwargs)

The count is the number of processes the job will be run across, and *args and **kwargs is whatever is passed into your child class’s __init__ method. The prepare method returns count rows containing a tuple ((), {}) of the arguments that will be passed to each map process.

The map method

map(self, *args, **kwargs)

The *args and **kwargs are whatever was returned from prepare. The map method returns whatever you want reduce to use to merge all the data together.

The reduce method

reduce(self, output, value)

The output is the global aggregation of all the value arguments the reduce method has seen. Basically, whatever you return from one reduce call will be passed back into the next reduce call as output. The value argument is whatever the recently finished map call returned.

Bringing it all together

So let’s bring it all together in our MrHelloWorld job, first let’s get the skeleton in place:

from mister import BaseMister


class MrHelloWorld(BaseMister):
    def prepare(self, count, *args, **kwargs): pass
    def map(self, *args, **kwargs): pass
    def reduce(self, output, value): pass

Now let’s flesh out the prepare method:

def prepare(self, count, name):
    # we're just going to return the number and the name we pass in
    for x in range(count):
        yield ([x, name], {})

And our map method:

def map(self, x, name):
    return "Process {} says 'hello {}'".format(x, name)

Finally, our reduce method:

def reduce(self, output, value):
    if output is None:
        output = []
    output.append(value)
    return output

Running our job:

mr = MrHelloWorld("Alice")
output = mr.run()
print(output)

will result in:

[
    "Process 1 says 'hello Alice'",
    "Process 0 says 'hello Alice'",
    "Process 2 says 'hello Alice'",
    "Process 3 says 'hello Alice'",
    "Process 4 says 'hello Alice'",
    "Process 5 says 'hello Alice'",
    "Process 6 says 'hello Alice'",
    "Process 7 says 'hello Alice'",
    "Process 8 says 'hello Alice'",
    "Process 9 says 'hello Alice'",
    "Process 10 says 'hello Alice'"
]

Congrats, you just ran a map/reduce job, you are now an AI and a ML engineer, remember me when you’re famous!