Skip to main content
This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (pypi.python.org).
Help us improve Python packaging - Donate today!

Python MapReduce framework

Project Description

mrjob is a Python 2.6+/3.3+ package that helps you write and run Hadoop Streaming jobs.

Stable version (v0.5.11) documentation

Development version documentation

mrjob fully supports Amazon’s Elastic MapReduce (EMR) service, which allows you to buy time on a Hadoop cluster on an hourly basis. mrjob has basic support for Google Cloud Dataproc (Dataproc) which allows you to buy time on a Hadoop cluster on a minute-by-minute basis. It also works with your own Hadoop cluster.

Some important features:

  • Run jobs on EMR, Google Cloud Dataproc, your own Hadoop cluster, or locally (for testing).
  • Write multi-step jobs (one map-reduce step feeds into the next)
  • Easily launch Spark jobs on EMR or your own Hadoop cluster
  • Duplicate your production environment inside Hadoop
    • Upload your source tree and put it in your job’s $PYTHONPATH
    • Run make and other setup scripts
    • Set environment variables (e.g. $TZ)
    • Easily install python packages from tarballs (EMR only)
    • Setup handled transparently by mrjob.conf config file
  • Automatically interpret error logs
  • SSH tunnel to hadoop job tracker (EMR only)
  • Minimal setup
    • To run on EMR, set $AWS_ACCESS_KEY_ID and $AWS_SECRET_ACCESS_KEY
    • To run on Dataproc, set up your Google account and credentials (see Dataproc Quickstart).
    • To run on your Hadoop cluster, just make sure $HADOOP_HOME is set.

Installation

From PyPI:

pip install mrjob

From source:

python setup.py install

A Simple Map Reduce Job

Code for this example and more live in mrjob/examples.

"""The classic MapReduce job: count the frequency of words.
"""
from mrjob.job import MRJob
import re

WORD_RE = re.compile(r"[\w']+")


class MRWordFreqCount(MRJob):

    def mapper(self, _, line):
        for word in WORD_RE.findall(line):
            yield (word.lower(), 1)

    def combiner(self, word, counts):
        yield (word, sum(counts))

    def reducer(self, word, counts):
        yield (word, sum(counts))


if __name__ == '__main__':
     MRWordFreqCount.run()

Try It Out!

# locally
python mrjob/examples/mr_word_freq_count.py README.rst > counts
# on EMR
python mrjob/examples/mr_word_freq_count.py README.rst -r emr > counts
# on Dataproc
python mrjob/examples/mr_word_freq_count.py README.rst -r dataproc > counts
# on your Hadoop cluster
python mrjob/examples/mr_word_freq_count.py README.rst -r hadoop > counts

Setting up EMR on Amazon

Setting up Dataproc on Google

Advanced Configuration

To run in other AWS regions, upload your source tree, run make, and use other advanced mrjob features, you’ll need to set up mrjob.conf. mrjob looks for its conf file in:

  • The contents of $MRJOB_CONF
  • ~/.mrjob.conf
  • /etc/mrjob.conf

See the mrjob.conf documentation for more information.

Release History

Release History

This version
History Node

0.5.11

History Node

0.5.10

History Node

0.5.9

History Node

0.5.8

History Node

0.5.7

History Node

0.5.6

History Node

0.5.5

History Node

0.5.4

History Node

0.5.3

History Node

0.5.2

History Node

0.5.1

History Node

0.5.0

History Node

0.4.6

History Node

0.4.5

History Node

0.4.4

History Node

0.4.3

History Node

0.4.2

History Node

0.4.1

History Node

0.4

History Node

0.4.0

History Node

0.4-RC1

History Node

0.4-dev

History Node

0.3.5

History Node

0.3.4.1

History Node

0.3.3.2

History Node

0.3.3.1

History Node

0.3.3

History Node

0.3.2

History Node

0.3.1

History Node

0.3.0

History Node

0.2.8

History Node

0.2.7

History Node

0.2.6

History Node

0.2.5

History Node

0.2.4

History Node

0.2.3

History Node

0.2.2

History Node

0.2.1

History Node

0.2.0

History Node

0.1.0

History Node

0.1.0-pre3

History Node

0.1.0-pre2

History Node

0.1.0-pre1

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
mrjob-0.5.11-py2.py3-none-any.whl (306.0 kB) Copy SHA256 Checksum SHA256 2.7 Wheel Aug 29, 2017
mrjob-0.5.11.tar.gz (440.3 kB) Copy SHA256 Checksum SHA256 Source Aug 29, 2017

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting