Skip to main content
Join the official Python Developers Survey 2018 and win valuable prizes: Start the survey!

A Python-based, distributed MapReduce solution.

Project description

**This project is still under active development, though largely finished. It is currently being tested in a production environment.**


Installing JobX


- *go*:

$ sudo apt-get install golang

- *etcd*:

$ git clone
$ cd etcd
$ ./build

$ sudo mkdir /var/lib/etcd
$ sudo bin/etcd -addr= -peer-addr= -data-dir=/var/lib/etcd -name=etcd1

- *nsq*:

$ sudo apt-get install gpm
$ mkdir ~/.go
$ GOPATH=~/.go go get
$ sudo mkdir /var/lib/nsq
$ cd /var/lib/nsq
$ sudo ~/.go/bin/nsqlookupd
$ sudo ~/.go/bin/nsqd --lookupd-tcp-address=

- Install Nginx.


1. Configure Nginx:

upstream mapreduce {
server unix:/tmp/mr.gunicorn.sock fail_timeout=0;

server {
listen 80;

server_name job1.domain;
keepalive_timeout 5;

access_log /tmp/nginx-mr-access.log;
error_log /tmp/nginx-mr-error.log;

location /s {
root /usr/local/lib/python2.7/dist-packages/mr/resources/static;
try_files $uri $uri/ =404;

location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;

proxy_pass http://mapreduce;

2. Create workflow:

MR_ETCD_HOST=job1.domain MR_WORKFLOW_NAMES=build mr_kv_workflow_create build "Jobs that assist build and deployment."

3. Load handlers:

..write and load handlers

4. Load steps:

..create step(s)

5. Load jobs:

..create job

6. Start:

MR_ETCD_HOST=job1.domain MR_ETCD_PORT=4001 MR_WORKFLOW_NAMES=build mr_start_gunicorn_dev

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
mapreduce-0.2.3-py2-none-any.whl (114.6 kB) Copy SHA256 hash SHA256 Wheel 2.7 Sep 25, 2014
mapreduce-0.2.3.tar.gz (57.2 kB) Copy SHA256 hash SHA256 Source None Sep 25, 2014

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page