A Python-based, distributed MapReduce solution.
Project description
**This project is still under active development, though largely finished. It is currently being tested in a production environment.**
==========
Vocabulary
==========
===============
Installing JobX
===============
------------
Dependencies
------------
- *go*:
$ sudo apt-get install golang
- *etcd*:
$ git clone git@github.com:coreos/etcd.git
$ cd etcd
$ ./build
$ sudo mkdir /var/lib/etcd
$ sudo bin/etcd -addr=127.0.0.1:4001 -peer-addr=127.0.0.1:7001 -data-dir=/var/lib/etcd -name=etcd1
- *nsq*:
$ sudo apt-get install gpm
$ mkdir ~/.go
$ GOPATH=~/.go go get github.com/bitly/nsq/...
$ sudo mkdir /var/lib/nsq
$ cd /var/lib/nsq
$ sudo ~/.go/bin/nsqlookupd
$ sudo ~/.go/bin/nsqd --lookupd-tcp-address=127.0.0.1:4160
- Install Nginx.
-------------
Configuration
-------------
1. Configure Nginx:
upstream mapreduce {
server unix:/tmp/mr.gunicorn.sock fail_timeout=0;
}
server {
listen 80;
server_name job1.domain;
keepalive_timeout 5;
access_log /tmp/nginx-mr-access.log;
error_log /tmp/nginx-mr-error.log;
location /s {
root /usr/local/lib/python2.7/dist-packages/mr/resources/static;
try_files $uri $uri/ =404;
}
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://mapreduce;
}
}
2. Create workflow:
MR_ETCD_HOST=job1.domain MR_WORKFLOW_NAMES=build mr_kv_workflow_create build "Jobs that assist build and deployment."
3. Load handlers:
..write and load handlers
4. Load steps:
..create step(s)
5. Load jobs:
..create job
6. Start:
MR_ETCD_HOST=job1.domain MR_ETCD_PORT=4001 MR_WORKFLOW_NAMES=build mr_start_gunicorn_dev
==========
Vocabulary
==========
===============
Installing JobX
===============
------------
Dependencies
------------
- *go*:
$ sudo apt-get install golang
- *etcd*:
$ git clone git@github.com:coreos/etcd.git
$ cd etcd
$ ./build
$ sudo mkdir /var/lib/etcd
$ sudo bin/etcd -addr=127.0.0.1:4001 -peer-addr=127.0.0.1:7001 -data-dir=/var/lib/etcd -name=etcd1
- *nsq*:
$ sudo apt-get install gpm
$ mkdir ~/.go
$ GOPATH=~/.go go get github.com/bitly/nsq/...
$ sudo mkdir /var/lib/nsq
$ cd /var/lib/nsq
$ sudo ~/.go/bin/nsqlookupd
$ sudo ~/.go/bin/nsqd --lookupd-tcp-address=127.0.0.1:4160
- Install Nginx.
-------------
Configuration
-------------
1. Configure Nginx:
upstream mapreduce {
server unix:/tmp/mr.gunicorn.sock fail_timeout=0;
}
server {
listen 80;
server_name job1.domain;
keepalive_timeout 5;
access_log /tmp/nginx-mr-access.log;
error_log /tmp/nginx-mr-error.log;
location /s {
root /usr/local/lib/python2.7/dist-packages/mr/resources/static;
try_files $uri $uri/ =404;
}
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://mapreduce;
}
}
2. Create workflow:
MR_ETCD_HOST=job1.domain MR_WORKFLOW_NAMES=build mr_kv_workflow_create build "Jobs that assist build and deployment."
3. Load handlers:
..write and load handlers
4. Load steps:
..create step(s)
5. Load jobs:
..create job
6. Start:
MR_ETCD_HOST=job1.domain MR_ETCD_PORT=4001 MR_WORKFLOW_NAMES=build mr_start_gunicorn_dev
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
mapreduce-0.2.3.tar.gz
(57.2 kB
view hashes)
Built Distribution
mapreduce-0.2.3-py2-none-any.whl
(114.6 kB
view hashes)
Close
Hashes for mapreduce-0.2.3-py2-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 560a681d8c98b7ed0c7524dcc4c1498ef76067f37036a0dd22685ff4d5f5ce37 |
|
MD5 | 158452a140b618c8cfbf4dde53fc4d4f |
|
BLAKE2b-256 | 4c3e648f5f4b41901de3f868c361981447f0228e3a6397e2ea42fc60b331b700 |