poopy 0.2.5

Pure Python implementation of Map-Reduce over AMQP

Poopy - An unusable map-reduce engine over AMQP

Install

From PYPI

1. Install rabitMQ

2. pip install poopy (please use virtualenv)

3. Download this file https://bitbucket.org/leliel12/poopy/raw/tip/example/iris.arff?at=default

4. Download this file https://bitbucket.org/leliel12/poopy/raw/tip/example/randomforest.py and put in the same directory of iris.arff

From REPO

1. Install rabitMQ

2. Clone this https://bitbucket.org/leliel12/poopy repo

3. pip install -e .

4. The archives iris.arff and randomforest.py are inside in examples/

Running

1. Open two consoles (consoleA, consoleB)

2. In consoleB run poopy deploy amqp://localhost

3. In consoleA execute poopy upload amqp://localhost path/to/iris.arff poopFS://iris.arff now your file are uploaded to the “distributed file sistem”

4. In consoleA run poopy run amqp://localhost path/to/randomforest.py Script out

5. Your output model are serialized in out/localtime

Reading the model

In python console

from poopy import serializers

with open("out/file") as fp:
    model = serializers.load(fp)

TODO:

• Implement correctly timeouts.

• Implement an error exchange.

• Real distributed file system.

• More than one map or reduce.

• More output formats than b64-pkl.

• Not use the central node as main memory.

• Reorder some modules into packages