Pure Python implementation of Map-Reduce over AMQP
Project description
Poopy - An unusable map-reduce engine over AMQP
Install
From PYPI
Install rabitMQ
pip install poopy (please use virtualenv)
Download this file https://bitbucket.org/leliel12/poopy/raw/tip/example/iris.arff?at=default
Download this file https://bitbucket.org/leliel12/poopy/raw/tip/example/randomforest.py and put in the same directory of iris.arff
From REPO
Install rabitMQ
Clone this https://bitbucket.org/leliel12/poopy repo
pip install -e .
The archives iris.arff and randomforest.py are inside in examples/
Running
Open two consoles (consoleA, consoleB)
In consoleB run poopy deploy amqp://localhost
In consoleA execute poopy upload amqp://localhost path/to/iris.arff poopFS://iris.arff now your file are uploaded to the “distributed file sistem”
In consoleA run poopy run amqp://localhost path/to/randomforest.py Script out
Your output model are serialized in out/localtime
Reading the model
In python console
from poopy import serializers
with open("out/file") as fp:
model = serializers.load(fp)
TODO:
Implement correctly timeouts.
Implement an error exchange.
Real distributed file system.
More than one map or reduce.
More output formats than b64-pkl.
Not use the central node as main memory.
Reorder some modules into packages
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.