Madoop: Michigan Hadoop
Michigan Hadoop (madoop) is a light weight MapReduce framework for education. Madoop implements the Hadoop Streaming interface. Madoop is implemented in Python and runs on a single machine.
For an in-depth explanation of how to write MapReduce programs in Python for Hadoop Streaming, see our Hadoop Streaming tutorial.
Quick start
Install Madoop.
$ pip install madoop
Create example MapReduce program with input files.
$ madoop --example
$ tree example
example
├── input
│ ├── input01.txt
│ └── input02.txt
├── map.py
└── reduce.py
Run example word count MapReduce program.
$ madoop \
-input example/input \
-output example/output \
-mapper example/map.py \
-reducer example/reduce.py
Concatenate and print the output.
$ cat example/output/part-*
Goodbye 1
Bye 1
Hadoop 2
World 2
Hello 2
Comparison with Apache Hadoop and CLI
Madoop implements a subset of the Hadoop Streaming interface. You can simulate the Hadoop Streaming interface at the command line with cat and sort.
Here's how to run our example MapReduce program on Apache Hadoop.
$ hadoop \
jar path/to/hadoop-streaming-X.Y.Z.jar
-input example/input \
-output output \
-mapper example/map.py \
-reducer example/reduce.py
$ cat output/part-*
Here's how to run our example MapReduce program at the command line using cat and sort.
$ cat input/* | ./map.py | sort | ./reduce.py
| Madoop | Hadoop | cat/sort |
|---|---|---|
| Implement some Hadoop options | All Hadoop options | No Hadoop options |
| Multiple mappers and reducers | Multiple mappers and reducers | One mapper, one reducer |
| Single machine | Many machines | Single Machine |
jar hadoop-streaming-X.Y.Z.jar argument ignored |
jar hadoop-streaming-X.Y.Z.jar argument required |
No arguments |
| Lines within a group are sorted | Lines within a group are sorted | Lines within a group are sorted |
Contributing
Contributions from the community are welcome! Check out the guide for contributing.
Acknowledgments
Michigan Hadoop is written by Andrew DeOrio awdeorio@umich.edu.
Metadata
Release files for madoop 1.3.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| madoop-1.3.2.tar.gz | 21.5 kB | Without provenance | Details for madoop-1.3.2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| madoop-1.3.2-py3-none-any.whl | Python 3 | none | any | Details for madoop-1.3.2-py3-none-any.whl |
Total release size: 33.2 kB
Release files / madoop-1.3.2.tar.gz
| Download URL | madoop-1.3.2.tar.gz |
|---|---|
| Size | 21.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
5299d1d6c6299cf150ea88fc38df9142dc39a2a0f64a2c4fa2cb713e7d07d2b4
|
|
BLAKE2b-256 checksum How to use checksums |
f07d7d9cdc3ea0be4bd09ec0e4e4a795266b3b4246545ce16d40b61f28be1dc9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.12
|
Release files / madoop-1.3.2-py3-none-any.whl
| Download URL | madoop-1.3.2-py3-none-any.whl |
|---|---|
| Size | 11.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e5a6f950debed50cc27b34074b717b31c69681f986931690a5594f5848014052
|
|
BLAKE2b-256 checksum How to use checksums |
56d0041b34b603a89449326a25a17c623e157075f0d21d26e2974d541cedfafc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.12
|
Security for madoop 1.3.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.