Documentation is available at Read the Docs
HOD is a set of scripts to start services, for example a Hadoop cluster, from within another resource management system (i.e. Torque/PBS). As such, it allows traditional users of HPC systems to experiment with Hadoop or use it as a production setup if there is no dedicated setup available.
Hadoop is not the only software supported. HOD can also create HBase databases, IPython notebooks, and set up a Spark environment.
There are two main benefits:
Hadoop used to ship it’s own HOD (Hadoop On Demand) but it was not maintained and only supported Hadoop without tuning. The HOD code that was shipped with Hadoop 1.0.0 release was buggy to say the least. An attempt was made to make it work on the UGent HPC infrastructure, and although a working Hadoop cluster was realised, it was a nightmare to extend it’s functionality. At that point (April 2012), hanythingondemand was started to be better maintainable and support more tuning and functionality out of the box. For example, HBase was a minimum requirement. Hence, why Hadoop on Demand became ‘Hanything’. Apart from the acronym ‘HOD’ nothing of Hadoop On Demand was reused.
More on the history of Hadoop On Demand can be found in section 2 of this paper on Yarn (PDF)
hanythingondemand works by launching an MPI job which uses the reserved nodes as a cluster-in-a-cluster. These nodes then have the various Hadoop services started on them. Users can launch a job at startup (batch mode) or login to worker nodes (using the hod connect command) where they can interact with their services.
The rest of the requirements can be installed using EasyBuild:
# submits a job to start a Hadoop cluster on 16 nodes $ hod create --dist Hadoop-2.3.0-cdh5.0 -n16 --label my-cluster ### Connect to your new cluster. $ hod connect my-cluster ### Then, in your session, you can run your hadoop jobs: $ hadoop jar somejob.jar SomeClass arg1 arg2
# Run a batch job on 1 node: $ hod batch --dist Hadoop-2.3.0-cdh5.0 --label my-cluster --script=my-script.sh