Setup and manage a Apache Spark cluster in EC2
The CGCloud plugin for Spark lets you setup a fully configured Apache Spark cluster in EC2 in just minutes, regardless of the number of nodes. While Apache Spark already comes with a script called spark-ec2 that lets you build a cluster in EC2, CGCloud Spark differs from spark-ec2 in the following ways:
The cgcloud-spark package requires that the cgcloud-core package and its prerequisites are present.
Read the entire section before pasting any commands and ensure that all prerequisites are installed. It is recommended to install this plugin into the virtualenv you created for CGCloud:
source ~/cgcloud/bin/activate pip install cgcloud-spark
If you get DistributionNotFound: No distributions matching the version for cgcloud-spark, try running pip install --pre cgcloud-spark.
Be sure to configure cgcloud-core before proceeding.
Modify your .profile or .bash_profile by adding the following line:
Login and out (or, on OS X, start a new Terminal tab/window).
Verify the installation by running:
The output should include the spark-box role.
Create a single t2.micro box to serve as the template for the cluster nodes:
cgcloud create -IT spark-box
The I option stops the box once it is fully set up and takes an image (AMI) of it. The T option terminates the box after that.
Now create a cluster by booting a master and the slaves from that AMI:
cgcloud create-cluster spark -s 2 -t m3.large
This will launch a master and two slaves using the m3.large instance type.
SSH into the master:
cgcloud ssh spark-master
… or the first slave:
cgcloud ssh -o 0 spark-slave
… or the second slave:
cgcloud ssh -o 1 spark-slave
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|File Name & Checksum SHA256 Checksum Help||Version||File Type||Upload Date|
|cgcloud_spark-1.6.0-py2.7.egg (22.3 kB) Copy SHA256 Checksum SHA256||2.7||Egg||Nov 22, 2016|
|cgcloud-spark-1.6.0.tar.gz (10.0 kB) Copy SHA256 Checksum SHA256||–||Source||Nov 22, 2016|