This release is a pre-release and may not be stable for production use.
The CGCloud Spark project lets you setup a functioning Apache Spark cluster in EC2 in just minutes, regardless of the number of nodes. It is a plugin to CGCloud. While Apache Spark already comes with a script called spark-ec2 that lets you build a cluster in EC2, CGCloud Spark differs from spark-ec2 in the following ways (bad news first):
Tachyon and Yarn are not yet supported.
Setup time does not scale linearly with the number of nodes. Setting up a 100 node cluster takes just as long as setting up a 10 node cluster (2-3 min, as opposed to 45min with spark-ec2). This is made possible by baking all required software into a single AMI. All slave nodes boot up concurrently and autonomously in just a few minutes.
Unlike with spark-ec2, the cluster can be stopped and started via the EC2 API or the EC2 console, without involvement of cgcloud.
The Spark services (master and worker) run as an unprivileged user, not root as with spark-ec2. Ditto for the HDFS services (namenode, datanode and secondarynamenode).
The Spark and Hadoop services are started automatically as the instance boots up, via a regular init script.
Nodes can be added easily, simply by booting up new instances from the AMI. They will join the cluster automatically. HDFS may have to be rebalanced after that.
You can customize the AMI that cluster nodes boot from by subclassing the SparkMaster and SparkSlave classes.
CGCloud Spark uses the CGCLoud Agent which takes care of maintaining a list of authorized keypairs on each node.
CGCloud Spark is based on the official Ubuntu Trusty 14.04 LTS, not the Amazon Linux AMI.
Prerequisites
The cgcloud-spark package requires that the cgcloud-core package and its prerequisites are present.
Installation
Read the entire section before pasting any commands and ensure that all prerequisites are installed. It is recommended to install cgcloud into a virtualenv. Create a virtualenv and use pip to install cgcloud-spark:
virtualenv cgcloud source cgcloud/bin/activate pip install cgcloud-spark
On OS X systems with a Python that was installed via HomeBrew, you should omit sudo. You can find out if that applies to your system by running which python. If it prints /usr/local/bin/python you are most likely using a HomeBrew Python and should therefore omit sudo. If it prints /usr/bin/python you need to run pip with sudo.
Be sure to configure cgcloud-core before proceeding.
Configuration
Modify your .profile or .bash_profile by adding the following line:
export CGCLOUD_PLUGINS=cgcloud.spark
Login and out (or, on OS X, start a new Terminal tab/window).
Verify the installation by running:
cgcloud list-roles
The output should include the spark-box role.
Usage
Create a single t2.micro box to serve as the template for the cluster nodes:
cgcloud create spark-box -I -T
The -I switch stops the box once it is fully set up and takes an AMI of it. The -T switch terminates it after that.
Create a cluster by booting a master and the slaves from that AMI:
cgcloud create-spark-cluster -s 2 -t m3.large
This will launch a master and two slaves using the m3.large instance type.
SSH into the master:
cgcloud ssh spark-master
… or the first slave:
cgcloud ssh spark-slave -o 0
… or the second slave:
cgcloud ssh spark-slave -o 1
Interactions with Spark and HDFS should be done as the sparkbox user:
cgcloud ssh spark-master -l sparkbox hdfs dfs -ls / spark-shell
Otherwise you are likely to run into permission problems.
1 maintainer
Metadata
1 maintainer
Release files for cgcloud-spark 1.1a1.dev129
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| cgcloud-spark-1.1a1.dev129.tar.gz | 11.9 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| cgcloud_spark-1.1a1.dev129-py2.7.egg | Legacy Egg format | - | - | Details |
Total release size: 38.6 kB
Release files / cgcloud-spark-1.1a1.dev129.tar.gz
| Download URL | cgcloud-spark-1.1a1.dev129.tar.gz |
|---|---|
| Size | 11.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ff28be255613ed92b353e9c807bc770f89e3290b0634c3677207edc2105d4ba5
|
|
BLAKE2b-256 checksum How to use checksums |
82b62307fb387d96cbe179cc2439b3bd7034730cce6d9a54ec3c3b16db2e3496
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
Release files / cgcloud_spark-1.1a1.dev129-py2.7.egg
| Download URL | cgcloud_spark-1.1a1.dev129-py2.7.egg |
|---|---|
| Size | 26.7 kB |
| Tags | Egg |
|
SHA-256 checksum How to use checksums |
647e2b2b8300c62f6e9732f332e2c06c15ecde594f564283196706f82d299092
|
|
BLAKE2b-256 checksum How to use checksums |
13af585220ff5ea7a24ecef810ba529db064fde2d4ffb8e261f2885258476cd0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
Security for cgcloud-spark 1.1a1.dev129
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
