simplespark 0.2.0

Spark environment manager and job orchestrator

Simple-Spark

Open source data platform for managing Apache Spark resources and job orchestration.

Project Goals

1. Provide a single CLI tool to perform all functions:
   • Orchestrate Spark clusters
   • Manage metastore / data warehouse
   • Run jobs and keep history
   • Setup notebook server with access to data warehouse
2. Compile all required configurations in single JSON/YAML file
3. Improve Spark job development:
   • Use same toolset to run jobs locally as on a cluster
   • Keep all resources in central directory for transparency and customization
   • Provide SimpleSpark object for easy access to easy access to all resources in codebase
   • Connect to remote resources through single API

Features

Modes

• local: Single node cluster setup on local machine
• standalone: Multi node cluster using Spark's standalone setup
• yarn: Multi node cluster built using Hadoop YARN setup (in development)
• kubernetes: Multi node cluster built Kubernetes setup (In development)

Services

• Cluster manager: Start/stop clusters, Spark UI for cluster
• Metastore: Central SQL server managing HIVE metastore (in development)
• Job orchestrator: Job scheduler via cron (in development)
• History server: SparkUI for past runs (in development)

Quick Start Guide

Once installed, the simplespark command can be used to create and switch between different Spark environments. Each environment contains a single cluster and a collection of resources to run on the cluster.

Install

The CLI tool is hosted on PyPi and can be installed using pip

pip install simplespark

Create Configuration

The configuration can be expressed in a single JSON file or be defined in multiple files which are merged on import.

Each mode will require different configuration proprieties to be defined and within each mode there are optional settings for specific add ins.

Templates

The easiest way to start is to create a template for the specific mode by running the command below:

simplespark template <mode> <file-path>

Required Properties

For all configurations, the following proprieties must be defined.

• name: Identifier of simplespark "environment"
• simplespark_home: Full path to directory used by simplespark to:
  • Store environment configurations
  • Store any required libraries
  • Store any scripts or custom modifications
• bash_profile_file: Full path to bash profile file used to set SIMPLESPARK_HOME environment variable
• packages: TODO
• driver: TODO

Import Configuration

TODO

Activate Environment

Activating a specific environment sets the JAVA/SCALA/SPARK_HOME variables for a shell session to point to that environment, as well as any other required setup.

This is done by calling an "activation" script generated in the import step:

source <environment-name>.sh

The environment will only be activated for the session in which this command called so that it is possible to interact with multiple environments at once.

The following native Spark commands will automatically connect to the activated environment:

• spark-shell
• spark-submit
• pyspark
• spark-sql

Start/Stop Clusters

A cluster can be started / stopped by running the command below. Once started, a cluster will stay up until stopped, even if a user switches to another environment.

simplespark start <environment-name>
simplespark stop <environment-name>