Skip to main content
Join the official 2019 Python Developers SurveyStart the survey!

Optimize AWS EMR spark settings (spark-config-cheatsheet)

Project description

# Spark-optimizer

[![Build Status](https://api.travis-ci.org/delijati/spark-optimizer.svg?branch=master)](https://travis-ci.org/delijati/spark-optimizer)

Optimize spark settings (for cluster aka yarn run)

Original source: http://c2fo.io/c2fo/spark/aws/emr/2016/07/06/apache-spark-config-cheatsheet/

## Usage

Install:

$ virtualenv env
$ env/bin/pip install spark-optimizer

Dev install:

$ virtualenv env
$ env/bin/pip install -e .


Generate settings for `c4.4xlarge` with `4` nodes:

$ env/bin/spark-optimizer c4.4xlarge 4
{'spark.default.parallelism': '108',
'spark.driver.cores': '2',
'spark.driver.maxResultSize': '3481m',
'spark.driver.memory': '3481m',
'spark.driver.memoryOverhead': '614m',
'spark.executor.cores': '2',
'spark.executor.instances': '27',
'spark.executor.memory': '3481m',
'spark.executor.memoryOverhead': '614m'}

Update instance info:

$ env/bin/python spark_optimizer/emr_update.py


# CHANGES

0.1.4 (2019-03-10)
------------------

- add ec2 and emr cost to yaml


0.1.3 (2019-03-08)
------------------

- add emr cost to yaml
- export load yaml file
- make ``memory_overhead_coefficient`` editable


0.1.2 (2019-02-20)
------------------

- unpin the versions
- rename cli from _ to -


0.1.1 (2018-09-12)
------------------

- fix email


0.1.0 (2018-09-12)

- initial release

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for spark-optimizer, version 0.1.4
Filename, size File type Python version Upload date Hashes
Filename, size spark_optimizer-0.1.4.tar.gz (13.3 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page