Launches an AWS Elastic MapReduce cluster using templated configuration files written in JSON. Meant to make deployments consistent and reproducable.
Project description
# EMR Launcher
Launches EMR clusters using config files for consistent run-time behavior when setting up a cluster.
## To Run
Starting a new cluster: ` PYTHONPATH=/path/to/emr_launcher python emr_launcher/launcher.py /path/to/config/<my_config>.json `
Adding steps to an existing cluster ` PYTHONPATH=/path/to/emr_launcher python emr_launcher/launcher.py config/<my_config>.json --job-id <job_id_of_existing_cluster> `
## Creating configs
Create a JSON file in the configs directory. Fill the config based on the parameters defined here: http://boto3.readthedocs.io/en/latest/reference/services/emr.html#EMR.Client.run_job_flow
or
build off the example config
## Using templating in configs
In any JSON config file function defined in the template_functions module can be used inside the config using jinja2 style templating. If you require a new function, add it to the template_functions module and it will be available to use in any config.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for emr_launcher-1.0.0-py2-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | acf305651eac5243105ec5fc9df717dcdcfcf8b4571c881a027b339427d813ab |
|
MD5 | 338cb2223c5ee2425189bf1a2eecc733 |
|
BLAKE2b-256 | 87db509d6c56970d7b1b6f8bad5d6f8e3520248a0044c44f967e72f5911b06f5 |