Simplify infrastructure/app testing/deployment
Project description
What?
A lightweight wrapper around linting (e.g. yamllint) & infrastructure deployment tools (e.g. CloudFormation, Terraform, Serverless) to ease management of per-environment configs & deployment.
Why?
Very simple configuration to:
Perform automatic linting/verification
Ensure deployments are only performed when an environment config is present
Define an IAM role to assume for each deployment
Wrangle Terraform backend/workspace configs w/ per-environment tfvars
Avoid long-term tool lock-in
runway is a simple wrapper around standard tools. It simply helps to avoid convoluted Makefiles / CI jobs
How?
Basic Concepts
Modules:
A single-tool configuration of an application/component/infrastructure (e.g. a set of CloudFormation stacks to deploy a VPC, a Serverless app)
Regions:
AWS regions
Environments:
A Serverless stage, a Terraform workspace, etc.
Environments are determined automatically from:
Git branches. We recommend promoting changes through clear environment branches (prefixed with ENV-). For example, when running a deployment in the ENV-dev branch dev will be the environment. The master branch can also be used as a special ‘shared’ environment called common (e.g. for modules not normally promoted through other environments).
The parent folder name of each module. For teams with a preference or technical requirement to not use git branches, each environment can be represented on disk as a folder. Instead of promoting changes via git merges, changes can be promoted by copying the files between the environment folders. See the ignore_git_branch runway.yml config option.
The folder name of the module itself (not its parent folder) if the ignore_git_branch and current_dir runway.yml config config options are both used (see “Directories as Environments with a Single Module” in “Repo Structure”).
The DEPLOY_ENVIRONMENT environment variable.
Deployments:
Mappings of modules to regions, optionally with AWS IAM roles to assume
runway.yml:
List of deployments
When the CI environment variable is set, all deployments are run in order; otherwise, the user is prompted for deployments to run.
Repo Structure
Git Branches as Environments
Sample repo structure, showing 2 modules using environment git branches (these same files would be present in each environment branch, with changes to any environment promoted through branches):
. ├── myapp.cfn │ ├── dev-us-west-2.env │ ├── prod-us-west-2.env │ ├── myapp.yaml │ └── templates │ └── foo.json ├── myapp.tf │ ├── backend.tfvars │ ├── dev-us-east-1.tfvars │ ├── prod-us-east-1.tfvars │ └── main.tf └── runway.yml
Directories as Environments
Another sample repo structure, showing the same modules nested in environment folders:
. ├── dev │ ├── myapp.cfn │ │ ├── dev-us-west-2.env | │ ├── prod-us-west-2.env │ │ ├── myapp.yaml │ │ └── templates │ │ └── myapp_cf_template.json │ ├── myapp.tf │ │ ├── backend.tfvars │ │ ├── dev-us-east-1.tfvars | │ ├── prod-us-east-1.tfvars │ │ └── main.tf │ └── runway.yml └── prod ├── myapp.cfn │ ├── dev-us-west-2.env │ ├── prod-us-west-2.env │ ├── myapp.yaml │ └── templates │ └── myapp_cf_template.json ├── myapp.tf │ ├── backend.tfvars │ ├── dev-us-east-1.tfvars │ ├── prod-us-east-1.tfvars │ └── main.tf └── runway.yml
Directories as Environments with a Single Module
Another sample repo structure, showing environment folders containing a single CloudFormation module at their root (combining the current_dir & ignore_git_branch “Runway Config File” options to merge the Environment & Module folders):
. ├── dev │ ├── dev-us-west-2.env │ ├── prod-us-west-2.env │ ├── myapp.yaml │ ├── runway.yml │ └── templates │ └── myapp_cf_template.json └── prod ├── dev-us-west-2.env ├── prod-us-west-2.env ├── myapp.yaml ├── runway.yml └── templates └── myapp_cf_template.json
Runway Config File
runway.yml example:
--- # Order that modules will be deployed. A module will be skipped if a # corresponding env/config file is not present in its folder. # (e.g., for cfn modules, if a dev-us-west-2.env file is not in the 'app.cfn' # folder when running a dev deployment of 'app' to us-west-2 then it will be # skipped.) deployments: - modules: - myapp.cfn regions: - us-west-2 - modules: - myapp.tf regions: - us-east-1 assume-role: # optional # When running multiple deployments, post_deploy_env_revert can be used # to revert the AWS credentials in the environment to their previous # values # post_deploy_env_revert: true dev: arn:aws:iam::account-id1:role/role-name prod: arn:aws:iam::account-id2:role/role-name # A single ARN can be specified instead, to apply to all environments # arn: arn:aws:iam::account-id:role/role-name account-alias: # optional # A mapping of environment -> alias mappings can be provided to have # Runway verify the current assumed role / credentials match the # necessary account dev: my_dev_account prod: my_dev_account account-id: # optional # A mapping of environment -> id mappings can be provided to have Runway # verify the current assumed role / credentials match the necessary # account dev: 123456789012 prod: 345678901234 # If using environment folders instead of git branches, git branch lookup can # be disabled entirely (see "Repo Structure") # ignore_git_branch: true
runway.yml can also be placed in a module folder (e.g. a repo/environment containing only one module doesn’t need to nest the module in a subfolder):
--- # This will deploy the module in which runway.yml is located deployments: - current_dir: true regions: - us-west-2 assume-role: arn: arn:aws:iam::account-id:role/role-name # If using environment folders instead of git branches, git branch lookup can # be disabled entirely (see "Repo Structure"). See "Directories as Environments # with a Single Module" in "Repo Structure". # ignore_git_branch: true
Installation
Install Python 2
On Linux (assuming default Bash shell; adjust for others appropriately):
Setup your shell for user-installed (non-root) pip packages:
echo 'export PATH=$HOME/.local/bin:$PATH' >> ${HOME}/.bashrc
source ${HOME}/.bashrc
Install Python/pip:
Debian-family (e.g. Ubuntu): sudo apt-get -y install python-pip python-minimal
Amazon Linux should should work out of the box
RHEL-family:
If easy_install is available: easy_install --user pip
Otherwise, enable EPEL and sudo yum install python-pip
On macOS (assuming default Bash shell; adjust for others appropriately):
if ! which pip > /dev/null; then easy_install --user pip; fi
echo 'export PATH="${HOME}/Library/Python/2.7/bin:${PATH}"' >> ${HOME}/.bash_profile
source ${HOME}/.bash_profile
On Windows:
This can be done via the Chocolately package manager (e.g. choco install python2), or manually from their website
If installing via Chocolately, default options will be sufficient. Close/reopen terminals after installation to use the updated PATH
If installing manually, use the default options with the exception of the “Add python to Path” (it should be enabled).
Add %USERPROFILE%\AppData\Roaming\Python\Scripts to PATH environment variable
Install runway (doesn’t require sudo/admin permissions):
pip install --user runway
If this produces an error like Unknown distribution option: 'python_requires', upgrade setuptools first pip install --user --upgrade setuptools
Use
runway test (aka runway preflight) - execute this in your environment to catch errors; if it exits 0, you’re ready for…
runway plan (aka runway taxi) - this optional step will show the diff/plan of what will be changed. With a satisfactory plan you can…
runway deploy (aka runway takeoff) - if running interactively, you can choose which deployment to run; otherwise (i.e. on your CI system) each deployment will be run in sequence.
Removing Deployments
runway destroy (aka runway dismantle) - if running interactively, you can choose which deployment to remove; otherwise (i.e. on your CI system) every deployment will be run in reverse sequence (use with caution).
Module Configurations
CloudFormation
CloudFormation modules are managed by 2 files: a key/value environment file, and a yaml file defining the stacks/templates/params.
Environment - name these in the form of env-region.env (e.g. dev-contoso.env):
# Namespace is used as each stack's prefix # We recommend an (org/customer)/environment delineation namespace: contoso-dev environment: dev customer: contoso region: us-west-2 # The stacker bucket is the S3 bucket (automatically created) where templates # are uploaded for deployment (a CloudFormation requirement for large templates) stacker_bucket_name: stacker-contoso-us-west-2
Stack config - these can have any name ending in .yaml (they will be evaluated in alphabetical order):
# Note namespace/stacker_bucket_name being substituted from the environment namespace: ${namespace} stacker_bucket: ${stacker_bucket_name} stacks: myvpcstack: # will be deployed as contoso-dev-myvpcstack template_path: templates/vpc.yaml # The enabled option is optional and defaults to true. You can use it to # enable/disable stacks per-environment (i.e. like the namespace # substitution above, but with the value of either true or false for the # enabled option here) enabled: true myvpcendpoint: template_path: templates/vpcendpoint.yaml # variables map directly to CFN parameters; here used to supply the # VpcId output from the myvpcstack to the VpcId parameter of this stack variables: VpcId: ${output myvpcstack::VpcId}
The config yaml supports many more features; see the full Stacker documentation for more detail (e.g. stack configuration options, additional lookups in addition to output (e.g. SSM, DynamoDB))
Serverless
Standard Serverless rules apply, with the following recommendations/caveats:
Runway environments map directly to Serverless stages.
A package.json file is required, specifying the serverless dependency and a sls script, e.g.:
{ "name": "mymodulename", "version": "1.0.0", "description": "My serverless module", "main": "handler.py", "devDependencies": { "serverless": "^1.25.0" }, "scripts": { "sls": "sls" }, "author": "Serverless Devs", "license": "ISC" }
We strongly recommend you commit the package-lock.json that is generated after running npm install
Each stage requires its own config file (even if empty for a particular stage), in one of the following forms:
config-STAGE-REGION.yml config-STAGE.yml config-STAGE-REGION.json config-STAGE.json
Terraform
Standard Terraform rules apply, with the following recommendations/caveats:
Each environment requires its own tfvars file, in the form of ENV-REGION.tfvars (e.g. dev-contoso.tfvars).
We recommend (but do not require) having a backend configuration separate from the terraform module code:
main.tf:
terraform { backend "s3" { key = "some_unique_identifier_for_my_module" # e.g. contosovpc } } # continue with code here...
backend-REGION.tfvars, or backend-ENV-REGION.tfvars, or backend-ENV.tfvars (e.g. backend-us-east-1.tfvars):
bucket = "SOMEBUCKNAME" region = "SOMEREGION" dynamodb_table = "SOMETABLENAME"
tfenv
If a .terraform-version file is placed in the module, tfenv will be invoked to ensure the appropriate version is installed prior to module deployment.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.