terrabuddy 0.1

A python wrapper for multipass

terrabuddy

Terrabuddy is a templating engine built on top of terraform. Terrabuddy allows terraform to be used in a way that is more DRY, more auditable, and more modular.

Terrabuddy Features

• easily install and update terraform binaries
• easily manage component interdependencies via bundles
• easily inject variables into your modules
• built in git workflow support

Installation

System Requirements

• python3 with pip3 in your $PATH
• pip install terrabuddy

or install using setup.py

git clone https://github.com/jumidev/terrabuddy.git
cd terrabuddy/tb
make install             # installs the tb CLI tool with python requirements

tb --setup               # downloads and installs terraform
tb --setup-terraformrc   # (optional) installs useful terraform default settings
tb --setup-shell         # (optional) installs useful tb shell aliases

Installing & using terraform modules

terrabuddy can work with any terraform code.

• A repo with modules for Azure is provided here
• For AWS, here. (WIP)

Background

tb is the terrabuddy command line interface. It facilitates setting up your machine (see installation), allows you to list components and run terraform commands such as plan, apply, destroy and manages shared variables.

tb introduces three notions for managing resoureces: projects, components and bundles. A project is a git repo with a specific purpose. Components are individual objects that you create on your cloud provider. Bundles are sets of components that depend upon one another (and tb knows how to create them in the correct order).

Anatomy of a project:

Below is an example project encompassing three environments

prep/
prod/
sbx/
.envrc.tpl
.gitignore
project.yml
README.md
tfstate_store.hclt

• The top level directories in the project correspond to the environments, sbx (sandbox), prep (pre-prod), and prod.
• .envrc.tpl contains a template for required environment variables, notably TF_MODULES_ROOT which points to the repo in to which terraform-modules-azure has been cloned
• project.yml contains project-specific variables. Other .yml files within the project contain
• tfstate_store.hclt is an hcl template that will be applied in all components

Anatomy of a component:

Components are hclt files, e.g. hcl templates. HCL is a json-like declarative language used by terraform.

cat component.hclt

source  {
    # local path
    path = "/path/to/terraform/files"

    ....
    # OR a git repo
    repo = "https://github.com/jumidev/terraform-modules-aws.git"
}
inputs {
    foo = "bar"
}

• the inputs block contains the inputs which will be injected into the terraform module
• the source block tells tb which terraform module to use with this component, can be a local path or a git repo.
• hclt files contain variables, formatted **${like_this}**

Component parsing and variables

When tb is run on a component, it:

1. lints/parses all hclt files, fails if there are syntax errors
2. loads all yml files in the component and all parent directories as variables.
3. searches in the component directory and all parent directories for hclt files and combines them into a single hclt file.
4. replaces all variables in the hclt. If variables are left unreplaced, the parser stops with an error message.
5. if all variables are replaced, it performs the requested terraform action (plan, apply, destroy, etc)

Component parsing in detail

combines hclt files in the component directory with those in its parent directories. For example, at the root of the project there is a tfstate_store.hclt file. The contents of this file will be included in all components.

Another example is to refactor the source.hclt in a situation where a folder contains lots of components that use the same terraform module.

az-platform-infra$ ll prep/network_security_groups/*
prep/network_security_groups/bastion:
component.hclt

prep/network_security_groups/db:
component.hclt

prep/network_security_groups/db-apps:
component.hclt

prep/network_security_groups/public-webserver:
component.hclt

All of the above component.hclt contain the same source block

We can add an hclt file with a source block in the top level folder, thusly:

az-platform-infra$ ll prep/network_security_groups/*

prep/network_security_groups:
source.hclt

prep/network_security_groups/bastion:
component.hclt

prep/network_security_groups/db:
component.hclt

prep/network_security_groups/db-apps:
component.hclt

prep/network_security_groups/public-webserver:
component.hclt

Template Overriding

hclt files in upper directoryies can be overriddden by placing files with the same name deeper in the filesystem. For example, if you have a component that requires a specific tfstate_store configuration, you can override the one in the root folder, thusly:

prep/network_security_groups/public-webserver-other-remote-state:
component.hclt
tfstate_store.hclt  # overrides tfstate_store.hclt in project root

Component variables in detail

tb loads variables in a cascade process, starting at the project root and moving down the filesystem to the component. For example for prep/bastion/managed_disk, the following yml files are loaded:

1. project.yml in the project root, contains various key/value pairs
2. prep/env.yml
   env: "prep"
3. prep/bastion/appname.yml
   appname: "bastion"

If, for example, project.yml contains appname, its value will be overridden by prep/bastion/appname.yml.

You can use tb to display component variables with the showvars command

$ tb showvars prep/bastion/managed_disk

COMPONENT_DIRNAME=managed_disk
COMPONENT_PATH=prep/bastion/managed_disk
PROJECT_ROOT=/home/user/myprojects/az-platform-infra
TB_INSTALL_PATH=/home/user/wf/terrabuddy/tb
appname=bastion
env=prep
location=westeurope
organization=wforg
private_subnet_cidr=172.16.4.0/22
project_name=platform-infra
public_subnet_cidr=172.16.2.0/24
vnet_cidr=172.16.0.0/19

Special Variables

In addition to variables loaded in .yml files, tb also provides special variables

• COMPONENT_PATH full path to component, relative to project
• COMPONENT_DIRNAME innermost component directory
• PROJECT_ROOT absolute path to project

Bundles

Anywhere in a project, components can be bound together as a bundle, simply by placing a bundle.yml file with an order object.

For example, prep/bastion/bundle.yml

order:
    - network_interface
    - a_record
    - managed_disk
    - virtual_machine

The above bundle tells tb that when the user runs tb <command> prep/bastion it will in fact run four components: prep/bastion/network_interface, prep/bastion/a_record, etc... in the specified order. When running the destroy command, this order is reversed.

You can use the --dry argument on a bundle to see its components:

$ tb show prep/bastion --dry

tb show prep/bastion/network_interface
tb show prep/bastion/a_record
tb show prep/bastion/managed_disk
tb show prep/bastion/virtual_machine

Bundles also support wildcards and other bundles, for example prep/bundle.yml

order:
    - resource_group
    - dns/zone/*                    # all components in this dir, alphabetical order
    - virtual_network
    - subnets/*                     # all components in this dir, alphabetical order
    - application_security_groups/* # all components in this dir, alphabetical order
    - network_security_groups/*     # all components in this dir, alphabetical order
    - bastion                       # this is another bundle

tb <command> prep is thus a single "monster" bundle that runs the entire prep environment

Listing Components and bundles

tb uses the same commands as terraform: plan, apply, destroy, refresh, etc. Components in a project can be listed with the tb plan|apply|show command.

$ pwd
~/az-platform-infra

$ tb plan
OOPS, no component specified, try one of these (bundles are bold underlined):

tb plan sbx
tb plan sbx/application_security_groups/db-apps
tb plan sbx/application_security_groups/public-webserver
tb plan sbx/application_security_groups/manager
tb plan sbx/application_security_groups/db
tb plan sbx/application_security_groups/bastion
tb plan sbx/virtual_network
tb plan sbx/storage_account/std
tb plan sbx/storage_account/premium
tb plan sbx/dns/zone/sbx.weatherforce.net
tb plan sbx/dns/zone/sbx.prv
tb plan prod/keybaseca
tb plan prod/keybaseca/network_interface
tb plan prod/keybaseca/managed_disk
tb plan prod/keybaseca/virtual_machine
tb plan prod/keybaseca/a_record
tb plan prod/application_security_groups/db-apps
tb plan prod/application_security_groups/public-webserver
tb plan prod/application_security_groups/db
tb plan prod/bastion
tb plan prod/bastion/network_interface
tb plan prod/bastion/virtual_machine
tb plan prod/bastion/a_record
tb plan prod/resource_group
tb plan prod/network_security_groups/db-apps
tb plan prod/network_security_groups/public-webserver
tb plan prod/network_security_groups/db
...

running tb commands

Each of the above lines is a component. Running tb plan <component> will run the plan command on the component in question.

# TODO REDO

The above result means that the component already exists in Azure and is up to date with the component.

Git workflow integration

tb was designed to take git workflow considerations into account. When working with terrabuddy components, special care must be taken so ensure that developers working on separate components do not clobber each other's work. tb includes git checking functions to inform developers if their local git repository is behind remote changes.

For instance, developers A and B work on two unrelated components.

1. Developer A is working on an application VM in prep/testappA/virtual_machine. Developer A sees that network security rules do not allow their application to access required resources. They amend the security group prep/network_security_groups/db-apps to add the required security rules. They apply their changes and push to remote.
2. Developer B is working on an unrelated component. They too need to amend prep/network_security_groups/db-apps to add their own required security rules (different from those that Developer A added). They have forgotten to git pull so the changes made by Developer A are not on their machine. If they run tb apply prep/network_security_groups/db-apps they will clobber developer A's changes.
3. However tb does a git fetch and compares branches before each command.
4. Since Developer A has pushed their changes, tb on Developer B'a machine will show this message: GIT ERROR: You are on branch master and are behind the remote. Please git pull and/or merge before proceeding. Below is a git status:...

The above also works for feature branches. If developer B is working on a feature branch that was made prior to developer A's changes (pushed to master branch), tb will detect that Developer B's FB is behind master and prompt them to merge before proceeding.

TODO

• remove terragrunt dependency
  • predictable temp dirs in home dir
  • auto cleanup previous runs older than x days
  • auto manage remote states,
  • enrypted remote states
• example spoke and hub architecture