agileupstate 0.0.44

State mangement tool for Future Agile CICD

AgileUp State

Python 3.8+ project to manage AgileUP pipeline states with the following features:

• Linux and Windows compatible project.
• Defines state model.
• Saves and fetches states from vault.
• Exports private key for Linux SSH connections.
• Exports client PKI data for Windows WinRM connections.
• Creates cloud init zip file for mTLS connection data to Windows WinRM hosts.
• Exports ansible inventories for both Linux(SSH) and Windows(WinRM) connections.
• Provides simple connectivity tests.

Prerequisites

This project uses poetry is a tool for dependency management and packaging in Python. It allows you to declare the libraries your project depends on, it will manage (install/update) them for you.

Use the installer rather than pip installing-with-the-official-installer.

poetry self add poetry-bumpversion

poetry -V
Poetry (version 1.2.0)

Windows Path

Install poetry from powershell in admin mode.

(Invoke-WebRequest -Uri https://install.python-poetry.org -UseBasicParsing).Content | py -

The path will be C:\Users\<YOURUSER>\AppData\Roaming\Python\Scripts\poetry.exe which you will need to add to your system path.

Windows GitBash

When using gitbash you can setup an alias for the poetry command:

alias poetry="\"C:\Users\<YOURUSER>\AppData\Roaming\Python\Scripts\poetry.exe\""

Getting Started

poetry update

poetry install

Development

This project uses the hvac python module and to develop locally you can run vault as a docker service as detailed here local docker vault. For local development vault setup follow the VAULT guide for information.

Check your connection with the following command, note in development mode vault should not be sealed.

export VAULT_ADDR='http://localhost:8200'
export VAULT_TOKEN='8d02106e-b1cd-4fa5-911b-5b4e669ad07a'

poetry run agileupstate check

Required Environment State Variables

Variable	Description
SIAB_ID	Unique environment ID.
SIAB_CLOUD	Cloud vendor API mnemonic.
SIAB_LOCATION1	Cloud vendor defined cloud location, uksouth, etc.
SIAB_LOCATION2	Cloud vendor defined cloud location, UK South, etc.
SIAB_CONTEXT	Environment context, e.g. dev. test, prod.
SIAB_VALUES_PATH	Vault path to common environment values to be exported.
SIAB_DOMAIN	Optional public domain that overrides cloud provided DNS names.

SIAB_LOCATION: Azure has a different location string between "accounts" and "resources" and only uksouth is useful to the automation, but we must also provide UK South for resource groups. FIXME: Needs to be verified.

SIAB_VALUES_PATH: Rather than load variables into the delivery platform environment, there can be many, a better option is to define a YML file that contains all the required common variables for a specific environment and have the user upload that to vault. This application can then download the YML data file and convert it into an exports file that can be sourced by the pipelines. These environment values that are exported can then be used by this project and other utilities such as terraform, ansible and powershell.

SIAB_DOMAIN: Cloud DNS services might in some cases provide a DNS domain that is not the same as the public internet domain required by the project, for example server1.uksouth.cloudapp.azure.com might optionally be server1.meltingturret.io.

username/password: These values are common across the environment, for example Ubuntu Azure images use a username=azureuser, and so it simplifies configuration if the same credentials are used for Linux and Windows environments running in Azure for administration access, client administration access as well as PFX certificate files used on Windows for WinRM certificate authentication. For AWS ubuntu is the username for Ubuntu images the same approach can be taken there.

Required environment inputs:

These values should be setup in your CD platforms environment variables.

export SIAB_ID=001
export SIAB_CLOUD=arm
export SIAB_LOCATION1=uksouth
export SIAB_LOCATION2="UK South"
export SIAB_CONTEXT=dev
export SIAB_VALUES_PATH=siab-state/001-arm-uksouth-dev/siab-values/siab-values.yml

Required values inputs (stored in vault path SIAB_VALUES_PATH):

connection:
  url: https://server1.meltingturret.io:5986
  username: azureuser
  password: mypassword
  ca_trust_path: siab-client/chain.meltingturret.io.pem
  cert_pem: siab-client/[e-mail]
  cert_key_pem: siab-client/[e-mail]
cloud:
  group_owner: Paul Gilligan
  group_department: DevOps
  group_location: uksouth

Required Supporting Data

Some data is generated only once and thus can be uploaded to vault manually.

Uploading values file:

base64 ./siab-values.yml | vault kv put secret/siab-state/001-arm-uksouth-dev/siab-values/siab-values.yml file=-

Uploading pfx files:

base64 ./server1.meltingturret.io.pfx | vault kv put secret/siab-pfx/server1.meltingturret.io.pfx file=-
base64 ./[e-mail] | vault kv put secret/siab-pfx/[e-mail] file=-

Uploading pki files:

base64 ./chain.meltingturret.io.pem | vault kv put secret/siab-client/chain.meltingturret.io.pem file=-
base64 ./[e-mail] | vault kv put secret/siab-client/[e-mail] file=-
base64 ./[e-mail] | vault kv put secret/siab-client/[e-mail] file=-

Provision Use Case

Example steps required for the Windows terraform provision use case shown below.

agileupstate cloud-init --server-path=siab-pfx/ags-w-arm1.meltingturret.io.pfx --client-path=siab-pfx/[e-mail]
agileupstate create
source ./siab-state-export.sh                                                
terraform init
terraform apply -auto-approve
agileupstate save

Example steps required for the Linux terraform provision use case shown below.

agileupstate create
source ./siab-state-export.sh                                                
terraform init
terraform apply -auto-approve
agileupstate save

Destroy Use Case

Example steps required for recovering system state use case shown below which might be for example to destroy an environment.

agileupstate load
source ./siab-state-export.sh                                                
terraform init
terraform destroy -auto-approve

Ansible Use Case

Example steps required for the Windows ansible use case shown below.

agileupstate load
source ./siab-state-export.sh                                                  
agileupstate inventory-windows --ca-trust-path=siab-client/chain.meltingturret.io.pem --cert-pem=siab-client/[e-mail] --cert-key-pem=siab-client/[e-mail]
ansible-inventory -i inventory.ini --list
ansible -i inventory.ini "${TF_VAR_siab_name_underscore}" -m win_ping

Example steps required for the Linux ansible use case shown below.

agileupstate load
source ./siab-state-export.sh                        
agileupstate inventory-linux
ansible-inventory -i inventory.ini --list
ANSIBLE_HOST_KEY_CHECKING=True ansible -i inventory.ini --user "${TF_VAR_admin_username}" "${TF_VAR_siab_name_underscore}" -m ping

Exports Use Case

The yml file from SIAB_VALUES_PATH is exported to the file siab-state-export.sh with the contents as shown in the example below which can then be used by downstream utilities.

export SIAB_URL=https://server1.meltingturret.io:5986
export SIAB_USERNAME=azureuser
export SIAB_PASSWORD=mypassword
export SIAB_CA_TRUST_PATH=siab-client/chain.meltingturret.io.pem
export SIAB_CERT_PEM=siab-client/[e-mail]
export SIAB_CERT_KEY_PEM=siab-client/[e-mail]
export TF_VAR_group_owner=Paul Gilligan
export TF_VAR_group_department=DevOps
export TF_VAR_group_location=UK South
export TF_VAR_admin_username=azureuser
export TF_VAR_admin_password=mypassword
export TF_VAR_siab_name=001-arm-uksouth-dev
export TF_VAR_siab_name_underscore=001_arm_uksouth_dev

source ./siab-state-export.sh

Cloud Init Data Use Case

Example cloud init command that generates the zip file that is loaded onto Windows machines for WimRM certificate authentication.

poetry run agileupstate cloud-init --server-path=siab-pfx/ags-w-arm1.meltingturret.io.pfx --client-path=siab-pfx/[e-mail]

Ansible Windows Inventory Use Case

inventory.ini is generated with the target(s) and configuration information for a successful SSH connection from Ansible.

When export SIAB_DOMAIN=meltingturret.io:

[001_arm_uksouth_dev]
ags-w-arm1.meltingturret.io
[001_arm_uksouth_dev:vars]
ansible_user=azureuser
ansible_password=heTgDg!J4buAv5kc
ansible_connection=winrm
ansible_port=5986
ansible_winrm_ca_trust_path=chain.meltingturret.io.pem
ansible_winrm_cert_pem=[e-mail]
ansible_winrm_cert_key_pem=[e-mail]
ansible_winrm_transport=certificate

Ansible Linux Inventory Use Case

inventory.ini is generated with the target(s) and configuration information for a successful SSH connection from Ansible.

When export SIAB_DOMAIN=meltingturret.io:

[001_arm_uksouth_dev]
ags-w-arm1.meltingturret.io ansible_ssh_private_key_file=vm-rsa-private-key.pem

Run

poetry run agileupstate

Lint

poetry run flake8

Test

poetry run pytest

Publish

• By default we are using PYPI packages.
• Create yourself an access token for PYPI and then follow the instructions.

export PYPI_USERNAME=__token__ 
export PYPI_PASSWORD=<Your API Token>
poetry publish --build --username $PYPI_USERNAME --password $PYPI_PASSWORD

Versioning

We use SemVer for versioning. For the versions available, see the tags on this repository.

Releasing

We are using poetry-bumpversion to manage release versions.

poetry version patch

Dependency

Once the release has been created it is now available for you to use in other python projects via:

pip install agileupstate

And also for poetry projects via:

poetry add agileupstate

Contributing

Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.

License

This project is licensed under the Apache License, Version 2.0 - see the LICENSE file for details