snpaas-cli 0.1.0

Manage SNPaaS

# ee-snpaas-cli

SNPaaS Command line program is based on Docker. This command is just a Python
wrapper for docker to execute:

```
docker run --rm -v $(pwd):/data -it platformengineering/snpaas-tools "${@}"
```

PIP makes it easy to distribute as client for everybody. All the important things
are done within the container.

Potentially it should run on Windows by installing Docker client command line
and delegating the execution to a box with docker daemon running (by defining
the proper environment variables), for example: the bastion host.

Environment variables for this program:

* `SNPAAS_DOCKER_IMAGE`: Docker image to launch. Default is `platformengineering/snpaas-tools:latest`.
* `SNPAAS_SUPPRESS_INFO_REGEX`: Default is `^# `. Regular expression to remove the motd generated by the image when it runs the .envrc file of the current folder.
* `SNPAAS_EXTRA_ENV`: Define extra variables to get exported inside the running container, plus the default ones defined above.


The script does not have any requirements, it should run on all platforms where
Python (2.7-3.x) and Docker-cli are available. In Windows, it should be possible
to delegate the execution of the Docker to a 3rd party server by setting the
proper environment variables.


# Docker development

The `docker` folder have all the sources to create the Docker container.
Creating and publishing the container is done with the script `publish-dockers-dockerhub.sh`
which takes all folders (they must have a `Dockerfile`) and creates a Docker image
with the name of each folder. Once the image is created is automatically pushed
to dockerhub.com using *platformengineering* account. You have to be loged in
with `docker login` in other to publish the Image.

* **Images are public available. DO NOT INCLUDE SECRETS THERE**
* **Images are public available. DO NOT INCLUDE SECRETS THERE**
* **Images are public available. DO NOT INCLUDE SECRETS THERE**
* **Images are public available. DO NOT INCLUDE SECRETS THERE**
* **Images are public available. DO NOT INCLUDE SECRETS THERE**
* **Images are public available. DO NOT INCLUDE SECRETS THERE**
* **Images are public available. DO NOT INCLUDE SECRETS THERE**
* **Images are public available. DO NOT INCLUDE SECRETS THERE**
* **Images are public available. DO NOT INCLUDE SECRETS THERE**
* **Images are public available. DO NOT INCLUDE SECRETS THERE**
* **Images are public available. DO NOT INCLUDE SECRETS THERE**
* **Images are public available. DO NOT INCLUDE SECRETS THERE**
* **Images are public available. DO NOT INCLUDE SECRETS THERE**

If you include them, your next task will be rotate all secrets everywhere!, without rest!
Anyway, **if you include secrets, you are doing something wrong. Docker images are not created to store secrets!**


## Upload to PyPI

1. Create a `.pypirc` configuration file. This file holds your information for authenticating with PyPI.

```
[distutils]
index-servers = pypi

[pypi]
repository=https://pypi.python.org/pypi
username=your_username
password=your_password
```
2. Login and upload it to PyPI

```
python setup.py register -r pypi
python setup.py sdist upload -r pypi
```


## How this thing works?

1. `Dockerfile` includes all software (with specific versions).
2. Docker reads the `ENTRYPOINT ["/bin/bash", "/run.sh"]`, which will be always executed.
3. Docker will make available the current folder to the running container inside `/data`, all programs and scripts will be executed there.
4. The `run.sh` script reads the `.envrc` files and presents the usage (help) if needed.
5. Depending on the first argument, it will execute a command like `bosh`, `credhub` ... or it will run `manage-deployment.sh` script


The script `manage-deployment.sh` parses all the folder structure and runs the commands to manage a deployment. It is very verbose and always shows what it is doing, with colors, you can replicate everything copying the blue commands.


# Usage

You can install it by running `setup.py`:
```
$ python setup.py install
```

or using `pip`:

```
$ pip install snpaas-cli
```

Also, you can copy and run it directly by calling the program
`snpaas.py` instead of `snpaas`.


Once it is installed, type `snpaas`

```
$ snpaas

*** Docker image: platformengineering/snpaas-tools ***

This docker image packages the tools SNPaaS team is using to
manage the deployments. It includes clients for cf, bosh, credhub ...


# Usage

You can execute them directly. In the current folder you can define a '.envrc' file
with all environment variables you want to be setup in the running container.
If you do not have a '.envrc' file but you have the following environment variables
in your environment, then Bosh-cli and Credhub-cli will automatically log-in:

"BOSH_CLIENT"
"BOSH_CLIENT_SECRET"
"BOSH_ENVIRONMENT"
"BOSH_CA_CERT"
"CREDHUB_SERVER"
"CREDHUB_CLIENT"
"CREDHUB_SECRET"
"CREDHUB_CA_CERT"
"GCP_PROJECT"
"GCP_ZONE"
"GCP_REGION"

Then you are ready to manage to manage deployments, with this subcommands and options:

<subcommand> <folder> [options]

Options:
-m Specify a manifest file, instead of generating a random one
-p Deployments path. Default is
-h Shows usage help

Subcommands:
help Shows usage help
interpolate Create the manifest for an environment
deploy [-f] Update or upgrade deployment after applying cloud/runtime configs
destroy [-f] Delete deployment (does not delete cloud/runtime configs)
cloud-config Apply cloud-config operations files from Director cloud-config
runtime-config Apply runtime-config operations files from Director runtime-config
import-secrets Set secrets in Credhub from <deployment-folder>/ file
list-secrets List secrets from Credhub for <deployment-folder>
export-secrets Download secrets from Credhub to <deployment-folder>/


# Folder structure:

<deployment-folder>
├── <boshrelease-git-submodule-folder>
├── base.yml -> <boshrelease-git-submodule-folder>/manifest/logstash.yml
├── operations
│ ├── 10-operation.yml -> ../<boshrelease-git-submodule-folder>/manifest/operations/operation.yml
│ ├── 20-operation2.yml -> ../<boshrelease-git-submodule-folder>/manifest/operations/operation2.yml
│ ├── 99-springer-nature-operation-custom.yml
├── secrets.yml
├── cloud-config
├── runtime-config
└── variables
├── variables-custom1.yml
├── variables-custom1.yml
└── variables-provided.yml -> ../<boshrelease-git-submodule-folder>/manifest/vars.yml


Credhub secrets have to be type 'value' and they will be imported/exported inside a
'secrets.yml' file, which, in case it exits, the script will read.


# Usage examples

Given a deployment folder called 'app-logging' with this structure:

app-logging
├── logstash.yml -> cf-logging-boshrelease/manifest/logstash.yml
├── operations
│ ├── 20-cf-apps-es.yml -> ../cf-logging-boshrelease/manifest/operations/pipelines/cf-apps-es-throttling.yml
│ ├── 25-add-statsd-conf.yml -> ../cf-logging-boshrelease/manifest/operations/add-statsd-conf.yml
│ ├── 30-add-throttle-param.yml -> ../cf-logging-boshrelease/manifest/operations/add-throttle-param.yml
│ ├── 50-add-es-cloud-id.yml -> ../cf-logging-boshrelease/manifest/operations/add-es-cloud-id.yml
│ ├── 50-add-es-xpack.yml -> ../cf-logging-boshrelease/manifest/operations/add-es-xpack.yml
│ ├── 80-add-logstash-exporter.yml -> ../cf-logging-boshrelease/manifest/operations/add-logstash-exporter.yml
│ ├── 90-add-release-version.yml -> ../cf-logging-boshrelease/manifest/operations/add-release-version.yml
│ ├── 99-deployment-settings.yml
│ └── 99-iaas.yml -> ../cf-logging-boshrelease/manifest/operations/add-iaas-parameters.yml
├── secrets.yml
└── variables
├── iaas.yml
├── throttle-param.yml
└── vars-release-version.yml -> ../cf-logging-boshrelease/manifest/vars-release-version.yml


* To deploy or update the deployment called 'app-logging', execute: 'deploy app-logging'.
If you do not want to answer 'y/n' question when bosh runs, just use '-f' option:
'deploy app-logging -f'
* To list secrets of the deployment from Credhub: 'list-secrets app-logging'
* Exporting the secrets of the deployment from Credhub to file 'app-logging/secrets.yml'
is done with: 'export-secrets app-logging'. Only credentials type value are supported.
* Import the secrets of the deployment to Credhub from a file 'app-logging/secrets.yml':
'import-secrets app-logging'. All credentials will be imported as type value.
* In order to apply the name of the folder to the deployment, you need to provide
an operations file with this operation.

- type: replace
path: /name
value: ((deployment_name))

The variable 'deployment_name' is provided by the program and taken from the
folder name

# Why?

* Avoid writting documentation about each deployment. Each folder has a specific
structure, the script only goes through the folder structure and generates
and applies the operations to the base manifest.
* Make it easy to manage bosh deployments. If the deployment folder is properly
done, replicating it for testing purposes is as easy as clonning the folder
and deploy again.
* Easily manage Credhub secrets in a deployment. One can import, export and
list secrets from files to Credhub (Only secrets type value, the other ones
are managed via the 'variables' section in a manifest!).
* Show you the commands it runs each time, with colorized output in order to
stand out if something went wrong.
* Loose coupled. The sript is a wrapper for Bosh and Credhub clients. If you do
not like the script, you do not have to use it. The folder structure is
self-documented, so you only need to build the bosh and credhub parameters
from these files.


# Programs and versions installed

Additionally you can directly execute all the following programs installed, just
by typing then as argument.

SPIFF: 1.0.8
CREDHUB: 1.7.7
GCLOUD_SDK: 220.0.0
CF_CLI: 6.40.0
BOSH_CLI: 5.2.1
CERTSTRAP: 1.1.1
JQ: 1.5
BBR: 1.2.2
SPRUCE: 1.18.0
TERRAFORM: 0.11.8
FLY: 3.14.1
```


# Author

Springer Nature Engineering Enablement (EE), Jose Riguera Lopez ([e-mail])