aenv 2.0.7

A Python 3 tool to fetch secure strings from the aws parameter store and injecting those into environment variables.

AEnv

A Python-based CLI tool to simplify the process of fetching and injecting environment variables from AWS Parameter Store.

Table of Contents

1. Description
2. Installation
3. Usage
4. Setup
5. Permissions
6. Concept
7. Access-parameter-store-entries
8. Authentication
9. Todos
10. Acknowledgments
11. License

Description

This CLI tool (aenv) allows you to fetch environment variables from AWS Parameter Store, injecting them into your local environment for use in your applications. The tool also supports authentication with MFA, including Yubikeys.

Installation

Prerequisites

• Install python3 and pip

• Install AWS CLI

• Boto3 (Will be installed with the aenv package)

  • Windows: setup Boto3 credentials

Main Package (aenv)

pip install aenv

Usage

aenv --help
# or
aenv -h

All current options:

aenv [-s <service/application>] [-i] [-n] [-e <env>] [-t <2fa key>] [-T] [-Y] [-u <aws username>] [-a <account number>] [-p <aws profile>] [-r <region>] <command>

Option	Explanation	Sample	Comment
-h	Shows help	aenv -h
-i	Starts aenv in interactive mode	aenv -i	Gives you a command line that you can interact with
-s <service/application>	For which service should the environment variables be loaded?	aenv -s CustomerService
-S	Sets a default service for aenv and writes it to a config file	aenv -S CustomerService	from now on "CustomerService" is the default service which means "-s CustomerService" is redundant
-n	Do not query the parameter store at all	aenv -n	Can be used to auth the current session with MFA
-e <env>	For which environment should the environment variables be loaded? For example Dev, Test or Prod (permission required)	aenv -e Prod
-t <2fa key>	Takes the 2FA key from your aws account	aenv -t 987123
-T	Lets you type in the 2FA key from your aws account during runtime	aenv -T	When you run your command aenv will ask for the token
-Y	Uses Yubikey for MFA auth	aenv -Y	During runtime aenv will use ykman to fetch the MFA-Key from your yubikey
-r <region>	Overwrites temporary the awscli default region	aenv -r eu-central-1	aenv will use the given region for example Frankfurt
-v	Verbose mode (more output)	aenv -v
-u <aws username>	Sets a specific username combined with -a gives you a faster runtime (otherwise this data needs to be retrieved via aws)	aenv -u [e-mail]
-a <account number>	Sets a specific account number combined with -u gives you a faster runtime (otherwise this data needs to be retrieved via aws)	aenv -a 999999999999
-p <aws profile>	If multiple aws profiles are available you can choose the profile otherwise aenv will use the default profile	aenv -p testUser1
-c <aws profile>	Container mode(enable this to make aenv work in ecs and codebuild)	aenv -c	permissions
<command>	Is the command to execute with environment variables injected.	aenv code	Will run VS Code with access to given environment variables

Setup

Lets start with setting up a simple example service. Let's call it UserService.

The UserService needs a database hostname, a database username and a database password.

Let's assume, we have two environments, "Dev" and "Test" and we want to inject the correct UserService, database hostname,... for the related environment.

Step 1 is to create the AWS parameter store entries. Let's create two entries:

/Dev/UserService/DB/hostname
# and 
/Test/UserService/DB/hostname

Both are SecureString, that hold the values:

db.dev.example.com for /Dev/UserService/DB/hostname
and
db.test.example.com for /Test/UserService/DB/hostname

Step 2 is to fetch those values and have them available as environment variables.

For the UserService variables on Dev run:

aenv -e Dev -s UserService 

This command fetches all entries from the parameter store with the path /Dev/UserService/* and makes them available as environment variables.

If you just want to echo out the DB hostname we created (/Dev/UserService/DB/hostname) you can run:

aenv -e Dev -s UserService echo '$SECRET_USERSERVICE_DB_HOSTNAME'

You can also run your service with aenv to have the correct DB hostname available in the service as an environment variable. The call for a Python or JVM service would be:

aenv -e Dev -s UserService java -jar service.jar
# or
aenv -e Dev -s UserService python service2.py

Both services now have access to the environment variable "SECRET_USERSERVICE_DB_HOSTNAME" containing the value that we defined for "/Dev/UserService/DB/hostname".

Permissions

Permissions (User)

Here is the minimal suggested set of IAM permissions to use aenv for all services that can be found for our Dev environment:

(Do not forget to adapt to account ID(123456789098) to your own )

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ssm:GetParametersByPath",
                "ssm:GetParameters",
                "ssm:ListTagsForResource",
                "ssm:GetParameter"
            ],
            "Resource": [
                "arn:aws:ssm:*:123456789098:parameter/Dev/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "ssm:DescribeParameters",
            "Resource": "*"
        }
    ]
}

These permissions allow to fetch all entries for any service in the Dev environment.

For example, fetching all environment variables for a service called UserService:

aenv -e Dev -s UserService

To further limit access to only allow loading environment variables for a specific service like "UserService" we need to adapt the "Resource":

"Resource": [
                "arn:aws:ssm:*:123456789098:parameter/Dev/UserService/*"
            ]

Permissions (AWS)

This is a work in progress!

Currently, the yubikey needs to be added as a virtual mfa and needs to be the first device in our Multi-factor authentication devices. Feel free to also add your Yubikey as a hardware mfa afterwards.(The AWS web console works flawless with multiple mfa's)

pip install --user yubikey-manager

Official documentation

Permissions (IAM policies / Instance roles)

Permission	Used in the code	Documentation	Comment
"ec2:DescribeTags"	clientEC2.describe_tags()	https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeTags.html
"sts:GetCallerIdentity"	clientSTS.get_caller_identity()	https://docs.aws.amazon.com/STS/latest/APIReference/API_GetCallerIdentity.html	Optional(No permissions are required to perform this operation.)
"sts:GetSessionToken"	clientSTS.get_session_token()	https://docs.aws.amazon.com/STS/latest/APIReference/API_GetSessionToken.html
"ssm:GetParametersByPath"	clientSSMMFA.get_parameters_by_path()	https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-paramstore-access.html
"iam:ListMFADevices"	boto3.client('iam').list_mfa_devices()	https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_examples_iam_mfa-selfmanage.html	Optional! (At the moment not in use but as soon aws API supports hardware tokens this can be enabled to let aenv support hardware MFA's)

tldr Minimal permissions:

“ec2:DescribeTags”

“sts:GetSessionToken”

“ssm:GetParametersByPath”

Advanced permission 1 (Enforce MFA authentication for accessing Prod parameters)

To enforce MFA authentication for all Prod parameters you can make use of the condition "MultiFactorAuthPresent" in your IAM permission.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ssm:PutParameter",
                "ssm:DeleteParameter",
                "ssm:GetParameterHistory",
                "ssm:GetParametersByPath",
                "ssm:GetParameters",
                "ssm:ListTagsForResource",
                "ssm:GetParameter",
                "ssm:DeleteParameters"
            ],
            "Resource": [
                "arn:aws:ssm:*:123456789098:parameter/Prod/*"
            ],
            "Condition": {
                "Bool": {
                    "aws:MultiFactorAuthPresent": "true"
                }
            }
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "ssm:DescribeParameters",
            "Resource": "*"
        }
    ]
}

Advanced permission 2 (Enforce MFA authentication for AWS feature / function)

Add the condition "MultiFactorAuthPresent" to your IAM permission:

    "Condition": {"Bool": {"aws:MultiFactorAuthPresent": "true"}}

Sample for sts:AssumeRole:

{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Principal": {"AWS": "ACCOUNT-B-ID"},
    "Action": "sts:AssumeRole",
    "Condition": {"Bool": {"aws:MultiFactorAuthPresent": "true"}}
  }
}

Now you need MFA authentication to run assume role commands. Sample call for this would be:

aenv -v -n -Y aws sts assume-role --role-arn "arn:aws:iam::123456789012:role/example-role" --role-session-name AWSCLI-Session

# -v enables verbose mode
# -n puts aenv in only authentication mode
# -Y authenticates the session with your YubiKey, alternatively you could use -t or -T

Concept

AEnv uses the parameter store path to define the environment and service name, following this schema:

/<Environment>/<Service-Name>/

# Which could look like:
/Prod/CustomerManagement/DB/USER
# Or
/Prod/CustomerManagement/DB/PASSWORD

Having those two in place would enable our CustomerManagement service running in our Prod environment to access, the environment variables: SECRET_CUSTOMERMANAGEMENT_DB_USER and SECRET_CUSTOMERMANAGEMENT_DB_PASSWORD

With both parameters in place, your CustomerManagement application/service, launched with aenv, could now access the database with the provided username and password.

aenv -e Prod -s CustomerManagement java -jar service.jar

Format for these environment variables:

Every environment variable that is loaded with aenv starts with "SECRET_".

Then the service-name and path, separated by underliners. (of course in upper case)

For example:

/Prod/CustomerManagement/DB/USER

would be accessible with:

SECRET_CUSTOMERMANAGEMENT_DB_USER

or

/Prod/CustomerManagement/DB/PASSWORD/USER1

would be accessible with:

SECRET_CUSTOMERMANAGEMENT_DB_PASSWORD_USER1

More about environment variables: Guide to Unix/Environment Variables

Access-parameter-store-entries

Testing single variables

Linux/Mac

aenv -e Dev -s UserService echo '$SECRET_USERSERVICE_UI_URL'

or

Windows

aenv -e Dev -s UserService echo %SECRET_USERSERVICE_UI_URL%  

How to access the environment variables in Kotlin and Python

How to access the environment variables

To access those environment variables you have to run your application/service with aenv.

aenv -e Dev -s UserService java -jar service.jar
//or
aenv -e Dev -s UserService python service2.py

Now these two services have access to all Dev environment variables for the UserService. Here are easy examples for Python and Kotlin:

Python:

import os
os.getenv('SECRET_USERSERVICE_HOSTNAME')

# For example conneting to a host depending on the environment:
....

hostname = os.getenv('SECRET_USERSERVICE_HOSTNAME')
....

Kotlin:

val envVar : String? = System.getenv("SECRET_USERSERVICE_HOSTNAME")

Bonus:

Running a local application with access to environment variables of a given service

Linux/Mac running IntelliJ with Test environment variables for the UserService

aenv -e Test -s UserService "/Applications/IntelliJ\ IDEA\ CE.app/Contents/MacOS/idea"

This can come in handy if you want to debug something that only seems to occure in the Test environment.

Authentication

AWS Server:

Easy!

Done by boto3. Boto3 automatically uses the in the instance role defined permissions.

(Details "Permissions" section)

Developer:

boto3 uses the aws CLI's authentication so make sure you set this up before ;)

AWS CLI

By default, aenv uses the aws CLI default profile, but of course, you can choose the profile, that you want to use, simply do:

aenv -p <awscli profile name>
#or 
aenv -h 
#to see more configuration options

(More details in the "Usage" section)

MFA

Multi-factor authentication is highly suggested!

https://lmgtfy.com/?q=why+mfa+is+important

Ok, all jokes aside especially for production parameters your IAM users should require MFA authentication at least for production parameters.

At least in my humble opinion, this should be a "better be safe than sorry" point.

Especially for your production systems!

AEnv supports multiple MFA options, details in the "Usage" section, here the short overview:

# Normal virtual mfa token:
aenv -t <TOKEN>

# Asks for the token during runtime:
aenv -T

# leads to an interactive token query during runtime:
$ aenv -T
$ Please enter token: 

#Yubikey Authenticator:
aenv -Y

Todos

• Add managing mode with TerminalMenu to read specific values
  • Function to add new entries
  • Function to update existing entries
  • Function to delete existing entries
• refactor whole code base
• Add better permission error handling (Especially the auth part with Yubikey handling)
• Add check if service exists/can be read with proper error message
• Update and correct -h / --help output
• Add regex filter for only loading specific variables
• Add regex filter to leave out variables from loading
• add -P to save a default profile
• Update initial setup instructions + console output for this(ykman + output for missing service)
• Option to list all available environments / services(discover/list env / list services)
• Check for ykman on -Y calls improve output
• Currently only the fist MFA device of any given account is used -> add mfa device selection + option for default selection
• cleanup/refactor documentation / improve overall structure
• add auth only mode(no env and no service name given)
• Add more information about container mode and necessary IAM permissions
• Enhance local profile/config setup/usage
• Handle the region in the same way services are handled
• Load multiple services at once instead of concatenating multiple aenv calls ( "aenv -s Service1 aenv -s Service2 ")
• Add feature for only loading certain variables to speed up loading
• Add assume role feature to support this setup more ease -> https://aws.amazon.com/de/blogs/security/enhance-programmatic-access-for-iam-users-using-yubikey-for-multi-factor-authentication/
• Add testing

Acknowledgments

Inspired by:

• Gunnar Zarncke - LinkedIn and his/troy gmbh's opensource credo projekt - Git Credo

Bug reports:

• Arif PEHLİVAN

• Carlos Freund

License

MIT Link

Support me :heart: :star: :money_with_wings:

If this project provided value, and you want to give something back, you can give the repo a star or support by buying me a coffee.