typed-config-aws-sources 0.4.2

AWS config sources for the typedconfig package

typed-config-aws-sources

AWS config sources for the typed-config package.

pip install typed-config-aws-sources

Requires python 3.6 or above.

Basic usage

Please read the readme for typed-config first.

# my_app/config.py
from typedconfig_awssource import IniS3ConfigSource
from typedconfig import Config, key, section

@section('database')
class AppConfig(Config):
    port = key(cast=int)

config = AppConfig()
config.add_source(IniS3ConfigSource('my_bucket_name', 'config_key.cfg'))
config.read()

# my_app/main.py
from my_app.config import config
print(config.host)

Supplied Config Sources

IniS3ConfigSource

This loads configuration from an INI file stored in an S3 bucket

from typedconfig_awssource import IniS3ConfigSource
source = IniS3ConfigSource('bucket_name', 'key_name.cfg', encoding='utf8', must_exist=True)

• Supply the bucket name and key name as the first two arguments
• encoding defaults to 'utf8' if not supplied
• must_exist defaults to True if not supplied. If must_exist is False, and the bucket or key can't be found, or AWS credentials fail, then no error is thrown and this config source will just return than it cannot find the requested config value every time.

An example INI file might look like this:

[database]
port = 2000

DynamoDBConfigSource

This reads configuration from a DynamoDB table. The table should have a partition key which holds the config section, a sort key which holds the config key name, and another 'column' containing the config value as a string.

So an item in DynamoDB corresponding to the above INI file example would look like this

{
    "section": "database",
    "key": "port",
    "value": "2000"
}

Create the DynamoDBConfigSource like this:

from typedconfig_awssource import DynamoDbConfigSource
source = DynamoDbConfigSource('table_name', 
                               section_attribute_name='config_section_column_name',
                               key_attribute_name='config_key_column_name',
                               value_attribute_name='config_value_column_name')

• The first argument is the DynamoDB table name and is required
• The other three arguments are optional, and are supplying the attribute (or "column") names in the table which store the three things defining a config parameter (section, key, and value)
• Default attribute names are "section", "key", and "value"

SecretsManagerConfigSource

This reads secret values from secrets manager. Permission to read AWS secrets is required. One secrets should be stored for each config section with the name format prefix/section, and contain json key-value pairs. For example, for a project called myproject there may be a secret called myproject/database containing the following value. Note that even numeric values should be stored as strings.

{
    "user": "secretuser",
    "password": "secretpassword"
}

Create the SecretsManagerConfigSource like this:

from typedconfig_awssource import SecretsManagerConfigSource
source = SecretsManagerConfigSource('myproject', must_exist=False, only_these_keys={('s', 'a'), ('s', 'b')})

• The first argument passed is the prefix which is placed before the / in the secret name. So when I try to get the database password, the secret myproject/database is retrieved, the JSON is parsed and value the field password is returned.
• The must_exist argument specifies whether to error if AWS secretsmanager cannot be accessed, or if the key does not exist. Default is False.
• The only_these_keys argument specifies a limited set of configuration keys. They are provided as (section, key) tuples. If provided, the config source will only act when these parameters are requested. This prevents unnecessary AWS API calls, which slow down configuration setup, for config values which you know are not available from secrets manager. Setting only_these_keys=None (the default) will check secrets manager for all config keys.

ParameterStoreConfigSource

This reads (optionally secret) values from AWS SSM parameter store. Storing secrets here is cheaper than using secrets manager. Permission to read from SSM parameter store is required. Each config parameter should be stored in parameter store as an individual SecureString parameter. For example, I would store the database password in a key called

myproject/database/password

where database is the section name and password is the configuration key name.

Create a ParameterStoreConfigSource like this:

from typedconfig_awssource import ParameterStoreConfigSource
source = ParameterStoreConfigSource('myproject',
                                    must_exist=False,
                                    only_these_keys={('s', 'a'), ('s', 'b')},
                                    batch_preload=False)

• The first argument passed is the prefix at the start of the SSM parameter name, before the first /.
• The must_exist argument specifies whether to error if AWS parameter store cannot be accessed, or if the requested key does not exist.
• The only_these_keys argument specifies a limited set of configuration keys. They are provided as (section, key) tuples. If provided, the config source will only act when these parameters are requested. This prevents unnecessary AWS API calls, which slow down configuration setup, for config values which you know are not available from parameter store. Setting only_these_keys=None (the default) will check parameter store for all config keys.
• The batch_preload argument specifies whether to make GetParametersByPath AWS API calls when the source is constructed. Only applicable when only_these_keys is supplied. If only_these_keys is None, this argument is ignored. This is an optimisation which means that everything supplied in only_these_keys can be loaded in far fewer API calls, thus speeding up configuration loading.

Contributing

Ideas for new features and pull requests are welcome. PRs must come with tests included. This was developed using Python 3.7 but Travis tests run with v3.6 too.

Development setup

1. Clone the git repository
2. Create a virtual environment virtualenv venv
3. Activate the environment venv/scripts/activate
4. Install development dependencies pip install -r requirements.txt

Running tests

pytest

To run with coverage:

pytest --cov

Making a release

1. Bump version number in typedconfig_awssource/__version__.py
2. Add changes to CHANGELOG.md
3. Commit your changes and tag with git tag -a 0.1.0 -m "Summary of changes"
4. Travis will deploy the release to PyPi for you.

Staging release

If you want to check how a release will look on PyPi before tagging and making it live, you can do the following:

1. pip install twine if you don't already have it
2. Bump version number in typedconfig_awssource/__version__.py
3. Clear the dist directory rm -r dist
4. python setup.py sdist bdist_wheel
5. twine check dist/*
6. Upload to the test PyPI twine upload --repository-url https://test.pypi.org/legacy/ dist/*
7. Check all looks ok at https://test.pypi.org/project/typed-config-aws-sources
8. If all looks good you can git tag and push for deploy to live PyPi

Here is a good tutorial on publishing packages to PyPI.