dynaconf-aws-loader 0.5.0

A custom loader for Dynaconf that uses AWS Systems Manager Parameter Store as a source of truth.

When configured, this loader will permit Dynaconf to query AWS Systems Manager Parameter Store for slash-delimited hierarchical configuration data.

Loader Configuration

An example:

from dynaconf import Dynaconf

settings = Dynaconf(
    environments=True,
    settings_file="settings.toml",
    LOADERS_FOR_DYNACONF=[
        "dynaconf_aws_loader.loader",
        "dynaconf.loaders.env_loader"
    ],
)

Note that for the basic functioning of this loader, the environments option for Dynaconf must be set, and an environment must be used.

Configuration Variables

Both of the following configuration values should be set in the environment to avoid a chicken/egg scenario for initializing this custom loader:

• SSM_PARAMETER_PROJECT_PREFIX_FOR_DYNACONF: Required. The project prefix in the parameter store path. For example, if the parameter hierarchy looks something like /baldur/development/database_uri, then in this case SSM_PARAMETER_PROJECT_PREFIX_FOR_DYNACONF=baldur.

• SSM_PARAMETER_NAMESPACE_FOR_DYNACONF: Optional. This provides an additional level of grouping once the project and environment have been determined. For example, if the parameter hierarchy looks something like /baldur/pr-123/development/database_uri, then SSM_PARAMETER_NAMESPACE_FOR_DYNACONF=pr-123.

Note

If a namespace is utilized, be aware that namespaced settings will be merged with non-namespaced settings. This merge is a naive one, where namespaced settings will completely overwrite non-namespaced settings with the same key.

The following optional variables should be set in your settings.toml (or equivalent format), if desired:

• SSM_ENDPOINT_URL_FOR_DYNACONF: If your AWS SSM uses a different endpoint than the AWS default. This can be useful for local development when you are running something like LocalStack.

• SSM_SESSION_FOR_DYNACONF: If you require custom boto3.session.Session arguments, you can specify then as a dictionary here. Note that this will override the default boto3 credential configuration.

• SSM_LOAD_DEFAULT_ENV_FOR_DYNACONF: Boolean, defaults to True. If you want the SSM loader to load keys under the default environment name. The key name itself can be set via the Dynaconf setting of DEFAULT_ENV_FOR_DYNACONF if you want it to be something other than default.

Parameter Store Details

The structure that this loader expects from the path-based organization in SSM is:

/<project-name>/<environment>/<parameter-name>

An optional namespace can be specified as a sub-project grouping for parameters:

/<project-name>/<environment>/<namespace>/<parameter-name>

Note that if you choose to use a namespace identifier, it must not conflict with existing environment identifiers.

If SSM_LOAD_DEFAULT_ENV_FOR_DYNACONF is set to True (which is the default value), the loader will add whatever the value of DEFAULT_ENV_FOR_DYNACONF as an environment key to load from SSM. The typical use case here is to have a default value for all environments that can be overriden on a per-environment basis as necessary.

Security Considerations

For this loader to function correctly and securely, the use of IAM policies to restrict which parameters can be read/mutated is highly encouraged.

Policies can be enacted on a glob-path basis, which will ensure that the only parameters that can be fetched/hydrated into the local object instance are the ones that the current environment is permitted to load.

The following policy for a fictional account allows a user to call the DescribeParameters and GetParameters API operations for parameters that begin with the path /testapp/production:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ssm:DescribeParameters"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ssm:GetParameters"
            ],
            "Resource": "arn:aws:ssm:us-east-1:000000000000:parameter/testapp/production*"
        }
    ]
}

Warning

If a user has access to a path, then the user can access all levels of that path. For example, if a user has permission to access path /testapp, then the user can also access testapp/production. Even if a user has explicitly been denied access in IAM for parameter /testapp/production, they can still call the GetParametersByPath API operation recursively for /testapp and view /testapp/production.

Setting Parameters via Boto3

Parameters may be set via the AWS Web Console UI, or one of their many client libraries. The boto3 library is perhaps the most well-known, and the process is relatively straightforward:

import boto3
ssm_client = boto3.client("ssm")

ssm_client.put_parameter(
    Name="/testapp/development/database/host",
    Value="localhost",
    Type="String",
)

ssm_client.put_parameter(
    Name="/testapp/production/database/password",
    Value="sekrit",
    Type="SecureString",
)

ssm_client.put_parameter(
    Name="/testapp/production/database/host",
    Value="db.example.com",
    Type="String",
)

ssm_client.put_parameter(
    Name="/testapp/production/admin_email",
    Value="[e-mail]",
    Type="String",
)

This creates a parameter hierarchy with the following structure:

{
    "testapp": {
        "development": {"database": {"host": "localhost"}},
        "production": {
            "database": {"host": "db.example.com", "password": "sekrit"},
            "admin_email": "[e-mail]",
        },
    },
}

Parameter Name Limitations

AWS SSM has the following key (and thus path) limitations:

• Parameter names are case sensitive

• A parameter name must be unique within an Amazon Web Services Region

• A parameter name can’t be prefixed with “aws” or “ssm” (case-insensitive)

• Parameter names can include only the following symbols and letters: a-zA-Z0-9_.-

• The slash character ( / ) is used to delineate hierarchies in parameter names

• A parameter name can’t include spaces

• Parameter hierarchies are limited to a maximum depth of fifteen levels

Testing

0. Have Docker installed and running

1. Clone this repository

2. Ensure you have poetry available on your system

3. poetry run pytest

The test suite will spin up an ephemeral Docker container; it may take a few seconds for it to load. The relevant test fixtures will handle setting parameters and their values in the Localstack SSM service.

TODO

• [ ] CI configuration for matrix-based python/dynaconf version testing

• [ ] Handle Parameter Store references to AWS Secrets Manager

• [ ] Make tests/docker-compose.yml more configurable, e.g. ports, in case a different Localstack container is already running for the user