confp 1.0.1

Configuration management using Jinja2 templates and pluggable backends

confp

A configuration management tool, similar to confd using Python and the Jinja2 templating language.

Configuration files are created as Jinja2 templates, pulling values from one or more backends, and can be run continuously as a daemon or as a single-execution application.

Currently confp supports pulling data from the following backends:

• Environment variables
• Redis
• etcd
• Terraform state on S3

Installation

pipx install confp

Or with pip:

pip install confp

Some backends require additional dependencies. Install them as extras:

pipx install confp[redis]      # For the Redis backend
pipx install confp[etcd]       # For the etcd backend
pipx install confp[terraform]  # For the Terraform S3 backend

Multiple extras can be combined:

pipx install confp[redis,etcd]

To add extras to an existing pipx installation, use --force and include all the extras you need (this replaces the existing spec):

pipx install --force "confp[redis,etcd]"

The same extras syntax works with pip (pip install confp[redis], etc.).

The environment variables backend requires no additional dependencies.

Configuration

Configuration of confp is done in one YAML file. The file contains a backends section which handles the retrieval of configuration values used in templates from the backend(s), and a templates section which specifies how and where to deploy templates.

It's also possible to use this file for the Python logging configuration. Refer to the logging section in config.example.yml for an example of this.

backends section

Each of the keys in the backends dictionary is your name for a backend. They must contain a type key, plus whichever configuration keys the specific backend requires. For example, the redis backend optionally takes host and port keys (among others).

Another example is the env backend which pulls values from environment variables. It optionally takes a prefix key which all of the values will have at the beginning.

Here's an example using both:

backends:
  my_redis:
    type: redis
    host: redis.example.com
    port: 6379
  my_env:
    type: env
    prefix: MY_SERVICE_  # Will be removed when specifying keys in templates

templates section

The templates list specifies one or more templates to render. Each of the dictionaries must contain values for the src and dest keys. You may optionally specify the following:

• owner - The owner of the rendered template (only when running as root).
• mode - The file mode of the rendered template.
• check_cmd - The command to run in order to check that the rendered template is valid. Exit code 0 means OK, >0 means it's invalid and we'll roll back to the existing version of the template.
• restart_cmd - The command to run which will either restart the daemon, or tell it to reload its config.

You may also include a vars key, which contains a dictionary where keys are the names for global values within templates and the value dict contains backend and key to specify where its value comes from:

templates:
  - src: /templates/nginx-mysite.conf.j2
    dest: /etc/nginx/sites-available/mysite.conf
    owner: nginx
    mode: '0664'
    check_cmd: /usr/sbin/nginx -t -c {{ dest }}
    restart_cmd: /usr/sbin/service nginx reload
    vars:
      FQDN:
        backend: my_redis  # The name we gave to our redis backend
        key: server/fqdn
      WWW_ROOT:
        backend: my_env
        key: WWW_ROOT  # Pulls the value from env var MY_SERVICE_WWW_ROOT (see prefix above)
        default: /var/www  # Fallback in case MY_SERVICE_WWW_ROOT env var isn't set

Templates

These are standard Jinja2 templates which define your config files and where to pull the variables from. You may pull values from one or more backends within the template, or leave the source of the values up to the configuration (see the template vars section above).

In order to pull values from a specific backend, use the syntax {{ my_redis('server/fqdn') }} where my_redis is the name you gave your backend and the server/fqdn value is the key.

Some backends such as env create a global variable called <backend name>__all containing a dict of all matching variables. In the case of the my_env backend name the global would be called my_env__all. This is useful in order to iterate through all vars in a template.

To use values which have been set globally with the above vars configuration, simply use the name you assigned such as {{ FQDN }}.

Here's an example using both methods:

upstream backend {
    server {{ my_redis('backend/server/host') }}:{{ my_redis('backend/server/port') }};
}
server {
    listen 80;
    {#
        The second argument to the functions optionally sets a default value
        in case the key doesn't exist on the backend. Omitting the default will
        cause a missing key to raise an exception.
    #}
    server_name {{ my_env('FQDN', 'www.example.com') }};
    return 301 https://$server_name$request_uri;
}
server {
    listen 443 ssl;
    server_name {{ my_redis('server/fqdn', 'www.example.com') }};

    ssl_certificate /etc/letsencrypt/live/{{ FQDN }}/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/{{ FQDN }}/privkey.pem;

    root {{ WWW_ROOT }};

    location / {
        proxy_pass http://backend/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Scheme $scheme;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

All Jinja2 features such as conditionals and loops are supported.