confight 2.0

Common config loading for Python and the command line

confight

One simple way of parsing configs

• Extensible "Unix-like" conf.d directory
• Allow for multiple formats (toml, json, yaml, ini)
• Full unicode support
• User settings ~/.config support
• Nice out-of-the-box defaults
• See examples

confight focuses on making application configuration easy to load, change, extend, generate and audit. It does so by allowing to use separated files for different topics, simplifying changes and new additions, without messing with already existing defaults or deleting or moving protected files.

This is achieved by using at least one config file (/etc/app/config.toml) and an extra directory (/etc/app/conf.d) for extra files.

Those extra files are called droplets which consist in is a small config file that is "dropped" into a conf.d directory where they will be parsed and merged nicely into a single final configuration.

This approach is very common in Unix and used in several applications such as:

• cron (/etc/cron.d)
• bash profiles (/etc/profile.d)
• apt (/etc/apt/sources.list.d)
• systemd
• and many others.

The idea is to "map reduce" configurations, by parsing all files in order, giving more relevance to the latest appearance and then merge them into a single config that holds all the data:

 C₀ -- parse -----|
    C₁ -- parse --|
    C₂ -- parse --|-- merge --> C
       ⋮          |
    Cₙ -- parse --|

The name of those files will determine the order in which they're parsed and the priority their values will have when merging. The last one wins.

1. /etc/app/config.toml
2. /etc/app/conf.d/*
3. ~/.config/app/config.toml
4. ~/.config/app/conf.d/*

Is specially good for externally managed configs or debian-packaged applications, avoiding clashes between installed files and generated configs, for changes that would stay forever unless manually merged.

Usage

>>> import confight
>>> confight.load_app('myapp')
{
    "section": {
        "key": "value"
    }
}

The previous fragment got all the config files at /etc/myapp/config.toml and within the /etc/myapp/conf.d directory and merged them into a single config.

# /etc/myapp/config.toml    /etc/myapp/conf.d/00_first.json    /etc/myapp/conf.d/99_second.ini
[section]                   {                                  [section]
key = "base config"           "section": {                     key = value
                                 "key": "not this"
                              }
                            }

Default file locations for an application named myapp would be at:

• /etc/myapp/config.toml
• /etc/myapp/conf.d/*

User custom configurations would be read (if any) from:

• ~/.config/myapp/config.toml
• ~/.config/myapp/conf.d/*

See the examples section for more information on how to use these functions.

Loading

The load family of functions take a list of names, files or directories to easily parse and merge a related set of configurations:

confight.load_app('myapp')
confight.load_user_app('myapp')
confight.load_paths(['/path/to/config', '/path/to/dir'])
confight.load(['/path/to/config.toml', '/path/to/dir/droplet.toml'])

Each function offers different parameters to improve the ease of use.

The extension of the configuration file can be given with the extension parameter. For instance, load_app('myapp', extension='json') would look for the /etc/myapp/config.json file.

All files in the conf.d directory are read by default regardless the extension. To enforce that only .extension files are read, add the force_extension flag.

Formats

Some formats are builtin in the default installation and some others are optional and must be declared when installing confight.

The list of builtin file formats:

• toml (default)
• json
• ini

The list of optional file formats:

• yaml
• hcl

In order to install confight with optional formats see installation with optional features.

Parsing

Given a path to an existing configuration file, it will be loaded in memory using basic types (string, int, float, list, dict).

The given file can be in one of the allowed formats. For a complete list see the confight.FORMATS list.

confight.parse('/path/to/config', format='toml')

When no format is given, it tries to guess by looking at file extensions:

confight.parse('/path/to/config.json')  # will gess json format

You can see the list of all available extensions at confight.FORMAT_EXTENSIONS.

A custom parsing can be provided by passing a parser function to the load family of functions, matching the signature:

def parser(path, format=None)

The function takes a filesystem path and a format and the result should be a single dictionary with all the loaded data. When format is None the parser is expected to guess it.

Merging

Given a list of parsed configs in order, merge them into a single one. For values that appears several times, the last one wins.

Sections and subsections are recursively merged, keeping all keys along the way and overriding the ones in more than one file with the latest appearance.

A custom merging can be provided by passing a merger function to the load family of functions, matching the signature:

def merger(configs)

The function takes a list of dictionaries containing the parsed configuration in ascending order of priority. It should return a single dictionary with all the configuration.

Finding configs

The default behaviour is that all files at the conf.d directory will be opened, in lexicographical order, and parsed.

A custom config locator can be provided by passing a finder function to the load family of functions, matching the signature:

def finder(path)

The function takes a filesystem path (a conf.d directory supposedly) and returns a list of paths to config files in the desired order of parsing and merging, this is from less to more priority for their values.

Examples

Load application config from the default locations by using the load_app function which will look by default at the /etc/myapp/config.toml and configuration directory at /etc/myapp/conf.d:

# /etc/myapp/config.toml    # /etc/myapp/conf.d/production.toml
user = myapp                password = aX80@klj
password = guest

>>> confight.load_app('myapp')
{
  "user": "myapp",
  "password": "aX80@klj"
}

Allow the user to override the default value when wanting to use a different configuration. When None is given, the default is used:

import argparse
import confight

parser = argparse.ArgumentParser()
parser.add_argument('--config', default=None)
parser.add_argument('--config-dir', default=None)
args = parser.parse_args()

config = confight.load_app('myapp',
                           file_path=args.config,
                           dir_path=args.config_dir)

If the application supports user configuration the function load_user_app might come handy as it will first load the regular app config and then the one defined in the user directory ~/.config/myapp/config.toml and ~/.config/myapp/conf.d/*:

# /etc/myapp/config.toml      # ~/.config/myapp/conf.d/mysettings.toml
url = http://teg.avature.net  password = Avature123!

>>> confight.load_user_app('myapp')
{
  "url": "http://teg.avature.net",
  "password": "Avature123!"
}

To ignore config file extensions, set a format and all files will be parsed using such:

# /etc/myapp/config.toml      # /etc/myapp/config.d/extra
name = test                   name = erebus

>>> confight.load_app('myapp', format='toml')
{
    "name": "erebus"
}

To load configs from a dev or debug location use the prefix option. This will change the base to calculate default paths.

# Loads from ./configs/config.toml and ./configs/config.d/*
>>> confight.load_app('myapp', prefix='./configs')

The user_prefix option can be used altogether for user config files:

# Loads from regular places and ./user/config.toml and ./user/config.d/*
>>> confight.load_user_app('myapp', user_prefix='./user')

Added in version 1.0

Command line

confight allows to inspect configuration from the command line.

By using the confight command it would load the myapp configuration from it's default places and display the output in toml format:

confight show myapp

This allows to preview the resulting config for an application after all merges have been resolved. It can come handy when figuring out what the application has loaded or to debug complex config scenarios.

By passing the --verbose INFO interesting data such as all visited files is listed.

Added in version 0.3

Command line options

usage: confight [-h] [--version] [-v {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                {show} ...

One simple way of parsing configs

positional arguments:
{show}

optional arguments:
-h, --help            show this help message and exit
--version             show program's version number and exit
-v {DEBUG,INFO,WARNING,ERROR,CRITICAL}, --verbose {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        Logging level default: ERROR ['DEBUG', 'INFO',
                        'WARNING', 'ERROR', 'CRITICAL']

Installation

Install it via pip using:

pip install confight

Also with yaml support using optional features:

pip install confight[yaml]

Similarly, for hcl support:

pip install confight[hcl]

Development

Run application tests

tox

Install the application and run tests in development:

pip install -e .
python -m pytest

Changelog

• 2.0 (2023-12-13)

  • [dd040ae2] feat(confight): Only support Python v3.8+
  • [266de3d4] chore(confight): Drop support for Python v2

• 1.4.0 (2023-12-12)

  [ Federico Fapitalle ]

  • [3e618f3b] feat: adds support for HCL languaje

  [ Frank Lenormand ]

  • [a9b3b9a2] fix(confight): Stick to older ruamel.yaml API
  • [4de6a3ff] fix(setup): Do not freeze pyhcl package version

• 1.3.2 (2023-10-25)

  • [241dca89] Update ownership to Platform Operability
  • [96a1def0] fix(setup): Freeze version of ruamel.yaml

• 1.3.1 (2020-04-28)

  • [f6183081] Adds tests for ignore load file behaviour
  • [9686602f] Import pkg_resources only when needed
  • [7e7cbe5c] Update README with optional dependencies
  • [a766b09f] Pin hamcrest version to be python3.5 compatible

• 1.3 (2019-09-19)

  • [995d00f6] Adds a default config parameter for the load_app family of functions
  • [a99f3604] Deprecate support for Python 3.4
  • [f73a8931] Add setuptools to dependencies

• 1.2.3 (2019-02-25)

  • [c1f5c919] Show missing file warnings as debug info

• 1.2.2 (2019-02-19)

  • [7344c929] Fixes man generation in debian rules

• 1.2.1 (2019-02-19)

  • [491f8b05] Fixes find path expansion

• 1.2 (2019-02-14)

  • [3c266c8d] Force all loaded files to have the same extension

• 1.1.1 (2019-01-31)

  [ javier.lasheras ]

  • [a1646871] OrderedDict for yaml too

• 1.1 (2019-01-29)

  • [4a5920af] Adds pypi version badge to README
  • [59c47a5e] Drops support for Python 3.3 and Python 3.4
  • [dfa9c436] Adds support for Python 3.7
  • [6979074d] Fix manpage generation
  • [8f6b58f5] Create a parser with ExtendedInterpolation
  • [7d74246d] Avoid DeprecationWarnings
  • [633b1571] Ordered dicts everywhere

• 1.0 (2018-06-26)

  • [736a6493] Adds prefix and user_prefix options
  • [023158e5] Adds --prefix and --user-prefix cli options
  • [f395fc44] Adapt tests to run in python 3.3 and 3.4
  • [a144dab1] Update package metadata

• 0.3 (2018-06-14)

  • [a7b46ef1] Adds travis config file
  • [5f625da9] Add tox-travis integration
  • [1b678173] Adds confight command line tool
  • [691e042a] Adds cli unit tests

• 0.2.2 (2018-04-13)

  • [3322a7a4] Allow custom file extensions when format is defined

• 0.2.1 (2018-04-09)

  • [93cd8a1c] Update README

• 0.2 (2018-04-04)

  • [63d55fa8] Add Yaml support

• 0.1.1 (2018-04-03)

  • [80087037] Allows to pass extra paths in load functions

• 0.1.0 (2018-03-27)

  • [23927421] Reorganize pretty functions and find behaviour
  • [fade6dd0] Adds debian packaging
  • [c818857a] Update README

• 0.0.1 (2018-03-27)

  • Initial release.