Skip to main content

A library to load configuration parameters from multiple sources and formats

Project description

python-configuration

A library to load configuration parameters from multiple sources and formats

version python wheel license build codecov Documentation Status

This library is intended as a helper mechanism to load configuration files hierarchically. Current format types are:

  • Python files
  • Dictionaries
  • Environment variables
  • Filesystem paths
  • JSON files
  • INI files

and optionally

  • YAML files
  • TOML files

Installing

To install the library:

pip install python-configuration

To include the optional TOML and/or YAML loaders, install the optional dependencies toml and yaml. For example,

pip install python-configuration[toml, yaml]

Getting started

This library converts the config types above into dictionaries with dotted-based keys. That is, given a config cfg from the structure

{
    'a': {
        'b': 'value'
    }
}

we are able to refer to the parameter above as any of

cfg['a.b']
cfg['a']['b']
cfg['a'].b
cfg.a.b

and extract specific data types such as dictionaries:

cfg['a'].as_dict == {'b': 'value'}

This is particularly useful in order to isolate group parameters. For example, with the JSON configuration

{
  "database.host": "something",
  "database.port": 12345,
  "database.driver": "name",
  "app.debug": true,
  "app.environment": "development",
  "app.secrets": "super secret",
  "logging": {
    "service": "service",
    "token": "token",
    "tags": "tags"
  }
}

one can retrieve the dictionaries as

cfg.database.as_dict()
cfg.app.as_dict()
cfg.logging.as_dict()

or simply as

dict(cfg.database)
dict(cfg.app)
dict(cfg.logging)

Configuration

There are two general types of objects in this library. The first one is the Configuration, which represents a single config source. The second is a ConfigurationSet that allows for multiple Configuration objects to be specified.

Single Config

Python Files

To load a configuration from a Python module, the config_from_python can be used. The first parameter must be a Python module and can be specified as an absolute path to the Python file or as an importable module.

Optional parameters are the prefix and separator. The following call

config_from_python('foo.bar', prefix='CONFIG', separator='__')

will read every variable in the foo.bar module that starts with CONFIG__ and replace every occurrence of __ with a .. For example,

# foo.bar
CONFIG__AA__BB_C = 1
CONFIG__AA__BB__D = 2
CONF__AA__BB__D = 3

would result in the configuration

{
    'aa.bb_c': 1,
    'aa.bb.d': 2,
}

Note that the single underscore in BB_C is not replaced and the last line is not prefixed by CONFIG.

Dictionaries

Dictionaries are loaded with config_from_dict and are converted internally to a flattened dict.

{
    'a': {
        'b': 'value'
    }
}

becomes

{
    'a.b': 'value'
}

Environment Variables

Environment variables starting with prefix can be read with config_from_env:

config_from_env(prefix, separator='_')

Filesystem Paths

Folders with files named as xxx.yyy.zzz can be loaded with the config_from_path function. This format is useful to load mounted Kubernetes ConfigMaps or Secrets.

JSON, INI, YAML, TOML

JSON, INI, YAML, TOML files are loaded respectively with config_from_json, config_from_ini, config_from_yaml, and config_from_toml. The parameter read_from_file controls whether a string should be interpreted as a filename.

Caveats

In order for Configuration objects to act as dict and allow the syntax dict(cfg), the keys() method is implemented as the typical dict keys. If keys is an element in the configuration cfg then the dict(cfg) call will fail. In that case, it's necessary to use the cfg.as_dict() method to retrieve the dict representation for the Configuration object.

The same applies to the methods values() and items().

Configuration Sets

Configuration sets are used to hierarchically load configurations and merge settings. Sets can be loaded by constructing a ConfigurationSet object directly or using the simplified config function.

To construct a ConfigurationSet, pass in as many of the simple Configuration objects as needed:

cfg = ConfigurationSet(
    config_from_env(prefix=PREFIX),
    config_from_json(path, read_from_file=True),
    config_from_dict(DICT),
)

The example above will read first from Environment variables prefixed with PREFIX, and fallback first to the JSON file at path, and finally use the dictionary DICT.

The config function simplifies loading sets by assuming some defaults. The example above can also be obtained by

cfg = config(
    ('env', PREFIX),
    ('json', path, True),
    ('dict', DICT),
)

or, even simpler if path points to a file with a .json suffix:

cfg = config('env', path, DICT, prefix=PREFIX)

The config function automatically detects the following:

  • extension .py for python modules
  • dot-separated python identifiers as a python module (e.g. foo.bar)
  • extension .json for JSON files
  • extension .yaml for YAML files
  • extension .toml for TOML files
  • extension .ini for INI files
  • filesystem folders as Filesystem Paths
  • the strings env or environment for Environment Variables
Caveats

As long as the data types are consistent across all the configurations that are part of a ConfigurationSet, the behavior should be straightforward. When different configuration objects are specified with competing data types, the first configuration to define the elements sets its datatype. For example, if in the example above element is interpreted as a dict from environment variables, but the JSON file specifies it as anything else besides a mapping, then the JSON value will be dropped automatically.

Developing

To develop this library, download the source code and install a local version.

Features

  • Load multiple configuration types
  • Hierarchical configuration
  • Ability to override with environment variables
  • Merge parameters from different configuration types

Contributing

If you'd like to contribute, please fork the repository and use a feature branch. Pull requests are welcome.

Links

Licensing

The code in this project is licensed under MIT license.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

python-configuration-0.4.0.tar.gz (12.2 kB view details)

Uploaded Source

Built Distribution

python_configuration-0.4.0-py3-none-any.whl (28.6 kB view details)

Uploaded Python 3

File details

Details for the file python-configuration-0.4.0.tar.gz.

File metadata

  • Download URL: python-configuration-0.4.0.tar.gz
  • Upload date:
  • Size: 12.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.0.0a1 CPython/3.7.4 Linux/5.3.5-arch1-1-ARCH

File hashes

Hashes for python-configuration-0.4.0.tar.gz
Algorithm Hash digest
SHA256 40010aecddc84b028836c976452f1802f71af00f0cca4b9986cb9e8fe0b0002b
MD5 5b1a7011302e78d5934f0bcffa3a3221
BLAKE2b-256 76a14e51cd60a919f524f43e15313a9d42b79c56af934a7d1f963850475587bf

See more details on using hashes here.

File details

Details for the file python_configuration-0.4.0-py3-none-any.whl.

File metadata

  • Download URL: python_configuration-0.4.0-py3-none-any.whl
  • Upload date:
  • Size: 28.6 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.0.0a1 CPython/3.7.4 Linux/5.3.5-arch1-1-ARCH

File hashes

Hashes for python_configuration-0.4.0-py3-none-any.whl
Algorithm Hash digest
SHA256 93f3683e2756cd4a0122dc49289e2a85925aae1b9b242ebd6a5fa0bc8dcc3bfe
MD5 fd0b07d55859fb6bd27f2b71c0eb1a66
BLAKE2b-256 f9c1ee3c46729fa3196316ceab209fd9dd152397997bd594b5d2b6facb426d9a

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page