Skip to main content

Configuration library for Python applications

Project description

Everett is a Python configuration library for your app.

Code:

https://github.com/willkg/everett

Issues:

https://github.com/willkg/everett/issues

License:

MPL v2

Documentation:

https://everett.readthedocs.io/

Goals

Goals of Everett:

  1. flexible configuration from multiple configured environments

  2. easy testing with configuration

  3. easy documentation of configuration for users

From that, Everett has the following features:

  • is composeable and flexible

  • makes it easier to provide helpful error messages for users trying to configure your software

  • supports auto-documentation of configuration with a Sphinx autocomponent directive

  • has an API for testing configuration variations in your tests

  • can pull configuration from a variety of specified sources (environment, INI files, YAML files, dict, write-your-own)

  • supports parsing values (bool, int, lists of things, classes, write-your-own)

  • supports key namespaces

  • supports component architectures

  • works with whatever you’re writing–command line tools, web sites, system daemons, etc

Everett is inspired by python-decouple and configman.

Quick start

Fast start example

You’re writing an app and want it to look for configuration:

  1. in an .env file in the current working directory

  2. then in the process environment

You set up a configuration manager like this:

from everett.manager import ConfigManager

config = ConfigManager.basic_config()

and use it to get configuration values like this:

api_host = config("api_host")

max_bytes = config("max_bytes", default=1000, parser=int)

debug_mode = config("debug", default="False", parser=bool)

When you outgrow that or need different variations of it, you can switch to creating a ConfigManager instance that meets your needs.

More complex example

You’re writing an app and want to pull configuration (in order of prcedence):

  1. from the process environment

  2. from an INI file stored in a place specified by (in order of precedence): 1. MYAPP_INI in the environment 2. ~/.myapp.ini 3. /etc/myapp.ini

First, you need need to install the additional requirements for INI file environments:

pip install everett[ini]

Then set up the ConfigManager:

import os
import sys

from everett.ext.inifile import ConfigIniEnv
from everett.manager import ConfigManager, ConfigOSEnv


def build_config_manager():
    return ConfigManager(
        # Specify one or more configuration environments in
        # the order they should be checked
        environments=[
            # Look in OS process environment first
            ConfigOSEnv(),

            # Look in INI files in order specified
            ConfigIniEnv([
                os.environ.get("MYAPP_INI"),
                "~/.myapp.ini",
                "/etc/myapp.ini"
            ]),
        ],

        # Provide users a link to documentation for when they hit
        # configuration errors
        doc="Check https://example.com/configuration for docs."
    )

Then use it:

def is_debug(config):
    return config(
        "debug", default="False", parser=bool,
        doc="Set to True for debugmode; False for regular mode."
    )

config = build_config_manager()

if is_debug(config):
    print('DEBUG MODE ON!')

Let’s write some tests that verify behavior based on the debug configuration value:

from myapp import get_config, is_debug

from everett.manager import config_override


@config_override(DEBUG="true")
def test_debug_true():
    assert is_debug(get_config()) is True


def test_debug_false():
    with config_override(DEBUG="false"):
        assert is_debug(get_config()) is False

If the user sets DEBUG with a bad value, they get a helpful error message with the documentation for the configuration option and the ConfigManager:

$ DEBUG=foo python myprogram.py
<traceback>
namespace=None key=debug requires a value parseable by bool
Set to True for debugmode; False for regular mode.
Check https://example.com/configuration for docs.

Configuration classes

Everett supports centralizing your configuration in a class. Instead of having configuration-related bits defined across your codebase, you can define it in a class. Let’s rewrite the above example using a configuration class.

First, create a configuration class:

import os
import sys

from everett.component import RequiredConfigMixin, ConfigOptions
from everett.ext.inifile import ConfigIniEnv
from everett.manager import ConfigManager, ConfigOSEnv


class AppConfig(RequiredConfigMixin):
    required_config = ConfigOptions()
    required_config.add_option(
        "debug",
        parser=bool,
        default="false",
        doc="Switch debug mode on and off.")
    )

Then we set up our ConfigManager:

def get_config():
    manager = ConfigManager(
        # Specify one or more configuration environments in
        # the order they should be checked
        environments=[
            # Look in OS process environment first
            ConfigOSEnv(),

            # Look in INI files in order specified
            ConfigIniEnv([
                os.environ.get("MYAPP_INI"),
                "~/.myapp.ini",
                "/etc/myapp.ini"
            ]),
        ],

        # Provide users a link to documentation for when they hit
        # configuration errors
        doc="Check https://example.com/configuration for docs."
    )

    # Apply the configuration class to the configuration manager
    # so that it handles option properties like defaults, parsers,
    # documentation, and so on.
    return manager.with_options(AppConfig())

Then use it:

config = get_config()

if config("debug"):
    print("DEBUG MODE ON!")

Further, you can auto-generate configuration documentation by including the everett.sphinxext Sphinx extension and using the autocomponent directive:

.. autocomponent:: path.to.AppConfig

That has some niceties:

  1. your application configuration is centralized in one place instead of spread out across your code base

  2. you can use the autocomponent Sphinx directive to auto-generate configuration documentation for your users

Everett components

Everett supports components that require configuration. Say your app needs to connect to RabbitMQ. With Everett, you can define the component’s configuration needs in the component class:

from everett.component import RequiredConfigMixin, ConfigOptions


class RabbitMQComponent(RequiredConfigMixin):
    required_config = ConfigOptions()
    required_config.add_option(
        "host",
        doc="RabbitMQ host to connect to"
    )
    required_config.add_option(
        "port",
        default="5672",
        doc="Port to use",
        parser=int
    )
    required_config.add_option(
        "queue_name",
        doc="Queue to insert things into"
    )

    def __init__(self, config):
        # Bind the configuration to just the configuration this
        # component requires such that this component is
        # self-contained
        self.config = config.with_options(self)

        self.host = self.config("host")
        self.port = self.config("port")
        self.queue_name = self.config("queue_name")

Then instantiate a RabbitMQComponent that looks for configuration keys in the rmq namespace:

queue = RabbitMQComponent(config.with_namespace("rmq"))

The RabbitMQComponent has a HOST key, so your configuration would need to define RMQ_HOST.

You can auto-generate configuration documentation for this component in your Sphinx docs by including the everett.sphinxext Sphinx extension and using the autocomponent directive:

.. autocomponent:: path.to.RabbitMQComponent
   :namespace: rmq

Say your app actually needs to connect to two separate queues–one for regular processing and one for priority processing:

from everett.manager import ConfigManager

config = ConfigManager.basic_config()

# Apply the "rmq" namespace to the configuration so all keys are
# prepended with RMQ_
rmq_config = config.with_namespace("rmq")

# Create a RabbitMQComponent with RMQ_REGULAR_ prepended to keys
regular_queue = RabbitMQComponent(rmq_config.with_namespace("regular"))

# Create a RabbitMQComponent with RMQ_PRIORITY_ prepended to keys
priority_queue = RabbitMQComponent(rmq_config.with_namespace("priority"))

In your environment, you provide the regular queue configuration with RMQ_REGULAR_HOST, etc and the priority queue configuration with RMQ_PRIORITY_HOST, etc.

Same component code. Two different instances pulling configuration from two different namespaces.

Components support subclassing, mixins and all that, too.

Install

Install from PyPI

Run:

$ pip install everett

If you want to use the ConfigIniEnv, you need to install its requirements as well:

$ pip install everett[ini]

If you want to use the ConfigYamlEnv, you need to install its requirements as well:

$ pip install everett[yaml]

Install for hacking

Run:

# Clone the repository
$ git clone https://github.com/willkg/everett

# Create a virtualenvironment
$ mkvirtualenv --python /usr/bin/python3 everett
...

# Install Everett and dev requirements
$ pip install -r requirements-dev.txt

Why not other libs?

Most other libraries I looked at had one or more of the following issues:

  • were tied to a specific web app framework

  • didn’t allow you to specify configuration sources

  • provided poor error messages when users configure things wrong

  • had a global configuration object

  • made it really hard to override specific configuration when writing tests

  • had no facilities for auto-generating configuration documentation

History

1.0.3 (October 28th, 2020)

Backwards incompatible changes:

  • Dropped support for Python 3.4. (#96)

  • Dropped support for Python 3.5. (#116)

Fixes:

  • Add support for Python 3.7. (#68)

  • Add support for Python 3.8. (#102)

  • Add support for Python 3.9. (#117)

  • Reformatted code with Black, added Makefile, switched to GitHub Actions.

  • Fix get_runtime_config() to infer namespaces. (#118)

  • Fix RemovedInSphinx50Warning. (#115)

  • Documentation fixes and clarifications.

1.0.2 (February 22nd, 2019)

Fixes:

  • Improve documentation.

  • Fix problems when there are nested BoundConfigs. Now they work correctly. (#90)

  • Add “meta” to options letting you declare additional data on the option when you’re adding it.

    For example, this lets you do things like mark options as “secrets” so that you know which ones to ****** out when logging your configuration. (#88)

1.0.1 (January 8th, 2019)

Fixes:

  • Fix documentation issues.

  • Package missing everett.ext. Thank you, dsblank! (#84)

1.0.0 (January 7th, 2019)

Backwards incompatible changes:

  • Dropped support for Python 2.7. Everett no longer supports Python 2. (#73)

  • Dropped support for Python 3.3 and added support for Python 3.7. Thank you, pjz! (#68)

  • Moved ConfigIniEnv to a different module. Now you need to import it like this:

    from everett.ext.inifile import ConfigIniEnv

    (#79)

Features:

  • Everett now logs configuration discovery in the everett logger at the logging.DEBUG level. This is helpful for trouble-shooting some kinds of issues. (#74)

  • Everett now has a YAML configuration environment. In order to use it, you need to install its requirements:

    $ pip install everett[yaml]

    Then you can import it like this:

    from everett.ext.yamlfile import ConfigYamlEnv

    (#72)

Fixes:

  • Everett no longer requires configobj–it’s now optional. If you use ConfigIniEnv, you can install it with:

    $ pip install everett[ini]

    (#79)

  • Fixed list parsing and file discovery in ConfigIniEnv so they match the docs and are more consistent with other envs. Thank you, apollo13! (#71)

  • Added a .basic_config() for fast opinionated setup that uses the process environment and a .env file in the current working directory.

  • Switching to semver.

0.9 (April 7th, 2017)

Changed:

  • Rewrite Sphinx extension. The extension is now in the everett.sphinxext module and the directive is now .. autocomponent::. It generates better documentation and it now indexes Everett components and options.

    This is backwards-incompatible. You will need to update your Sphinx configuration and documentation.

  • Changed the HISTORY.rst structure.

  • Changed the repr for everett.NO_VALUE to "NO_VALUE".

  • InvalidValueError and ConfigurationMissingError now have namespace, key, and parser attributes allowing you to build your own messages.

Fixed:

  • Fix an example in the docs where the final key was backwards. Thank you, pjz!

Documentation fixes and updates.

0.8 (January 24th, 2017)

Added:

  • Add :namespace: and :case: arguments to autoconfig directive. These make it easier to cater your documentation to your project’s needs.

  • Add support for Python 3.6.

Minor documentation fixes and updates.

0.7 (January 5th, 2017)

Added:

  • Feature: You can now include documentation hints and urls for ConfigManager objects and config options. This will make it easier for your users to debug configuration errors they’re having with your software.

Fixed:

  • Fix ListOf so it returns empty lists rather than a list with a single empty string.

Documentation fixes and updates.

0.6 (November 28th, 2016)

Added:

  • Add RequiredConfigMixin.get_runtime_config() which returns the runtime configuration for a component or tree of components. This lets you print runtime configuration at startup, generate INI files, etc.

  • Add ConfigObjEnv which lets you use an object for configuration. This works with argparse’s Namespace amongst other things.

Changed:

  • Change :show-docstring: to take an optional value which is the attribute to pull docstring content from. This means you don’t have to mix programming documentation with user documentation–they can be in different attributes.

  • Improve configuration-related exceptions. With Python 3, configuration errors all derive from ConfigurationError and have helpful error messages that should make it clear what’s wrong with the configuration value. With Python 2, you can get other kinds of Exceptions thrown depending on the parser used, but configuration error messages should still be helpful.

Documentation fixes and updates.

0.5 (November 8th, 2016)

Added:

  • Add :show-docstring: flag to autoconfig directive.

  • Add :hide-classname: flag to autoconfig directive.

Changed:

  • Rewrite ConfigIniEnv to use configobj which allows for nested sections in INI files. This also allows you to specify multiple INI files and have later ones override earlier ones.

Fixed:

  • Fix autoconfig Sphinx directive and add tests–it was all kinds of broken.

Documentation fixes and updates.

0.4 (October 27th, 2016)

Added:

  • Add raw_value argument to config calls. This makes it easier to write code that prints configuration.

Fixed:

  • Fix listify(None) to return [].

Documentation fixes and updates.

0.3.1 (October 12th, 2016)

Fixed:

  • Fix alternate_keys with components. Previously it worked for everything but components. Now it works with components, too.

Documentation fixes and updates.

0.3 (October 6th, 2016)

Added:

  • Add ConfigManager.from_dict() shorthand for building configuration instances.

  • Add .get_namespace() to ConfigManager and friends for getting the complete namespace for a given config instance as a list of strings.

  • Add alternate_keys to config call. This lets you specify a list of keys in order to try if the primary key doesn’t find a value. This is helpful for deprecating keys that you used to use in a backwards-compatible way.

  • Add root: prefix to keys allowing you to look outside of the current namespace and at the configuration root for configuration values.

Changed:

  • Make ConfigDictEnv case-insensitive to keys and namespaces.

Documentation fixes and updates.

0.2 (August 16th, 2016)

Added:

  • Add ConfigEnvFileEnv for supporting .env files. Thank you, Paul!

  • Add “on” and “off” as valid boolean values. This makes it easier to use config for feature flippers. Thank you, Paul!

Changed:

  • Change ConfigIniEnv to take a single path or list of paths. Thank you, Paul!

  • Make NO_VALUE falsy.

Fixed:

  • Fix __call__ returning None–it should return NO_VALUE.

Lots of docs updates: finished the section about making your own parsers, added a section on using dj-database-url, added a section on django-cache-url and expanded on existing examples.

0.1 (August 1st, 2016)

Initial writing.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

everett-1.0.3.tar.gz (62.3 kB view hashes)

Uploaded source

Built Distribution

everett-1.0.3-py2.py3-none-any.whl (31.8 kB view hashes)

Uploaded py2 py3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page