Skip to main content

The easiest way to manage configuration files in Python

Project description

configzen vv0.8.0

configzen – easily create and maintain complex, statically-typed configurations with validation in Python.

It's important to keep your configuration safe and consistent. Give a shot to configzen 🚀

⭐ Supports Python 3.8 or above,
⭐ Is fully typed,
⭐ Supports YAML, JSON, TOML, INI, XML, ConfigObj, BSON, CBOR, Amazon Ion, properties and shellvars,
⭐ Supports reading and writing configuration files, fully or partially, with the ability to preserve the original file structure (without comments[^1]),
⭐ Supports configuration preprocessing (extending, including and copying configuration files without the need to change the code),
⭐ Supports variable interpolation (runtime value substitution),
⭐ Supports modular configuration with type validation (wrapping runtime Python modules in-place and outside them),
⭐ Supports synchronous and asynchronous file operations,
⭐ Supports loading configuration from environment variables and secret files.

While being built on top of pydantic, configzen inherits most of its features, including data validation, schema generation, custom data types, good integration with Rich, and more.

Learn more below.

Features

Managing content

Having a YAML configuration file like this:

# database.yml
host: 127.0.0.1
port: 5432
user: postgres

You can create a configzen configuration model for it like this:

# model.py
from ipaddress import IPv4Address, IPv6Address

from configzen import ConfigField, ConfigMeta, ConfigModel


class DatabaseConfig(ConfigModel):
    host: IPv4Address | IPv6Address
    port: int
    user: str
    password: str = ConfigField(exclude=True)

    class Config(ConfigMeta):
        resource = "database.yml"
        env_prefix = "DB_"


db_config = DatabaseConfig.load()

And you can load your configuration from a file as well as from the environment variables DB_HOST, DB_PORT, DB_USER and DB_PASSWORD. Since password is a field created with the option exclude=True, it will not be included in the configuration's exported data: that guarantees that your password does never leak into database.yml on save – but you may still pass it through an environment variable (here – the mentioned DB_PASSWORD). Secret files are also supported, see the pydantic documentation for more information.

pydantic will naturally take care of parsing and validating the loaded data. Configuration models inherit from the pydantic.BaseSettings class, so you can use all of its features: schema generation, type conversion, validation, etc.

There are additional features brought to you by configzen worth checking out, though.

You can use the db_config object defined above to access the configuration values:

>>> db_config.host
IPv4Address('127.0.0.1')

modify them, if the pydantic model validation allows it (<Your model>.Config.validate_assignment will be True by default):

>>> db_config.host = "0.0.0.0"
>>> db_config.host
IPv4Address('0.0.0.0')

as well as reload particular values, without touching the rest of the configuration:

>>> db_config.at("port").reload()
5432
>>> db_config
DatabaseConfig(host=IPv4Address('0.0.0.0'), port=5432, user='postgres', password='password')
>>> db_config.at("host").reload()
IPv4Address('127.0.0.1')
>>> db_config
DatabaseConfig(host=IPv4Address('127.0.0.1'), port=5432, user='postgres', password='password')

or reload the whole configuration:

>>> db_config.port = 1234
>>> db_config.reload()
DatabaseConfig(host=IPv4Address('127.0.0.1'), port=5432, user='postgres', password='password')

or save a particular value, without touching the rest of the configuration:

>>> db_config.host = "0.0.0.0"
>>> db_config.port = 443
>>> db_config
DatabaseConfig(host=IPv4Address('0.0.0.0'), port=443, user='postgres', password='password')
>>> db_config.at("host").save()
40
>>> db_config.reload()
DatabaseConfig(host=IPv4Address('0.0.0.0'), port=5432, user='postgres', password='password')

or save the whole configuration:

>>> db_config.save()
39

Preprocessing

To see supported preprocessing directives, see Supported preprocessing directives.

Basic usage

Having a base configuration file like this (base.json):

{
  "i18n": {
    "language": "en",
    "timezone": "UTC"
  },
  "app": {
    "debug": true,
    "expose": 8000
  }
}

create another configuration file like this, overriding desired sections as needed:

# production.yml
^extend: base.json

+app:
  debug: false

and load the production.yml configuration file. No explicit changes to the code indicating the use of the base.json file are needed.

Note: Using + in front of a key will update the section already defined at that key, instead of overwriting it entirely.

Notice how configuration file formats do not matter in configzen: you can extend JSON configurations with YAML, but that might be as well any other format among the supported ones (see the Supported file formats section).

The above example is equivalent to as if you used:

# production.yml
i18n:
  language: en
  timezone: UTC
app:
  debug: false
  expose: 8000

but with a significant difference: when you save the above configuration, the ^extend relation to the base configuration file base.json is preserved. This basically means that changes made in the base configuration file will apply to the configuration model instance loaded from the ^extend-ing configuration file. Any changes made locally to the model will result in + sections being automatically added to the exported configuration data.

Supported preprocessing directives

Directive Is the referenced file preprocessed? Is the directive preserved on export?
^extend Yes Yes
^include Yes No
^copy No No

Interpolation

Basic interpolation

You can use interpolation in your configuration files:

cpu:
  cores: 4
num_workers: ${cpu.cores}
>>> from configzen import ConfigModel
...
>>> class CPUConfig(ConfigModel):
...     cores: int
...
>>> class AppConfig(ConfigModel):
...     cpu: CPUConfig
...     num_workers: int
...
>>> app_config = AppConfig.load("app.yml")
>>> app_config
AppConfig(cpu=CPUConfig(cores=4), num_workers=4)

Reusable configuration with namespaces

You can share independent configuration models as namespaces through inclusion:

# database.yml
host: ${app_config::db_host}
port: ${app_config::expose}
# app.yml
db_host: localhost
expose: 8000
>>> from configzen import ConfigModel, include
>>> from ipaddress import IPv4Address
>>>
>>> @include("app_config")
... class DatabaseConfig(ConfigModel):
...     host: IPv4Address
...     port: int
...
>>> class AppConfig(ConfigModel):
...     db_host: str
...     expose: int
...
>>> app_config = AppConfig.load("app.yml")
>>> app_config
AppConfig(db_host='localhost', expose=8000)
>>> db_config = DatabaseConfig.load("database.yml")
>>> db_config
DatabaseConfig(host=IPv4Address('127.0.0.1'), port=8000)
>>> db_config.dict()
{'host': IPv4Address('127.0.0.1'), 'port': 8000}
>>> db_config.export()  # used when saving
{'host': '${app_config::db_host}', 'port': '${app_config::expose}'}

You do not have to pass a variable name to @include, though. @include lets you overwrite the main interpolation namespace or one with a separate name (here: app_config) with configuration models, dictionaries and their factories.

Modular configuration

Wrapping modules in-place

You can wrap modules in-place with configuration models:

  1. Without writing a model class:
# config.py
from configzen import ConfigModule

# Annotate config fields
HOST: str = "localhost"
PORT: int = 8000

ConfigModule.wrap_this_module()
  1. With a model class:
# config.py
from configzen import ConfigModel

# Annotations are optional
HOST = "localhost"
PORT = 8000

class AppConfig(ConfigModel):
    HOST: str
    PORT: int

AppConfig.wrap_this_module()

Now values HOST and PORT will be validated as str and int data types, respectively:

>>> import config  # <configuration module 'config' from 'config.py'>
>>> config.HOST
'localhost'
>>> config.PORT
8000
>>> config.PORT = "8000"
>>> config.PORT
8000
>>> config.PORT = "abc"
Traceback (most recent call last):
  ...

Wrapping interchangeable modules

You can wrap modules outside them with configuration models:

# setup.py
from configzen import ConfigModel

class AppConfig(ConfigModel):
    HOST: str = "localhost"
    PORT: int = 8000

config_model = AppConfig.wrap_module("config")
# config.py
HOST: str = "0.0.0.0"
PORT: int = 443
>>> from setup import config_model
>>> config_model.HOST
'0.0.0.0'
>>> config_model.PORT
443
>>> config_model.PORT = "8000"
>>> config_model.PORT
8000
>>> import config
>>> config.HOST
'0.0.0.0'
>>> config.PORT
8000

Supported file formats

configzen uses anyconfig to serialize and deserialize data and does not operate on any protocol-specific entities. As an example result, comments in your configuration files are lost on save[^1], but you can exchange file formats without any hassle.

The following table shows the supported file formats, their requirements, file extensions, and the backend libraries used to accomplish this goal.

File Format To use, install: Recognized File Extension(s) Backend Library
JSON - json json (standard library)
INI - ini, cfg, conf configparser (standard library)
TOML - toml toml
YAML - yaml, yml pyyaml / ruamel.yaml
XML - xml xml (standard library)
ConfigObj anyconfig-configobj-backend configobj configobj
BSON anyconfig-bson-backend bson bson
CBOR (RFC 8949) anyconfig-cbor2-backend cbor, cbor2 cbor2
Amazon Ion anyconfig-ion-backend ion ion
CBOR (deprecated, RFC 7049) anyconfig-cbor-backend cbor cbor
properties - properties (native)
shellvars - shellvars (native)

If your file extension is not recognized, you can register your own file extension by calling ConfigAgent.register_file_extension(file_extension, parser_name).

If your favorite backend library is not supported, please let me know by reporting it as an issue. Using custom backends is to be supported in the future.

[^1]: A suggested alternative for comments is to use the description parameter in your configuration models' fields: ConfigField(description=...). The provided field descriptions are included in JSON schemas generated by the default implementation of the ConfigModel.schema() method.

Setup

In order to use configzen in your project, install it with your package manager, for example pip:

pip install configzen

If you are willing to contribute to configzen, which is awesome, simply clone this repository and install its dependencies with poetry:

poetry install --with dev --all-extras

After that, install the pre-commit hooks:

pre-commit install --hook-type pre-commit --hook-type pre-push

You might also need to install required stubs. First, activate your virtual environment:

poetry shell

and run (on Linux or PowerShell 7.0+):

mypy configzen/ || (echo yes | mypy --install-types)

or, if you are using an older version of PowerShell:

mypy configzen/; if (-not $?) { echo yes | mypy --install-types }

Note: Using WSL2 + pyenv (with Python 3.8) for developing configzen is strongly recommended in case you use Windows.

And you are good to go. 🚀

Contributions are welcome! Feel free to open an issue whenever you encounter a bug or have a feature request or submit a pull request with your changes.

License

MIT License

Credits

Author

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

configzen-0.8.0.tar.gz (42.3 kB view hashes)

Uploaded Source

Built Distribution

configzen-0.8.0-py3-none-any.whl (41.3 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page