nested-config 2.1.4

Parse configuration files that include paths to other config files into a single configuration object

nested-config

   GitLab repository   GitHub mirror

If you've ever wanted to have the option of replacing part of a configuration file with a path to another configuration file that contains those sub-parameters, then nested-config might be for you.

nested-config allows you to parse configuration files that contain references to other configuration files using a series of models. If a model includes a nested model as one of its attributes and nested-config finds a string value for that parameter in the configuration file instead of an associative array[^assoc-array], then it assumes that this string is a path to another configuration file that should be parsed and whose contents should replace the string in the main configuration file. If the string appears to be a relative path, it is assumed to be relative to the path of its parent configuration file.

Contents

• Contents
• Basic Usage
• Nomenclature
  • loader
  • model
  • nested model
  • config dict
• API
  • nested_config.expand_config(config_path, model, *, default_suffix = None)
  • nested_config.config_dict_loaders
    • Included loaders
    • Adding loaders
  • Deprecated features in v2.1.0, to be removed in v3.0.0
• Pydantic 1.0/2.0 Compatibility
• Footnotes

Basic Usage

Given the following configuration files /tmp/house.toml and /tmp/tmp2/dimensions.toml:

Figure 1: /tmp/house.toml

name = "my house"
dimensions = "tmp2/dimensions.toml"

Figure 2: /tmp/tmp2/dimensions.toml

length = 10
width = 20

You can expand these into a single dict with the following:

Figure 3: Expand /tmp/house.toml

import nested_config

class Dimensions:
    length: int
    width: int


class House:
    name: str
    dimensions: Dimensions


house_dict = nested_config.expand_config("/tmp/house.toml", House)
print(house_dict)
# {'name': 'my house', 'dimensions': {'length': 10, 'width': 20}}

Note that in /tmp/house.toml, dimensions is not a mapping but is a path to another toml file at a path relative to house.toml.

See tests for more detailed use-cases, such as where the root model contains lists or dicts of other models and when those may be included in the root config file or specified as paths to sub-config files.

Nomenclature

loader

A loader is a function that reads a config file and returns a dict containing the key-value pairs from the file. nested-config includes loaders for JSON, TOML, and (if PyYAML is installed) YAML. For example, the JSON loader looks like this:

import json

def json_load(path):
    with open(path, "rb") as fobj:
        return json.load(fobj)

model

nested-config uses the term model to refer to a class definition that includes annotated attributes. For example, this model, Dimensions, includes three attributes, each of float type, x, y, and z:

class Dimensions:
    x: float
    y: float
    z: float

A model can be decorated as a dataclass or using attrs.define or can subclass pydantic.BaseModel to provide some method for instantiating an object instance of the model but they aren't necessary to use nested-config.

The only criterion for a type to be a model is that is has a __dict__ attribute that includes an __annotations__ member. Note: This does not mean that instances of the model must have a __dict__ attribute. For example, instances of classes with __slots__ and NamedTuple instances may not have a __dict__ attribute.

nested model

A nested model is a model that is included within another model as one of its class attributes. For example, the below model House includes an name of string type, and an attribute dimensions of Dimensions type (defined above). Since Dimensions is a model type, this is an example of a nested model.

class House:
    name: str
    dimensions: Dimensions

config dict

A config dict is simply a dict with string keys such as may be obtained by reading in configuration text. For example reading in a string of TOML text with tomllib.loads returns a config dict.

import tomllib

config = "x = 2\ny = 3"
print(tomllib.loads(config))
# {'x': 2, 'y': 3}

API

nested_config.expand_config(config_path, model, *, default_suffix = None)

This function first loads the config file at config_path into a config dict using the appropriate loader. It then uses the attribute annotations of model and/or any nested models within model to see if any of the string values in the configuration file correspond to a nested model. For each such case, the string is assumed to be a path and is loaded into another config dict which replaces the string value in the parent config dict. This continues until all paths are converted and then the fully-expanded config dict is returned.

Note that all non-absolute string paths are assumed to be relative to the path of their parent config file.

The loader for a given config file is determined by file extension (AKA suffix). If default_suffix is specified, any config file with an unknown suffix or no suffix will be assumed to be of that type, e.g. ".toml". (Otherwise this is an error.) It is possible for one config file to include a path to a config file of a different format, so long as each file has the appropriate suffix and there is a loader for that suffix.

nested_config.config_dict_loaders

config_dict_loaders is a dict that maps file suffixes to loaders.

Included loaders

nested-config automatically loads the following files based on extension:

Format	Extensions(s)	Library
JSON	.json	json (stdlib)
TOML	.toml	tomllib (Python 3.11+ stdlib) or tomli
YAML	.yaml, .yml	pyyaml (extra dependency[^yaml-extra])

Adding loaders

To add a loader for another file extension, simply update config_dict_loaders:

import nested_config
from nested_config import ConfigDict  # alias for dict[str, Any]

def dummy_loader(config_path: Path | str) -> ConfigDict:
    return {"a": 1, "b": 2}

nested_config.config_dict_loaders[".dmy"] = dummy_loader

# or add another extension for an existing loader
nested_config.config_dict_loaders[".jsn"] = nested_config.config_dict_loaders[".json"]

# or use a different library to replace an existing loader
import rtoml

def rtoml_load(path) -> ConfigDict:
    with open(path, "rb") as fobj:
        return rtoml.load(fobj)

nested_config.config_dict_loaders[".toml"] = rtoml_load

Deprecated features in v2.1.0, to be removed in v3.0.0

The following functionality is available only if Pydantic is installed:

• nested_config.validate_config() expands a configuration file according to a Pydantic model and then validates the config dictionary into an instance of the Pydantic model.
• nested_config.BaseModel can be used as a replacement for pydantic.BaseModel to include a from_config() classmethod on all models that uses nested_config.validate_config() to create an instance of the model.
• By importing nested_config, PurePath validators and JSON encoders are added to pydantic in Pydantic 1.8-1.10 (they are included in Pydantic 2.0+)

Pydantic 1.0/2.0 Compatibility

The pydantic functionality in nested-config is runtime compatible with Pydantic 1.8+ and Pydantic 2.0.

The follow table gives info on how to configure the mypy and Pyright type checkers to properly work, depending on the version of Pydantic you are using.

Pydantic Version	mypy config	mypy cli	Pyright config
2.0+	always_false = PYDANTIC_1	--always-false PYDANTIC_1	defineConstant = { "PYDANTIC_1" = false }
1.8-1.10	always_true = PYDANTIC_1	--always-true PYDANTIC_1	defineConstant = { "PYDANTIC_1" = true }

Footnotes

[^yaml-extra]: Install pyyaml separately with pip or install nested-config with pip install nested-config[yaml].

[^assoc-array]: Each language uses one or more names for an associative arrays. JSON calls it an object, YAML calls is a mapping, and TOML calls is a table. Any of course in Python it's a dictionary, or dict.