Parse configuration files that include paths to other config files into a singleconfiguration object
Project description
nested-config
If you've ever wanted to have the option of replacing part of a configuration file with a path to another configuration file that contains those sub-parameters, then nested-config might be for you.
nested-config allows you to parse configuration files that contain references to other configuration files using a series of models. If a model includes a nested model as one of its attributes and nested-config finds a string value for that parameter in the configuration file instead of an associative array[^assoc-array], then it assumes that this string is a path to another configuration file that should be parsed and whose contents should replace the string in the main configuration file. If the string appears to be a relative path, it is assumed to be relative to the path of its parent configuration file.
Contents
Basic Usage
Given the following configuration files /tmp/house.toml
and /tmp/tmp2/dimensions.toml
:
You can expand these into a single dict with the following:
import nested_config
class Dimensions:
length: int
width: int
class House:
name: str
dimensions: Dimensions
house_dict = nested_config.expand_config("/tmp/house.toml", House)
print(house_dict)
# {'name': 'my house', 'dimensions': {'length': 10, 'width': 20}}
Note that in /tmp/house.toml
, dimensions
is not a mapping but is a path to another
toml file at a path relative to house.toml
.
See tests for more detailed use-cases, such as where the root model contains lists or dicts of other models and when those may be included in the root config file or specified as paths to sub-config files.
Nomenclature
loader
A loader is a function that reads a config file and returns a dict
containing the
key-value pairs from the file. nested-config includes loaders for JSON, TOML, and (if
PyYAML is installed) YAML. For example, the JSON loader looks like this:
import json
def json_load(path):
with open(path, "rb") as fobj:
return json.load(fobj)
model
nested-config uses the term model to refer to a class definition that includes
annotated attributes. For example, this model, Dimensions
, includes three attributes,
each of float type, x
, y
, and z
:
class Dimensions:
x: float
y: float
z: float
A model can be decorated as a dataclass or using attrs.define
or
can subclass pydantic.BaseModel
to provide some method for instantiating an
object instance of the model but they aren't necessary to use nested-config.
The only criterion for a type to be a model is that is has a __dict__
attribute that
includes an __annotations__
member. Note: This does not mean that instances of
the model must have a __dict__
attribute. For example, instances of classes with
__slots__
and NamedTuple
instances may not have a __dict__
attribute.
nested model
A nested model is a model that is included within another model as one of its class
attributes. For example, the below model House
includes an name
of string type, and an
attribute dimensions
of Dimensions
type (defined above). Since Dimensions
is a
model type, this is an example of a nested model.
class House:
name: str
dimensions: Dimensions
config dict
A config dict is simply a dict
with string keys such as may be obtained by reading in
configuration text. For example reading in a string of TOML text with tomllib.loads
returns a config dict.
import tomllib
config = "x = 2\ny = 3"
print(tomllib.loads(config))
# {'x': 2, 'y': 3}
API
nested_config.expand_config(config_path, model, *, default_suffix = None)
This function first loads the config file at config_path
into a config
dict using the appropriate loader. It then uses the attribute
annotations of model
and/or any nested models within model
to see if any of the string values in the configuration file correspond to a nested model.
For each such case, the string is assumed to be a path and is loaded into another config
dict which replaces the string value in the parent config dict. This continues until all
paths are converted and then the fully-expanded config dict is returned.
Note that all non-absolute string paths are assumed to be relative to the path of their parent config file.
The loader for a given config file is determined by file extension (AKA suffix). If
default_suffix
is specified, any config file with an unknown suffix or no suffix will be
assumed to be of that type, e.g. ".toml"
. (Otherwise this is an error.) It is possible
for one config file to include a path to a config file of a different format, so long as
each file has the appropriate suffix and there is a loader for that suffix.
nested_config.config_dict_loaders
config_dict_loaders
is a dict
that maps file suffixes to loaders.
Included loaders
nested-config automatically loads the following files based on extension:
Format | Extensions(s) | Library |
---|---|---|
JSON | .json | json (stdlib) |
TOML | .toml | tomllib (Python 3.11+ stdlib) or tomli |
YAML | .yaml, .yml | pyyaml (extra dependency[^yaml-extra]) |
Adding loaders
To add a loader for another file extension, simply update config_dict_loaders
:
import nested_config
from nested_config import ConfigDict # alias for dict[str, Any]
def dummy_loader(config_path: Path | str) -> ConfigDict:
return {"a": 1, "b": 2}
nested_config.config_dict_loaders[".dmy"] = dummy_loader
# or add another extension for an existing loader
nested_config.config_dict_loaders[".jsn"] = nested_config.config_dict_loaders[".json"]
# or use a different library to replace an existing loader
import rtoml
def rtoml_load(path) -> ConfigDict:
with open(path, "rb") as fobj:
return rtoml.load(fobj)
nested_config.config_dict_loaders[".toml"] = rtoml_load
Deprecated features in v2.1.0, to be removed in v3.0.0
The following functionality is available only if Pydantic is installed:
nested_config.validate_config()
expands a configuration file according to a Pydantic model and then validates the config dictionary into an instance of the Pydantic model.nested_config.BaseModel
can be used as a replacement forpydantic.BaseModel
to include afrom_config()
classmethod on all models that usesnested_config.validate_config()
to create an instance of the model.- By importing
nested_config
,PurePath
validators and JSON encoders are added topydantic
in Pydantic 1.8-1.10 (they are included in Pydantic 2.0+)
Pydantic 1.0/2.0 Compatibility
The pydantic functionality in nested-config is runtime compatible with Pydantic 1.8+ and Pydantic 2.0.
The follow table gives info on how to configure the mypy and Pyright type checkers to properly work, depending on the version of Pydantic you are using.
Pydantic Version | mypy config | mypy cli | Pyright config |
---|---|---|---|
2.0+ | always_false = PYDANTIC_1 |
--always-false PYDANTIC_1 |
defineConstant = { "PYDANTIC_1" = false } |
1.8-1.10 | always_true = PYDANTIC_1 |
--always-true PYDANTIC_1 |
defineConstant = { "PYDANTIC_1" = true } |
Footnotes
[^yaml-extra]: Install pyyaml
separately with pip
or install nested-config with
pip install nested-config[yaml]
.
[^assoc-array]: Each language uses one or more names for an associative arrays. JSON calls
it an object, YAML calls is a mapping, and TOML calls is a table.
Any of course in Python it's a dictionary, or dict
.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for nested_config-2.1.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | eeac03a333fdc7f08863a0dbb49edc82785874898750a484061ef10e398bf08b |
|
MD5 | cbab17ffba81a379b92e55c8320dd1bd |
|
BLAKE2b-256 | 56e7efd95dfd3ff7e0e1a3119d8e64c8491395e01d5bc5750c73c535d8d667f9 |