caep 1.6.0

Config Argument Env Parser (CAEP)

CAEP

Configuration library that supports loading configuration from ini, environment variables and arguments into a pydantic schema.

With the pydantic schema you will have a fully typed configuration object that is parsed at load time.

Change log

1.6.0

• Add support to retrieve unknown arguments using json_schema_extra={"caep_unknown_args": True}

1.5.0

• Allow options in ini files to have underscores (_)
• Handle unknown options in ini files. As default a warning will be emitted, but this can be configured in load() with the unknown_config_key option:
  • warning: emit warning (default)
  • ignore: ignore unknown options
  • error: fatal error - raise ValueError and will exit unless raise_on_validation_error is True

1.3.0

• Use TypeVar in load to support typing when loading configuration with a specified module
• Drop support for python 3.6, 3.7 and 3.8.

1.1.0

Support list/set/dict defaults, so you can now do:

intlist: list[int] = Field([0,1,2], description="List of ints")

The previous way to define defaults using strings is still supported, but will fail type checking with the pydantic mypy plugin, and will be removed in a later version:

intlist: list[int] = Field("0,1,2", description="List of ints")

1.0.0

Support for pydantic 2.x. It is advised to migrate models with these changes:

Use min_length instead of min_size

Pydantic has built-in support for size of list, dictionaries and sets using min_length so you should change

intlist: list[int] = Field(description="Space separated list of ints", min_size=1)

to

intlist: list[int] = Field(description="Space separated list of ints", min_length=1)

Migrate split and kv_split to json_schema_extra

Do not use split and kv_split directly on the field, but put them in a dictionary json_schema_extra. E.g. change

intlist: list[int] = Field(description="Space separated list of ints", split=" ")

to

intlist: list[int] = Field(
    description="Space separated list of ints", json_schema_extra={"split": " "}
)

and change

dict_int: Dict[str, int] = Field(
    description="Int Dict split by slash and dash", split="-", kv_split="/"
)

to

dict_int: Dict[str, int] = Field(
    description="Int Dict split by slash and dash",
    json_schema_extra={"split": "-", "kv_split": "/"},
)

Migrate root_validator to model_validator

root_validator are still supported, but it is advised to migrate to model_validator. Example using helper function raise_if_some_and_not_all:

    @model_validator(mode="after")  # type: ignore
    def check_arguments(cls, m: "ExampleConfig") -> "ExampleConfig":
        """If one argument is set, they should all be set"""

        caep.raise_if_some_and_not_all(
            m.__dict__, ["username", "password", "parent_id"]
        )

        return m

Example

#!/usr/bin/env python3

from pydantic import BaseModel, Field

import caep


class Config(BaseModel):

    text: str = Field(description="Required String Argument")
    number: int = Field(default=1, description="Integer with default value")
    switch: bool = Field(description="Boolean with default value")
    intlist: list[int] = Field(description="Space separated list of ints", json_schema_extra={"split": " "})


# Config/section options below will only be used if loading configuration
# from ini file (under ~/.config)
config = caep.load(
    Config,
    "CAEP Example",
    "caep",  # Find .ini file under ~/.config/caep
    "caep.ini",  # Find .ini file name caep.ini
    "section",  # Load settings from [section] (default to [DEFAULT]
)

print(config)

Sample output with a intlist read from environment and switch from command line:

$ export INTLIST="1 2 3"
$ ./example.py --text "My value" --switch
text='My value' number=1 switch=True intlist=[1, 2, 3]

Load config without ini support

Specifying configuration location, name and section is optional and can be skipped if you do not want to support loading ini files from $XDG_CONFIG_HOME:

# Only load arguments from environment and command line
config = caep.load(
    Config,
    "CAEP Example",
)

Load config without command line support

opts argument to load() can be used to override what command line arguments should be parsed. This is used extensively in testing, and can also be used to disable handling of command line arguments.

# Only load from ini and environment variables
config = caep.load(
    Config,
    "CAEP Example",
    "caep",  # Find .ini file under ~/.config/caep
    "caep.ini",  # Find .ini file name caep.ini
    "section",  # Load settings from [section] (default to [DEFAULT]
    opts = [],
)

With the code above you can still specify an ini file with --config <ini-file>, and use environment variables and command line arguments.

Capture unknown command line arguments

You can capture and keep unknown CLI arguments by marking a single field with json_schema_extra={"caep_unknown_args": True}. This field will receive the raw tokens returned by argparse.parse_known_args when unknown_config_key="ignore":

class Config(BaseModel):
    text: str = Field(description="Required String Argument")
    unknown: list[str] = Field(
        default_factory=list,
        description="Unknown CLI arguments",
        json_schema_extra={"caep_unknown_args": True},
    )


config = caep.load(
    Config,
    "CAEP Example",
    unknown_config_key="ignore",
)

In this mode, unknown CLI tokens are stored in config.unknown while known arguments are parsed as usual. Only one field can be marked as unknown-argument.

If you run the above script with this command line:

script.py --text hello --other value1 value2

the config object will have these values:

test: hello
unknown: ["--other", "value1", "value2"]

Pydantic field types

Pydantic fields should be defined using Field and include the description parameter to specify help text for the command line.

Unless the Field has a default value, it is a required field that needs to be specified in the environment, configuration file or on the command line.

Many of the types described in https://docs.pydantic.dev/usage/types/ should be supported, but not all of them are tested. However, nested schemas are not supported.

Tested types:

str

Standard string argument.

int

Values parsed as integer.

float

Value parsed as float.

pathlib.Path

Value parsed as Path.

ipaddress.IPv4Address

Values parsed and validated as IPv4Address.

ipaddress.IPv4Network

Values parsed and validated as IPv4Network.

bool

Value parsed as booleans. Booleans will default to False, if no default value is set. Examples:

Field	Input	Configuration
enable: bool = Field(description="Enable")		False
enable: bool = Field(value=False, description="Enable")	yes	True
enable: bool = Field(value=False, description="Enable")	true	True
disable: bool = Field(value=True, description="Disable")		True
disable: bool = Field(value=True, description="Disable")	yes	False
disable: bool = Field(value=True, description="Disable")	true	False

list[str]

List of strings, split by specified character (default = comma, argument=split).

Some examples:

Field	Input	Configuration
list[int] = Field(description="Ints", json_schema_extra={"split": " "})	1 2	[1, 2]
list[str] = Field(description="Strs")	ab,bc	["ab", "bc"]

The argument min_length (pydantic built-in) can be used to specify the minimum size of the list:

Field	Input	Configuration
list[str] = Field(description="Strs", min_length=1)	``	Raises ValidationError

set[str]

Set, split by specified character (default = comma, argument=split).

Some examples:

Field	Input	Configuration
Set[int] = Field(description="Ints", json_schema_extra={"split": " "})	1 2 2	{1, 2}
Set[str] = Field(description="Strs")	ab,ab,xy	{"ab", "xy"}

The argument min_length can be used to specify the minimum size of the set:

Field	Input	Configuration
Set[str] = Field(description="Strs", min_length=1)	``	Raises ValidationError

dict[str, <TYPE>]

Dictionary of strings, split by specified character (default = comma, argument=split for splitting items and colon for splitting key/value).

Some examples:

Field	Input	Configuration
Dict[str, str] = Field(description="Dict")	x:a,y:b	{"x": "a", "y": "b"}
Dict[str, int] = Field(description="Dict of ints")	a b c:1, d e f:2	{"a b c": 1, "d e f": 2}

The argument min_length can be used to specify the minimum number of keys in the dictionary:

Field	Input	Configuration
Dict[str, str] = Field(description="Strs", min_length=1)	``	Raises ValidationError

Configuration

Arguments are parsed in two phases. First, it will look for the optional argument --config which can be used to specify an alternative location for the ini file. If not --config argument is given it will look for an optional ini file in the following locations (~/.config has precedence) if config_id and config_name is specified:

• ~/.config/<CONFIG_ID>/<CONFIG_FILE_NAME> (or directory specified by $XDG_CONFIG_HOME)
• /etc/<CONFIG_FILE_NAME>

The ini file can contain a [DEFAULT] section that will be used for all configurations. In addition it can have a section that corresponds with <SECTION_NAME> (if specified) that for specific configuration, that will override config from [DEFAULT]

Environment variables

The configuration step will also look for environment variables in uppercase and with - replaced with _. For the example below it will lookup the following environment variables:

• $NUMBER
• $BOOL
• $STR_ARG

The configuration precedence are (from lowest to highest):

• argparse default
• ini file
• environment variable
• command line argument

Validation

XDG

Helper functions to use XDG Base Directories are included in caep.xdg:

It will look up XDG environment variables like $XDG_CONFIG_HOME and use defaults if not specified.

get_xdg_dir

Generic function to get a XDG directory.

The following example will return a path object to ~/.config/myprog (if $XDG_CONFIG_HOME is not set) and create the directory if it does not exist.

get_xdg_dir("myprog", "XDG_CONFIG_HOME", ".config", True)

get_config_dir

Shortcut for get_xdg_dir("CONFIG").

get_cache_dir

Shortcut for get_xdg_dir("CACHE").

CAEP Legacy usage

Prior to version 0.1.0 the recommend usage was to add parser objects manually. This is still supported, but with this approach you will not get the validation from pydantic:

>>> import caep
>>> import argparse
>>> parser = argparse.ArgumentParser("test argparse")
>>> parser.add_argument('--number', type=int, default=1)
>>> parser.add_argument('--bool', action='store_true')
>>> parser.add_argument('--str-arg')
>>> args = caep.config.handle_args(parser, <CONFIG_ID>, <CONFIG_FILE_NAME>, <SECTION_NAME>)

Helper Functions

raise_if_some_and_not_all

Raise ArgumentError if some of the specified entries in the dictionary has non false values but not all

class ExampleConfig(BaseModel):
    username: Optional[str] = Field(description="Username")
    password: Optional[str] = Field(description="Password")
    parent_id: Optional[str] = Field(description="Parent ID")

    @model_validator(mode="after")  # type: ignore
    def check_arguments(cls, m: "ExampleConfig") -> "ExampleConfig":
        """If one argument is set, they should all be set"""

        caep.raise_if_some_and_not_all(
            m.__dict__, ["username", "password", "parent_id"]
        )

        return m

script_name

Return first external module that called this function, directly, or indirectly

Testing

We aim to have good test coverage in the library and you can get a coverage report by running:

uv run coverage run -m pytest
uv run coverage report -m