A tool to manage complex configurations

Load and resolve complex configurations from files, environment variables and command line arguments. Keep your data structures and favorite CLI library.

confarg is a Python library that helps you load configurations in a modular fashion from multiple sources: files, environment variables, and command line arguments.

It can handle deeply nested configurations, type unions, derived classes, expressions and variable interpolation, configuration compositions and more, and can integrate with your favorite argument parser library such as argparse, click, typer or cyclopts.

confarg is not a framework. No decorator, base class or special annotation type required: none are provided. It is just a tool for the deserialization and serialization of complex configurations. It is designed to have a minimal footprint in your code.

Install

pip install confarg

confarg comes with no dependency, but installing additional libraries such as pyyaml unlocks the support of extra configuration file formats.

TL;DR

This library lets you read configurations stored in classes like this

@dataclass
class Config:
    value: float
    flag: bool
    subconfig: SubConfig1 | SubConfig2

with this kind of code

config = confarg.load(Config)

which lets you build a configuration object from files,

value: 1.0
flag: false
subconfig:
  foo: 42

but also and simultaneously from environment variables,

MYAPP_VALUE=0.0

and command line arguments,

myapp --subconfig.foo=33

Still here? Read along, we are just getting started.

Getting started

All the examples presented in this section (and more) are available in the examples/ folder.

Imagine that you have an app that depends on some parameters that you have collected into a dataclass like so:

@dataclass
class DBConfig:
    host: str
    port: int
    name: str

In your app, you use confarg to instantiate this configuration:

config = confarg.load(DBConfig)

This allows you to construct a DBConfig object by collecting data from three possible sources.

1. From a configuration file. By passing --config <config_file> to your app, confarg will load the content of the file and fill the DBConfig object. For example, a config file could look like so:

   # config.yaml
   host: example.com
   port: 1234
   name: mydb
   

   You would then call your application as

   $ myapp.py --config config.yaml
   DBConfig(host='example.com', port=1234, name='mydb')
   

   Configuration files in TOML and JSON formats are also supported.

   You can change the default config flag to something else using the config_flag parameter.

2. From environment variables. You can declare

   MYAPP_HOST=example.com
   MYAPP_PORT=1234
   MYAPP_NAME=mydb
   

   for the same effect.

   Note that the environment variable prefix of your app should actually be passed to confarg.load like so:

   config = confarg.load(DBConfig, env_prefix="MYAPP_")
   

3. From command line arguments.

   $ my_app --host example.com --port 1234 --name mydb
   DBConfig(host='example.com', port=1234, name='mydb')
   

Progressive build-up

The examples above presented different sources to feed your configuration. They are not mutually exclusive — in fact, they are intended to be used simultaneously.

Note that no one source needs to provide a complete configuration, as long as the configuration resulting from this progressive build-up is complete.

For example, taking our previous example, you could have a partial configuration file containing only host information,

# partial_config.yaml
host: example.com
port: 1234

and provide the schema name from the command line:

$ myapp.py --config partial_config.yaml --name mydb
DBConfig(host='example.com', port=1234, name='mydb')

Source precedence

Configuration data is read in the following order, later read overwriting existing data:

1. configuration files are read first;
2. then environment variables;
3. finally, command line arguments.

This allows for surgical modifications of configuration files. For example, one could overwrite the schema configuration from our existing full_config from the command line like so:

$ # Overwrite the schema name defined in the config file from the command line
$ myapp.py --config config.yaml --name otherdb
DBConfig(host='example.com', port=1234, name='otherdb')

Unions

Let's say your app needs to support SQLite databases. You now have two different, incompatible DB configurations:

@dataclass
class DBServerConfig:
    host: str
    port: int
    name: str

@dataclass
class SQLiteConfig:
    dbpath: str

The DB configuration needs to be either one or the other, which we declare like so:

type DBConfig = SQLiteConfig | DBServerConfig

confarg can handle this new union type and figure out which configuration is desired based on the arguments it got:

$ # Pass DBServerConfig parameters, and you get a DBServerConfig
$ myapp.py --host example.com --port 1234 --name mydb
DBServerConfig(host='example.com', port=1234, name='mydb')
$ # Pass SQLiteConfig parameters, and you get a SQLiteConfig
$ myapp.py --dbpath db.sqlite
SQLiteConfig(dbpath='db.sqlite')

Disambiguation tags

For simple configurations, the above automatic disambiguation is enough and convenient.

In more complex configuration scenarios, this automatic disambiguation may not be not possible. For example, different configurations may share the exact same fields.

Even when disambiguation is possible, it may not be obvious to the human eye which object class should be return from the provided parameters.

Therefore, by necessity or for the sake of clarity, you can provide the class path of the required configuration by using the class tag, like so

$ # Explicitly ask for a SQLiteConfig
$ myapp.py --class myapp.SQLiteConfig --dbpath db.sqlite
SQLiteConfig(dbpath='db.sqlite')

One example where it is necessary to provide the class path is to overwrite the configuration with a new class. Without it, command line arguments are added to the configuration, resulting in an invalid input.

$ # Config file contains a DBServerConfig
$ myapp.py --config db_server.yaml
DBServerConfig(host='example.com', port=1234, name='mydb')
$ # Fails:  dbpath is not a DBServerConfig key
$ myapp.py --config db_server.yaml --dbpath db.sqlite
...
$ # OK: using class signals overwrite existing DB config
$ myapp.py --config db_server.yaml --class myapp.SQLiteConfig --dbpath db.sqlite
SQLiteConfig(dbpath='db.sqlite')

Inheritance

Another way to provide a flexible configuration is to derive akin configuration classes from a common base class.

@dataclass
class DBConfig:
    pass

@dataclass
class DBServerConfig(DBConfig):
    host: str
    port: int
    name: str

@dataclass
class SQLiteConfig(DBConfig):
    dbpath: str

This allows configurations to be easily extensible. Contrast with unions, where a class must be explicitly listed to be supported.

The downside is that the concrete class must be tagged, as confarg cannot discover classes derived from a given class.

$ # Fails:  derived class not specified
$ uv run myapp.py --dbpath db.sqlite
...
$ # OK: explicit class path provided
$ uv run myapp.py --dbpath db.sqlite --class myapp.SQLiteConfig
SQLiteConfig(dbpath='db.sqlite')

Configuration hierarchies

The configurations discussed so far has been rather simple, composed of values grouped together in a dataclass. However, it needs not be. Configurations are generally deeply nested hierarchies, which confarg supports.

Let's say you want to add a log level to your application. You place it at the root level of a new Config object, along with the DB configuration, that is now one level down under the db key.

@dataclass
class Config:
    db: DBConfig
    log_level: Literal["DEBUG", "INFO", "WARNING", "ERROR"] = "INFO"

You now parse your new top-level Config instead of DBConfig.

config = confarg.load(Config)

Our DB configuration, which used to be the root configuration, is now located under the db key. This has the following impact.

For command line arguments, we follow the common convention of using dot-separated paths to address nested fields. Previous command line arguments for DBConfig are now prefixed by db., like so:

$ myapp.py --db.class myapp.SQLiteConfig --db.dbpath db.sqlite
Config(db=SQLiteConfig(dbpath='db.sqlite'), log_level='INFO')

The configuration file is also modified accordingly,

# config.yaml
db:
  class: myapp.DBServerConfig
  host: example.com
  name: mydb
  port: 1234

and is used just like before:

$ myapp.py --config config.yaml
Config(db=DBServerConfig(host='example.com', port=1234, name='mydb'),
       log_level='DEBUG')

Leaf data type and type coercion

You may have noticed that the previous section introduced a log_level parameter that has two interesting features: first, it is not of a simple type (str, int, float, bool or None); second, it comes with a default value.

Default values are honored, and you may have noticed that we did not provide any value to log_level. You can of course override a default value.

As for leaf node data type, confarg coerces Enum and Path types as special exceptions to simple types. Other types are treated as classes and must follow the same rules.

Expressions and variable interpolation

Your application is becoming more complex by the day, and is now requiring a resources configuration.

@dataclass
class Resources:
    cpu_count: int
    memory_gb: int
    max_heap_size_mb: int

It is added to the global configuration under the resources key:

@dataclass
class Config:
    db: DBConfig
    resources: Resources
    log_level: Literal["DEBUG", "INFO", "WARNING", "ERROR"] = "INFO"

Your configuration file has become,

# config.yaml
db:
  class: myapp.DBServerConfig
  host: example.com
  name: mydb
  port: 1234

resources:
  cpu_count: 4
  memory_gb: 16
  max_heap_size_mb: 131072

This works fine. However, you want to better express the fact that max_heap_size_mb is chosen to be 80% of the host memory by default. To achieve this, you can write expressions relying on variable interpolation using the ${...} syntax, like so:

# expression_config.yaml
db:
  class: myapp.DBServerConfig
  host: example.com
  name: mydb
  port: 1234

resources:
  cpu_count: 4
  memory_gb: 16
  max_heap_size_mb: ${int(resources.memory_gb * 1024 * 0.8)}

$ myapp.py --config expression_config.yaml
Config(db=SQLiteConfig(dbpath='db.sqlite'),
       resources=Resources(cpu_count=4, memory_gb=16, max_heap_size_mb=13107),
       log_level='INFO')

Note that variable interpolation occurs after all configuration data is read. This means here that you can override memory_gb from the command line, and max_heap_size_mb will be adjusted accordingly, even though the expression is defined in the configuration file.

$ # Max heap is recomputed according to the expression in the config file
$ myapp.py --config expression_config.yaml --resources.memory_gb 8
Config(db=SQLiteConfig(dbpath='db.sqlite'),
       resources=Resources(cpu_count=4, memory_gb=8, max_heap_size_mb=6553),
       log_level='INFO')

Local variables

Sometimes the value several fields share is not itself part of the configuration — a base directory, a release tag, a deployment name. Adding it to the dataclass just so expressions can reach it pollutes the type with something the application never reads.

The reserved locals: block is where such intermediate values go. Expressions see them as ${locals.<name>}, and the block is dropped before the dataclass is built.

# paths_config.yaml
locals:
  base: /srv/myapp

data_dir: ${locals.base}/data
log_dir: ${locals.base}/logs
cache_dir: ${locals.base}/cache

A local variable may itself be an expression, over another local variable or over a real field, and __include__ works inside the block — so a variable library can be shared between configurations.

A local is also a configuration handle the dataclass never has to know about: once declared, it can be changed from the command line or the environment like any other value.

$ myapp.py --config paths_config.yaml --locals.base /home/bob
Config(data_dir='/home/bob/data', log_dir='/home/bob/logs', cache_dir='/home/bob/cache')

One flag moved three fields. The namespace has two operations, and only one is restricted: declaring a local — introducing the name, and with it its type — happens in configuration files, while modifying a declared one works from every channel. Declaring is restricted because a local has no field annotation, so its type is whatever declared it, and only a format that carries types can. Once declared the type is fixed: an override is converted to it, so ${locals.cpus * 2} keeps multiplying, and a value that will not convert is rejected. Setting a name no configuration file declared is an error rather than a new variable, so typos are caught.

To introduce a new local from outside, point at a file with --config.locals FILE.

The namespace answers to two names, locals and _locals, and either may be used. The second exists so it never steals a name you wanted: if your configuration class declares a field called locals, that field keeps the name — a real field always wins, as it does over the .json cast — and local variables move to _locals. Nothing is configurable, so every front-end and channel agrees on the name without being told; declaring under both names at once is rejected as ambiguous.

A locals: block is not restricted to the top level. Any configuration file may declare one, and it stays with that file wherever the file is loaded — so a reusable component can carry its own scratch values without the configuration including it knowing they exist. Such a variable is modified from the command line or environment by its full path, e.g. --db.locals.base /home/bob.

Building large configurations from parts

Large configurations are often made up of independent components, and as such, you may want to split them accordingly. It is easier to navigate, but it also makes it possible to reuse configuration parts and to build multiple complex configurations from the same set of atomic configuration components.

Some configuration components may even be generated automatically, in which case being able to isolate those parts from the rest is a must.

confarg lets you do this in different ways.

From the command line, the --config flag can be suffixed with a key path to load configurations there. For example,

# Load a config file specific to the `db` key
$ myapp.py --config.db db_config.yaml
Config(db=DBServerConfig(host='example.com', port=1234, name='mydb'), log_level='INFO')

A similar pattern applies to environment variables:

$ MYAPP_CONFIG_DB=db_config.py myapp.py
Config(db=DBServerConfig(host='example.com', port=1234, name='mydb'), log_level='INFO')

Note that db_config.yaml does not contain the db key. It does not need to know the path it is loaded to.

In config files, you can load a configuration by specifying the special __include__ key, followed by the path to the sub-configuration to load, like so:

# set everything under the `db` key from another file
db:
  __include__: ./db_config.yaml

The __include__ keyword can also be used at the top-level, to create a new config that amends an existing config.

# start from this base configuration
__include__: base_config.yaml

# set or overwrite everything under the `db` key
db:
  __include__: ./db_config.yaml

Expressions in a configuration part

A configuration file does not know where it will be loaded, and its expressions do not have to either: a path is relative to the file that wrote it. So db_config.yaml refers to its own values directly, and keeps working whether it is loaded at the root, under db, or deeper inside some larger configuration.

# db_config.yaml — reusable, because it only ever talks about itself
host: example.com
port: 1234
url: postgres://${host}:${port}/mydb

When a part genuinely does depend on another part, prefix the path with a dot to anchor it at the root of the whole configuration instead:

# only makes sense inside a configuration that has `deploy_env` at its root
name: mydb-${.deploy_env}

The dot is the visible difference between a file you can reuse anywhere and one that is a deliberate piece of a specific whole. In a configuration that is a single unnested file the two are the same thing, since that file's root is the configuration root.

Next steps

We have more than scratched the surface, and you should have enough knowledge to cover most of your needs.

Again, all of the examples above and more are in the examples/ folder, which is a great way to discover and experiment with the library features.

A documentation is also currently being written at https://confarg.github.io/confarg/.