slcfg (Simple Layered ConFiGuration)

slcfg is a small, dependency-free library for assembling application configuration from ordered layers. It reads values from files, environment variables, or application code, merges them into a nested dictionary, and passes the result to a model or factory for validation.

base file
    -> environment file
    -> optional local file
    -> encoded deployment configuration
    -> individual environment variables
    -> explicit application overrides
    -> validated configuration model

slcfg handles loading and merging. Validation, type coercion, defaults, and unknown-field handling belong to the model you provide. Pydantic works particularly well for this, but it is not required.

Requirements and Installation

slcfg requires Python 3.12 or newer within the Python 3 release series and has no third-party runtime dependencies.

pip install slcfg

The examples in this README use Pydantic 2, which is an optional dependency:

pip install slcfg pydantic

Quick Start

Define the shape of the application configuration:

import enum
from pathlib import Path

import pydantic
import slcfg


class ApiConfig(pydantic.BaseModel):
    model_config = pydantic.ConfigDict(extra='forbid', frozen=True)

    host: str
    port: int


class DatabaseConfig(pydantic.BaseModel):
    model_config = pydantic.ConfigDict(extra='forbid', frozen=True)

    url: pydantic.SecretStr
    pool_size: int


class Config(pydantic.BaseModel):
    model_config = pydantic.ConfigDict(extra='forbid', frozen=True)

    api: ApiConfig
    database: DatabaseConfig

Create a shared configuration file:

# config/base.toml

[api]
host = "127.0.0.1"
port = 8000

[database]
pool_size = 10

Environment-specific files only need to contain differences:

# config/local.toml

[api]
host = "0.0.0.0"

Select an environment and define the layer order:

class Env(slcfg.Environment):
    LOCAL = enum.auto()
    CONTAINER = enum.auto()


CONFIG_DIR = Path(__file__).parent / 'config'


def read_app_config(
    *,
    extra: dict[str, object] | None = None,
    env: Env | None = None,
    config_dir: Path = CONFIG_DIR,
) -> Config:
    active_env = env or Env.get_from_env('MYAPP_ENV', default=Env.LOCAL)

    layers = [
        slcfg.toml_file_layer(config_dir / 'base.toml'),
        slcfg.toml_env_file_layer(active_env, config_dir),
        slcfg.toml_file_layer(config_dir / 'env.toml', optional=True),
        slcfg.env_base64_toml_layer('MYAPP_CONFIG', optional=True),
        slcfg.env_layer('MYAPP_CONFIG__', nested_delimiter='__'),
    ]

    if extra is not None:
        layers.append(slcfg.value_layer(extra))

    return slcfg.read_config(Config, layers)

Provide values that are intentionally absent from the shared files and override individual fields with environment variables:

export MYAPP_CONFIG__DATABASE__URL='postgresql://localhost/app'
export MYAPP_CONFIG__API__PORT='9000'

config = read_app_config()

assert config.api.host == '0.0.0.0'
assert config.api.port == 9000
assert config.database.pool_size == 10

Layers are evaluated in order. Values from later layers replace values at the same leaf path, so the environment variable for api.port replaces the value from base.toml. Pydantic then coerces the string '9000' to an integer and validates the complete configuration.

Core Concepts

Items and Paths

Every layer produces Item objects. An item contains a nested path and a value:

slcfg.Item(['database', 'pool_size'], 10)

This item represents:

{'database': {'pool_size': 10}}

Nested dictionaries are flattened to leaf items before they are merged. Lists and other values remain complete leaf values.

Layers

A Layer is a source of items. File layers parse a file into items, environment layers map variable names to item paths, and value_layer() converts a dictionary supplied by the application.

Layers and their items are processed sequentially. The order is the precedence order.

Sources Are Lazy

A Source stores a zero-argument getter. Files and environment variables are read when read_config() evaluates the layer, not when the layer is created. Reusing a layer in another read_config() call rereads its input; values are not cached.

Model Construction

After merging all layers, read_config() calls the supplied callable with the resulting top-level mapping as keyword arguments:

result = model(**merged_values)

The callable can be a Pydantic model, a dataclass, an ordinary class, or a factory function. To retrieve the merged dictionary without model validation, use dict:

values = slcfg.read_config(
    dict,
    [
        slcfg.value_layer({'api': {'port': 8000}}),
        slcfg.value_layer({'api': {'host': '127.0.0.1'}}),
    ],
)

assert values == {'api': {'port': 8000, 'host': '127.0.0.1'}}

The final root must be a string-keyed mapping because it is expanded with **. Nested dataclass construction, field aliases, coercion, defaults, and extra-field behavior are responsibilities of the callable.

Typical Layer Order

A practical application configuration commonly uses this order:

1. Shared non-secret defaults in base.toml.
2. Environment-specific differences such as local.toml or container.toml.
3. An optional, ignored env.toml for developer-local values.
4. Base64-encoded TOML or JSON supplied by a deployment system.
5. Individual environment variables for targeted overrides.
6. Programmatic overrides supplied by tests or callers.

Required values can come from any layer. They do not need to appear in the base file as long as the final model receives them.

This order is a convention, not a requirement. slcfg does not assign special precedence to a particular layer type.

Built-in Layers

Layer	Input	Behavior
toml_file_layer(path, optional=False)	TOML file	Parses TOML and emits its leaf values.
json_file_layer(path, optional=False)	JSON file	Parses a JSON object and emits its leaf values.
toml_env_file_layer(env, directory, optional=False)	Environment enum and directory	Reads <env.name.lower()>.toml.
json_env_file_layer(env, directory, optional=False)	Environment enum and directory	Reads <env.name.lower()>.json.
env_layer(prefix, nested_delimiter, case_sensitive=False)	Process environment	Maps matching variable names to nested paths.
env_base64_toml_layer(name, optional=False)	One environment variable	Decodes a complete base64 TOML document.
env_base64_json_layer(name, optional=False)	One environment variable	Decodes a complete base64 JSON document.
value_layer(value)	In-memory value tree	Emits values supplied by application code.

TOML and JSON Files

File layers accept objects with a read_bytes() method, including pathlib.Path and compatible package-resource objects. Plain path strings are not accepted.

from pathlib import Path

layers = [
    slcfg.toml_file_layer(Path('config/base.toml')),
    slcfg.json_file_layer(Path('config/override.json'), optional=True),
]

Use the environment file helpers when an enum member selects a file in a configuration directory:

toml_layer = slcfg.toml_env_file_layer(Env.LOCAL, Path('config'))
json_layer = slcfg.json_env_file_layer(Env.CONTAINER, Path('config'), optional=True)

These resolve to config/local.toml and config/container.json. Filenames use the lowercase enum member name; enum values are ignored. The directory may be a pathlib.Path or another object with a compatible joinpath() method.

Setting optional=True only makes a missing file a no-op. Existing files still raise errors for invalid syntax, permissions, directories used as files, and other read failures.

TOML values retain the types produced by tomllib, including dates and times. JSON values retain the types produced by json.load(). A JSON configuration document should contain an object at its root so the final merged value is a mapping.

Nested Environment Variables

env_layer() selects variables by prefix, removes the prefix, and splits the remainder on the nested delimiter:

layer = slcfg.env_layer(
    prefix='MYAPP_CONFIG__',
    nested_delimiter='__',
)

MYAPP_CONFIG__API__PORT=9000
MYAPP_CONFIG__DATABASE__URL=postgresql://database/app

The layer emits values equivalent to:

{
    'api': {'port': '9000'},
    'database': {'url': 'postgresql://database/app'},
}

By default, variable names, the prefix, and the delimiter are compared in lowercase. This makes matching case-insensitive and produces lowercase paths. Set case_sensitive=True to preserve and compare the original casing.

Environment values always remain strings. env_layer() does not parse booleans, numbers, JSON, TOML, lists, or .env files. A model can coerce simple values. For structured values, use a model validator, a complete TOML/JSON layer, or a custom transformer.

Use a non-empty delimiter. Repeated or trailing delimiters create empty path components and are usually configuration mistakes.

Base64 TOML and JSON

The base64 layers store one complete configuration document in one environment variable. This is useful when a deployment system can inject environment variables but not mount configuration files.

import base64
import os
from pathlib import Path

os.environ['MYAPP_CONFIG'] = base64.b64encode(
    Path('config/container.toml').read_bytes()
).decode()

layer = slcfg.env_base64_toml_layer('MYAPP_CONFIG')

JSON works in the same way:

payload = b'{"api": {"port": 9000}}'
os.environ['MYAPP_CONFIG'] = base64.b64encode(payload).decode()

layer = slcfg.env_base64_json_layer('MYAPP_CONFIG')

With optional=True, an absent variable produces no items. Decoding and parsing errors from an existing value are not suppressed.

Separate ordinary and sensitive deployment data when that matches the deployment system:

layers = [
    slcfg.env_base64_toml_layer('MYAPP_CONFIG', optional=True),
    slcfg.env_base64_toml_layer('MYAPP_SECRET_CONFIG', optional=True),
]

Base64 is an encoding, not encryption. Environment-variable access and secret storage must still be secured.

Programmatic Values

Use value_layer() for runtime values, test overrides, or configuration retrieved by application code:

config = slcfg.read_config(
    Config,
    [
        slcfg.toml_file_layer(Path('config/base.toml')),
        slcfg.value_layer(
            {
                'api': {'port': 9999},
                'database': {'url': 'postgresql://localhost/test'},
            }
        ),
    ],
)

Prefer value_layer() over manually creating Source and Item objects when the input is already a nested dictionary.

Value Tree Helpers

build_value_tree() reconstructs a value tree from items. It is the inverse operation of slcfg.item.list_items() for ordinary string-keyed configuration mappings:

values = slcfg.build_value_tree(
    [
        slcfg.Item(['api', 'host'], '127.0.0.1'),
        slcfg.Item(['api', 'port'], 8000),
    ]
)

assert values == {
    'api': {
        'host': '127.0.0.1',
        'port': 8000,
    }
}

merge_value_trees() merges nested values in argument order using the same rules as read_config():

values = slcfg.merge_value_trees(
    {'api': {'host': '127.0.0.1', 'port': 8000}},
    {'api': {'port': 9000}},
)

assert values == {
    'api': {
        'host': '127.0.0.1',
        'port': 9000,
    }
}

Both functions accept on_conflict for structural conflicts. iter_item_omissions() supports configuration tests by yielding each leaf item alongside a rebuilt tree that excludes it:

for omitted_item, incomplete_values in slcfg.iter_item_omissions(values):
    print(omitted_item.path, incomplete_values)

The input tree is not modified. As with list_items(), empty dictionaries contain no leaf items and therefore produce no omission cases.

Selecting an Application Environment

Subclass Environment to select an application environment from an environment variable:

import enum

import slcfg


class Env(slcfg.Environment):
    LOCAL = enum.auto()
    TEST = enum.auto()
    CONTAINER = enum.auto()


env = Env.get_from_env('MYAPP_ENV', default=Env.LOCAL)

Selection uses enum member names, not their values. Names are matched case-insensitively, so local, LOCAL, and Local all select Env.LOCAL.

Input	Options	Result
Valid member name	Any	Matching member.
Unset or empty	default=Env.LOCAL	The default.
Unset or empty	No default	NoEnvironmentError.
Invalid non-empty value	Default behavior	InvalidValueError.
Invalid non-empty value	ignore_invalid=True, with a default	The default.
Invalid non-empty value	ignore_invalid=True, without a default	NoEnvironmentError.

Providing a default does not silently accept an invalid non-empty value. Set ignore_invalid=True explicitly if that behavior is desired.

Merging and Conflict Policies

Deep Merge

Different leaf paths merge naturally:

base = slcfg.value_layer({'service': {'host': 'localhost'}})
override = slcfg.value_layer({'service': {'port': 8000}})

values = slcfg.read_config(dict, [base, override])

assert values == {
    'service': {
        'host': 'localhost',
        'port': 8000,
    }
}

When layers provide the same leaf path, the later value wins:

values = slcfg.read_config(
    dict,
    [
        slcfg.value_layer({'service': {'port': 8000}}),
        slcfg.value_layer({'service': {'port': 9000}}),
    ],
)

assert values['service']['port'] == 9000

This ordinary leaf replacement does not require a conflict policy, and it behaves the same under all policies.

Structural Conflicts

A structural conflict occurs when one layer treats a path as a mapping and another treats the same path as a leaf:

{'service': {'port': 8000}}
{'service': 'disabled'}

Pass a ConflictPolicy as the third argument to read_config() to control these conflicts:

values = slcfg.read_config(
    dict,
    layers,
    slcfg.ConflictPolicy.OVERWRITE,
)

Existing shape	Incoming shape	KEEP	OVERWRITE	NEST	TRIM	RAISE or no policy
Non-empty mapping	Leaf at the same path	Keep mapping	Use leaf	Keep mapping	Use leaf	Raise ConflictError
Leaf	Child beneath that path	Keep leaf	Use mapping	Use mapping	Keep leaf	Raise ConflictError

The policies can be read as follows:

• KEEP: preserve the shape established by earlier layers.
• OVERWRITE: prefer the shape supplied by the later layer.
• NEST: prefer the nested mapping shape.
• TRIM: prefer the shallower leaf shape.
• RAISE: reject structural shape changes.

With no policy, structural conflicts also raise ConflictError. Leaving the policy unset is a useful way to detect accidental schema changes. Use OVERWRITE when later layers are intentionally allowed to replace complete sections with scalar values or vice versa.

ConflictError.conflict contains the existing value and incoming Item. Nested conflicts also receive exception notes identifying the path. These values may contain secrets, so avoid logging the complete exception data indiscriminately.

Custom Sources and Transformers

Convenience layers are pipelines built from Source and Transformer. The same primitives can be used for custom formats or configuration stores.

Raw Sources

Source	Value returned when evaluated
source(value)	The captured value. Mutable values are not copied.
file_source(path, default=None)	A BytesIO containing path.read_bytes().
env_source()	A snapshot list of (name, value) environment-variable pairs.
env_var_source(name, default=None)	One environment-variable string, a default, or KeyError.

Built-in Transformers

Transformer	Input and output
base64_transform	Base64 value to decoded BytesIO.
hex_transform	Hexadecimal string to decoded BytesIO.
utf8_transform	String to UTF-8 BytesIO.
json_transform	Binary stream to flattened JSON items.
toml_transform	Binary stream to flattened TOML items.
item_transform	(path, value) pairs to Item objects.

Source.__or__ and Transformer.__or__ compose pipelines with the | operator. For example, a plain JSON environment variable can be turned into a layer without base64:

json_environment_layer = (
    slcfg.env_var_source('MYAPP_JSON')
    | slcfg.utf8_transform
    | slcfg.json_transform
)

Create a custom layer by returning items from a getter:

from slcfg.item import list_items


def fetch_remote_config() -> dict[str, object]:
    # Retrieve and decode data using the application's client.
    return {'api': {'port': 9000}}


remote_layer = slcfg.Source(
    getter=lambda: list_items(fetch_remote_config())
)

The getter executes each time the layer is evaluated. Network retries, authentication, caching, and exception handling remain the application's responsibility.

Errors

slcfg generally preserves the exception raised by the failing source, parser, decoder, merge, or model. It does not wrap all failures in one library-specific exception.

Failure	Typical exception
Required file is missing	FileNotFoundError
Required environment variable is missing	KeyError
Invalid JSON	json.JSONDecodeError
Invalid TOML	tomllib.TOMLDecodeError
Invalid base64 or hex	Decode error or a later parse error
Structural merge conflict	slcfg.ConflictError
Invalid Environment value	slcfg.InvalidValueError
No selected environment or default	slcfg.NoEnvironmentError
Invalid final configuration	Exception raised by the model or factory

optional=True only handles an absent file or environment variable. It does not suppress malformed content or unrelated I/O failures.

Behavior and Limitations

• Configuration mappings should use string keys.
• Lists, tuples, dates, and other non-dictionary values are treated as complete leaf values.
• Empty dictionaries emit no items. A later value_layer({'section': {}}) therefore does not clear an earlier populated section.
• The final merged root must be a mapping because read_config() calls model(**values).
• Environment-variable values are not parsed and remain strings.
• Sources are lazy and are not cached.
• slcfg does not load .env files.
• slcfg does not provide interpolation, schema validation, command-line parsing, remote-store clients, or secret-manager clients.
• Parser, source, and model exceptions propagate to the caller.

These responsibilities can be handled before values enter a layer, in a custom source or transformer, or by the final model.

Secrets

Use secret-aware model fields where available:

class DatabaseConfig(pydantic.BaseModel):
    url: pydantic.SecretStr

Unwrap secrets only where the underlying client needs the raw value:

database_url = config.database.url.get_secret_value()

Base64 configuration is not encrypted. Avoid printing merged value trees, adding complete configuration values to exception messages, or logging Conflict objects that may retain secret values.

Testing Configuration

Use temporary files and a final value_layer() to make precedence explicit:

def test_programmatic_override(tmp_path):
    (tmp_path / 'base.toml').write_text(
        '''
[api]
host = "127.0.0.1"
port = 8000

[database]
url = "postgresql://localhost/test"
pool_size = 5
''',
        encoding='utf-8',
    )
    (tmp_path / 'local.toml').write_text('', encoding='utf-8')

    config = read_app_config(
        config_dir=tmp_path,
        env=Env.LOCAL,
        extra={'api': {'port': 1234}},
    )

    assert config.api.port == 1234

To verify that every leaf in a complete override tree is required, validate the complete tree and then validate each omission:

def test_required_overrides(monkeypatch):
    monkeypatch.delenv('MYAPP_CONFIG', raising=False)
    monkeypatch.delenv('MYAPP_CONFIG__DATABASE__URL', raising=False)

    complete_extra = {
        'database': {
            'url': 'postgresql://localhost/test',
        }
    }

    read_app_config(extra=complete_extra)

    for omitted_item, incomplete_extra in slcfg.iter_item_omissions(complete_extra):
        with pytest.raises(pydantic.ValidationError, match='Field required') as exc:
            read_app_config(extra=incomplete_extra)

        assert '.'.join(omitted_item.path) in str(exc.value)

This tests the contribution of each leaf in the supplied tree. The omission helper only generates value trees; it does not inspect model fields or depend on a particular validation library.

Useful configuration tests cover:

• The complete precedence order.
• Required and optional files.
• Missing and malformed encoded environment variables.
• Nested environment-variable names and model coercion.
• Invalid environment selection.
• Unknown and missing model fields.
• Structural conflict behavior.
• Secret redaction in logs and exceptions.

API Reference

Configuration

read_config(model, layers, on_conflict=None)

Evaluates each layer in order, merges its items, calls model(**merged_values), and returns the callable's result.

• model: Any callable accepting the top-level configuration keys as keyword arguments.
• layers: Ordered list of Layer objects.
• on_conflict: Optional ConflictPolicy for structural conflicts.

Convenience Layers

toml_file_layer(path, *, optional=False)

Reads and flattens a TOML file. A missing file is a no-op when optional.

json_file_layer(path, *, optional=False)

Reads and flattens a JSON file. A missing file is a no-op when optional.

toml_env_file_layer(env, directory, *, optional=False)

Reads <env.name.lower()>.toml from a directory. A missing file is a no-op when optional.

json_env_file_layer(env, directory, *, optional=False)

Reads <env.name.lower()>.json from a directory. A missing file is a no-op when optional.

env_layer(prefix, nested_delimiter, *, case_sensitive=False)

Maps matching environment variables to nested string-valued items.

env_base64_toml_layer(var_name, *, optional=False)

Reads a base64-encoded TOML document from one environment variable.

env_base64_json_layer(var_name, *, optional=False)

Reads a base64-encoded JSON document from one environment variable.

value_layer(value)

Flattens an in-memory value tree into a layer.

Environment Selection

Environment

Enum base class providing get_from_env().

Environment.get_from_env(var_name, *, default=None, ignore_invalid=False)

Selects a member by case-insensitive member name.

InvalidValueError

Raised for an invalid non-empty environment value unless invalid values are ignored.

NoEnvironmentError

Raised when no usable value or default is available.

Source Composition

Source(getter)

Lazy zero-argument value source. source | transformer returns another Source.

Transformer(handler)

Callable transformation wrapper. transformer | other returns a composed transformer.

Layer

Type alias for a Source that returns Items.

source(value)

Creates a source that returns the captured value.

file_source(path, *, default=None)

Reads bytes from a read_bytes()-compatible object into BytesIO. Only FileNotFoundError uses the optional default.

env_source()

Returns all process environment-variable pairs when evaluated.

env_var_source(name, *, default=None)

Returns one variable, its non-None default, or raises KeyError.

base64_transform

Decodes base64 data into BytesIO.

hex_transform

Decodes a hexadecimal string into BytesIO.

utf8_transform

Encodes a string as UTF-8 and returns BytesIO.

json_transform

Parses a binary JSON stream and returns flattened items.

toml_transform

Parses a binary TOML stream and returns flattened items.

item_transform

Converts (path, value) pairs to Item objects.

Items and Conflicts

Item(path, value)

Dataclass representing one value at a nested path.

Items

Type alias for an iterable of Item objects.

Conflict(existing, new)

Dataclass containing the existing value tree and incoming path-relative item.

ConflictError(conflict)

Raised when a structural conflict is rejected. The conflict is available as .conflict.

ConflictPolicy

Enum containing KEEP, OVERWRITE, NEST, TRIM, and RAISE.

build_value_tree(items, on_conflict=None)

Builds a value tree from items in iteration order. Later items replace values at the same leaf path, and the optional policy controls structural conflicts.

merge_value_trees(*trees, on_conflict=None)

Flattens and merges value trees in argument order. The input trees are not modified.

iter_item_omissions(tree)

Yields (omitted_item, incomplete_tree) for each leaf item in a value tree. Every incomplete tree is newly built and does not modify the input.

Low-level Item Helpers

These helpers are available from slcfg.item and are useful when implementing custom layers or testing value trees:

slcfg.item.list_items(tree)

Recursively flattens a nested string-keyed dictionary into leaf Item objects.

slcfg.item.set_item(tree, item, on_conflict)

Applies one item to a value tree and returns the resulting root. Dictionary nodes may be modified in place.

Most application code should use value_layer() and read_config() instead.

Development

Install the development environment and run all checks:

poetry install
poetry run ruff check .
poetry run pyright
poetry run python test --expects-full

License

slcfg is distributed under the MIT license.