config-patching

Tools for customising configuration files without maintaining a complete copy of the upstream configuration.

Many Python libraries and Linux packages build their configuration from a base configuration and user-specific customisations. config-patching provides a similar mechanism for Python libraries whose configuration is stored in JSON, JSON5, or YAML.

Instead of copying and modifying the entire configuration file, you can provide a small patch containing only the settings you want to change. The patch is applied to the base configuration when it is loaded.

config-patching is based on the JSON Merge Patch mechanism defined by RFC 7396.

Early Development

config-patching was originally developed as part of the OpenFlexure Microscope Server project and has now been extracted into its own standalone tool, and relicensed under the MIT license.

The core patching mechanism is already established, but config-patching is still in early development as an independent project. The API and surrounding functionality may continue to evolve as the project matures.

In the future, we plan to add additional functionality, such as tools for creating patches from existing configuration files.

Installation

Install config-patching with pip:

pip install config-patching

or

pip install config-patching[all]

if you need JSON5/YAML support.

Loading a configuration

The main entry point is load_patched_config():

from config_patching import load_patched_config

config = load_patched_config("config.json")

If no special keys are present then load_patched_config() loads the specified configuration file and returns its contents unchanged.

If the configuration contains a base_config_file field, it is instead treated as a patch configuration. The specified base configuration is loaded first, and an optional patch is then applied to it.

There are therefore three possible cases:

1. A normal configuration

If base_config_file is not present, the file is treated as a normal configuration and returned as-is.

{
  "foo": "bar"
}

Calling:

config = load_patched_config("config.json")

returns the python dictionary:

{"foo": "bar"}

2. A base configuration with no patch

If base_config_file is present but patch is not, the base configuration is loaded and returned unchanged.

{
  "base_config_file": "base.json"
}

For example, if base.json contains:

{
  "foo": "bar",
  "enabled": true
}

then:

config = load_patched_config("config.json")

returns the python dictionary:

{
  "foo": "bar",
  "enabled": True
}

3. A base configuration with a patch

If both base_config_file and patch are present, the patch is applied to the base configuration.

{
  "base_config_file": "base.json",
  "patch": {
    "enabled": false
  }
}

Given this base configuration:

{
  "foo": "bar",
  "enabled": true
}

the result is the python dictionary:

{
  "foo": "bar",
  "enabled": False
}

Patch files

A file is only treated as a patch file when it contains the base_config_file field. This means existing configuration files can be passed directly to load_patched_config() without requiring any special wrapper.

For example, this remains a normal configuration file:

{
  "server": {
    "port": 8080
  }
}

while this is a patch configuration:

{
  "base_config_file": "base.json",
  "patch": {
    "server": {
      "port": 9090
    }
  }
}

The base_config_file path is resolved relative to the directory containing the patch configuration file.

Supported formats

config-patching supports:

• JSON
• JSON5 - JSON with comments!
• YAML

Both YAML and JSON5 require separate libraries. To use these install with pip install config-patching[all]

TOML is not supported because the patching mechanism relies on null values, which are not available in TOML.

The patching mechanism in detail

Patches use JSON Merge Patch semantics.

Given this base configuration:

{
  "application": {
    "name": "example",
    "debug": false
  },
  "logging": {
    "level": "INFO",
    "format": "standard"
  }
}

A patch can contain only the keys you want to change, even if those keys are nested:

{
  "base_config_file": "base.json",
  "patch": {
    "application": {
      "debug": true
    },
    "logging": {
      "level": "DEBUG"
    }
  }
}

The resulting configuration is:

{
  "application": {
    "name": "example",
    "debug": True
  },
  "logging": {
    "level": "DEBUG",
    "format": "standard"
  }
}

Keys that are not included in the patch are left unchanged.

Removing a key

A value of null removes a key:

{
  "base_config_file": "config.json",
  "patch": {
    "logging": {
      "format": null
    }
  }
}

Replacing arrays

Arrays are replaced rather than merged:

{
  "base_config_file": "config.json",
  "patch": {
    "allowed_hosts": [
      "localhost",
      "example.org"
    ]
  }
}

The returned configuration contains the fully patched configuration.

YAML example

Patches can also be written in YAML:

base_config_file: /etc/my-library/config.yaml

patch:
  server:
    host: 0.0.0.0
    port: 8080
  logging:
    level: DEBUG

Contributing

As config-patching develops as a standalone project, there are plenty of opportunities to improve the implementation, documentation, testing, and tooling around it.

If you have an idea, find a bug, or would like to contribute a feature, please open an issue or submit a merge request.