cliconfig 0.4.0

Merge your config files and set parameters from the command line in a simple way.

CLI Config

Lightweight library that provides routines to merge your configs (optionally nested) and set parameters from command line. It also prevents you from adding new parameters if not desired. Finally, it provides helper routines to manage flatten dicts, unflatten (= nested) dicts or a mix of both, save config, load, display, etc.

Documentation :memo: here

Installation

In a new virtual environment, simply install the package with:

pip install cliconfig

Quick start

First create a default config that can be split in multiple files that will be merged (from left to right in make_config function). There is no limit of depth for the configuration parameters.

---  # default1.yaml
param1: 1
param2: 0
letters:
  letter1: a
  letter2: b

---  # default2.yaml
param1: 1
param2: 2  # will override param2 from default1.yaml
letters.letter3: c  # add a new parameter

Now you can set up your program to use the config:

# main.py
from cliconfig import make_config, show_config

config = make_config('default1.yaml', 'default2.yaml')
show_config(config)

Then add one or multiple additional config files that will override the default values.

---  # first.yaml
letters:
  letter3: C

---  # second.yaml
param1: -1
letters.letter1: A

The additional config files cannot add new parameters that are not in default configs. It is intended to prevent typos in the config files that would not be detected. It also improves the readability of the config files and the retro-compatibility.

Now you can launch the program with additional configurations and parameters. The additional configurations are indicated with --config (separate with comma, without space) and the parameters with --<param_name> (use dot for nested configs). The default configuration will be merged with the additional configurations (from left to right), then the parameters will be set.

python main.py --config first.yaml,second.yaml --param2=-2 --letters.letter2='B'

Will show:

[CONFIG] Merge 2 default configs, 2 additional configs and 2 CLI parameter(s).

Config:
    param1: -1
    param2: -2
    letters:
        letter1: A
        letter2: B
        letter3: C

Note that the configurations are native python dicts.

Edge cases

Be careful, tuples and sets are not supported by YAML and cannot be used in configs. Use lists instead if possible

None is not recognized as a None object by YAML but as a string, you may use null or Null instead if you want to set a None object.

Dicts are considered as sub-configs and so you may not be able to change the keys if allow_new_keys=False (default). If you want to modify or add dict keys, you should enclose it in a list.

For instance:

--- default.yaml
logging:
  metrics: ['train loss', 'val loss']
  styles: [{'train loss': 'red', 'val loss': 'blue'}]
--- experiment.yaml
logging:
  metrics: ['train loss', 'val loss', 'val acc']
  styles: [{'train loss': 'red', 'val loss': 'blue', 'val acc': 'cyan'}]

Manipulate configs

You are encouraged to use the routines provided by cliconfig to manipulate configs and even create your own config builder to replace make_config. You can use:

• merge_config and merge_config_file to merge your configs
• parse_cli to parse CLI arguments on config files and additional parameters
• flat_config and unflat_config to flatten and unflatten your configs
• clean_pre_flat to clean conflicting keys before flattening

You can also save, load and display configs with save_config, load_config and show_config functions.

Note that theses functions are aimed to be use with configs but can be used with any python dict.

How to contribute

For development, install the package dynamically and dev requirements with:

pip install -e .
pip install -r requirements-dev.txt

Everyone can contribute to CLI Config, and we value everyone’s contributions. Please see our contributing guidelines for more information 🤗

Todo

Priority:

• add a routine to check if a tag is in a key and robust to all other tags possible
• add an integration test with all built-in processing (and more)
• add ProcessCleanTags that raise en error when a tag is encountered in a key on postmerge.

Secondary:

• add make_processing_cond to make a processing that ensure a condition on a parameter across merged configs
• add make_processing_keep_status to make a processing that keep the status of a parameter across merged configs. The status can be any python object returned by a function that takes the parameter as input
• add ProcessSelect (with tag "@select") to select a subconfig (or parameter) and delete the others configs at the same level (to clean the global config)

License

Copyright (C) 2023 Valentin Goldité

This program is free software: you can redistribute it and/or modify it under the terms of the MIT License. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

This project is free to use for COMMERCIAL USE, MODIFICATION, DISTRIBUTION and PRIVATE USE as long as the original license is include as well as this copy right notice at the top of the modified files.