genconfig 1.4.0

Manage Json and Yaml config using folder structure

GenConfig

Manage config using folder structure, currently support Json and Yaml.

• Installation
• Example usage
• Project overview
• Usage
  • Parameters

Installation

pip install genconfig

Example usage

1. Arrange the config files into folders, sub-folders, and files.

   sample-config/config-mix/
   ├── basic.yml
   ├── function
   │   ├── function1.json
   │   └── function2.yml
   ├── param.yml
   └── pipeline.json
   
   1 directory, 5 files
   

2. Run the genconfig

   genconfig sample-config/config-mix -o config.json
   

3. Retrieve a single merged config

   {
       "name": "config-01",
       "training": true,
       "parameters": {
           "num_nodes": 200,
           "num_samples": 100,
           "max_time": 40
       },
       "function": {
           "function1": {
               "name": "transform",
               "param": "col1"
           },
           "function2": {
               "name": "load",
               "param": "col"
           }
       },
       "pipeline": [
           {
               "name": "extraction",
               "function": "etl.extraction"
           },
           {
               "name": "training",
               "function": "model.training"
           },
           {
               "name": "evaluation",
               "function": "model.evaluation"
           },
           {
               "name": "deployment",
               "function": "cloud.deploy"
           }
       ]
   }
   

Project overview

The dream of a mature project is to cover most of the essential functions, and the interactions users would require is to modify the configuration files based on specific needs. For example, a user running model training for a new region A does not need to write any new codes, instead simply modifying the configuration files.

However, as the project grows, so does the configuration file. There are various settings region A would require but not all of them need constant updates. Therefore, having a programmatic way to update the configuration files and to display them in an easy-to-understand format is crucial.

I propose to present the configuration files in a folder structure. The root folder represents the first level keys in the configs, and the subfolders represent nested keys.

Furthermore, we should allow both json and yaml config files. Json is easy to create while yaml allows comments and variables creation. Both Json and yaml have their advantages.

The API documentation is available at https://lingjie00.github.io/genconfig/

The GitHub repo is at https://github.com/lingjie00/genconfig

Usage

1. Convert a folder structure into a single json config file

   genconfig --path=folder_path --output="config.json"
   

Parameters

• ignored
  • User can choose to ignore some keys and not expand into sub-folders

  # supports regex matching
  genconfig config_path -o config.json --ignore "config1.json" "config2.yaml" "debug.*json"
  

• keep
  • Different from ignored, instead read only files with matching file name

  # supports regex matching
  genconfig config_path -o config.json --keep "only_this_config.yaml"
  

• verbose
  • the level of logging to display

  # refer to python logging for acceptable levels
  genconfig config_path -o config.json --verbose "DEBUG"
  

• append
  • useful for manual replace/update values

  # input has to be in json format
  genconfig config_path -o config.json --append "{'name': 'updated_name'}"
  

• read
  • Useful when there is a mixture of config file types but only want to read a specific file type

  genconfig config_path -o config.json --read "json"
  

• folder
  • useful to ignore folder names as key, then user can arrange config files in any subfolders without worrying about nested keys

  genconfig config_path -o config.json --folder False