quik-config 1.8.0

Project config files

What is this?

A config system that doesn't waste your time

• per-machine settings that stay in sync
• all values in the hierarchy can be overridden with CLI args
• select multiple profiles from CLI (ex: GPU & DEV or UNIX & GPU & PROD)
• a consistent way to handle filepaths (stop hardcoding filepaths as python strings!)
• hierarchical, with inheritable groups of settings (profiles)
• default works along side argparse, but also can just replace it entirely for rapid development
• can combine/import multiple config files

How do I use this?

pip install quik-config

In a config.py:

from quik_config import find_and_load

info = find_and_load(
    "info.yaml", # walks up folders until it finds a file with this name
    cd_to_filepath=True, # helpful if using relative paths
    fully_parse_args=True, # if you already have argparse, use parse_args=True instead
    show_help_for_no_args=False, # change if you want
)

print(info.config) # dictionary

Create info.yaml with a structure like this:

# names in parentheses are special, all other names are not!
# (e.g. add/extend this with any custom fields)
(project):
    # the local_data file will be auto-generated
    # (its for machine-specific data)
    # so probably git-ignore whatever path you pick
    (local_data): ./local_data.ignore.yaml
    
    # example profiles
    (profiles):
        (default):
            blah: "blah blah blah"
            mode: development # or production. Same thing really
            has_gpu: maybe
            constants:
                pi: 3 # its 'bout 3 
        
        PROFILE1:
            constants:
                e: 2.7182818285
        
        PROD:
            mode: production
            constants:
                pi: 3.1415926536
                problems: true

Then run it:

python ./config.py

Which will print out this config:

{
    "blah": "blah blah blah", # from (default)
    "mode": "development",    # from (default)
    "has_gpu": "maybe",       # from (default)
    "constants": {
        "pi": 3,              # from (default)
    },
}

Features

Builtin Help

python ./config.py --help --profiles

available profiles:
    - DEV
    - GPU
    - PROD

as cli argument:
   -- --profiles='["DEV"]'
   -- --profiles='["GPU"]'
   -- --profiles='["PROD"]'

    ---------------------------------------------------------------------------------
    QUIK CONFIG HELP
    ---------------------------------------------------------------------------------
    
    open the file below and look for "(profiles)" for more information:
        $PWD/info.yaml
    
    examples:
        python3 ./ur_file.py   --  --help --profiles
        python3 ./ur_file.py   --  --help key1
        python3 ./ur_file.py   --  --help key1:subKey
        python3 ./ur_file.py   --  --help key1:subKey key2
        python3 ./ur_file.py   --  --profiles='[YOUR_PROFILE, YOUR_OTHER_PROFILE]'
        python3 ./ur_file.py   --  thing1:"Im a new value"          part2:"10000"
        python3 ./ur_file.py   --  thing1:"I : cause errors"        part2:10000
        python3 ./ur_file.py   --  'thing1:"I : dont cause errors"  part2:10000
        python3 ./ur_file.py   --  'thing1:["Im in a list"]'
        python3 ./ur_file.py   --  'thing1:part_A:"Im nested"'
        python3 ./ur_file.py "I get sent to ./ur_file.py" --  part2:"new value"
        python3 ./ur_file.py "I get ignored" "me too"  --  part2:10000
    
    how it works:
        - the "--" is a required argument, quik config only looks after the --
        - given "thing1:10", "thing1" is the key, "10" is the value
        - All values are parsed as json/yaml
            - "true" is boolean true
            - "10" is a number
            - '"10"' is a string (JSON escaping)
            - '"10\n"' is a string with a newline
            - '[10,11,hello]' is a list with two numbers and an unquoted string
            - '{"thing": 10}' is a map/object
            - "blah blah" is an un-quoted string with a space. Yes its valid YAML
            - multiline values are valid, you can dump an whole JSON doc as 1 arg
        - "thing1:10" overrides the "thing1" in the (profiles) of the info.yaml
        - "thing:subThing:10" is shorthand, 10 is the value, the others are keys
          it will only override the subThing (and will create it if necessary)
        - '{"thing": {"subThing":10} }' is long-hand for "thing:subThing:10"
        - '"thing:subThing":10' will currently not work for shorthand (parse error)
    
    options:
        --help
        --profiles
    
    ---------------------------------------------------------------------------------
    
    your default top-level keys:
        - mode
        - has_gpu
        - constants
    your local defaults file:
        ./local_data.ignore.yaml
    your default profiles:
        - DEV
    
    ---------------------------------------------------------------------------------

Select Profiles from CLI

python ./config.py @PROFILE1

prints:

{
    "blah": "blah blah blah", # from (default)
    "mode": "development",    # from (default)
    "has_gpu": "maybe",       # from (default)
    "constants": {
        "pi": 3.1415926536,   # from (default)
        "e": 2.7182818285,    # from PROFILE1
    },
}

python ./config.py @PROFILE1 @PROD

prints:

{
    "blah": "blah blah blah", # from (default)
    "mode": "production",     # from PROD
    "has_gpu": "maybe",       # from (default)
    "constants": {
        "pi": 3.1415926536,   # from (default)
        "e": 2.7182818285,    # from PROFILE1
        "problems": True,     # from PROD
    },
}

Override Values from CLI

python ./config.py @PROFILE1 mode:custom constants:problems:99

prints:

{
    "blah": "blah blah blah", # from (default)
    "mode": "custom",         # from CLI
    "has_gpu": "maybe",       # from (default)
    "constants": {
        "pi": 3.1415926536,   # from (default)
        "e": 2.7182818285,    # from PROFILE1
        "problems": 99,       # from CLI
    },
}

Again but with really complicated arguments:
(each argument is parsed as yaml)

python ./run.py arg1 --  mode:my_custom_mode  'constants: { tau: 6.2831853072, pi: 3.1415926, reserved_letters: [ "C", "K", "i" ] }'

prints:

config: {
    "mode": "my_custom_mode", 
    "has_gpu": False, 
    "constants": {
        "pi": 3.1415926, 
        "tau": 6.2831853072, 
        "reserved_letters": ["C", "K", "i", ], 
    }, 
}
unused_args: ["arg1"]

Working Alongside Argparse (quick)

Remove fully_parse_args and replace it with just parse_args

info = find_and_load(
    "info.yaml",
    parse_args=True, # <- will only parse after -- 
)

Everthing in the CLI is the same, but it waits for -- For example:

# quik_config ignores arg1 --arg2 arg3, so argparse can do its thing with them
python ./config.py arg1 --arg2 arg3 -- @PROD

Working Alongside Argparse (advanced)

Arguments can simply be passed as a list of strings, which can be useful for running many combinations of configs.

info = find_and_load(
    "info.yaml",
    args=[ "@PROD" ],
)

Relative and absolute paths

Add them to the info.yaml

(project):
    (local_data): ./local_data.ignore.yaml
    
    # filepaths (relative to location of info.yaml)
    (path_to):
        this_file:       "./info.yaml"
        blah_file:       "./data/results.txt"
    
    # example profiles
    (profiles):
        (default):
            blah: "blah blah blah"

Access them in python

info = find_and_load("info.yaml")
info.path_to.blah_file
info.absolute_path_to.blah_file # nice when then PWD != folder of the info file

Import other yaml files

(project):
    (profiles):
        (GPU): !load_yaml_file ./profiles/gpu.yaml

Different Profiles For Different Machines

Lets say you've several machines and an info.yaml like this:

(project):
    (profiles):
        DEV:
            cores: 1
            database_ip: 192.168.10.10
            mode: dev
        LAPTOP:
            cores: 2
        DESKTOP:
            cores: 8
        UNIX:
            line_endings: "\n"
        WINDOWS:
            line_endings: "\r\n"
        PROD:
            database_ip: 183.177.10.83
            mode: prod
            cores: 32

And lets say you have a config.py like this:

from quik_config import find_and_load
info = find_and_load(
    "info.yaml",
    defaults_for_local_data=["DEV", ],
    # if the ./local_data.ignore.yaml doesnt exist,
    # => create it and add DEV as the default no-argument choice
)

Run the code once to get a ./local_data.ignore.yaml file.

Each machine gets to pick the profiles it defaults to.
So, on your Macbook you can edit the ./local_data.ignore.yaml to include something like the following:

(selected_profiles):
    - LAPTOP # the cores:2 will be used (instead of cores:1 from DEV)
    - UNIX   #     because LAPTOP is higher in the list than DEV
    - DEV

On your Windows laptop you can edit it and put:

(selected_profiles):
    - LAPTOP
    - WINDOWS
    - DEV

Command Line Arguments

If you have run.py like this:

from quik_config import find_and_load

info = find_and_load("info.yaml", parse_args=True)

print("config:",      info.config     )
print("unused_args:", info.unused_args)

# 
# call some other function you've got
# 
#from your_code import run
#run(*info.unused_args)

Example 0

Using the python file and config file above

python ./run.py

Running that will output:

config: {
    "mode": "development",
    "has_gpu": False,
    "constants": {
        "pi": 3
    }
}
unused_args: []

Example 1

Show help. This output can be overridden in the info.yaml by setting (help): under the (project): key.

python ./run.py -- --help

Note the -- is needed in front of the help.

You can also add show_help_for_no_args=True if you want that behavior.
Ex:

from quik_config import find_and_load
info = find_and_load(
    "info.yaml",
    show_help_for_no_args=True
    parse_args=True,
)

Example 2

Again but selecting some profiles

python ./run.py arg1 -- --profiles='[PROD]'
# or
python ./run.py arg1 -- @PROD

Output:

config: {
    "mode": "production",
    "has_gpu": False,
    "constants": {
        "pi": 3.1415926536,
        "problems": True,
    },
}
unused_args: ["arg1"]

Example 3

Again but with custom arguments:

python ./run.py arg1 --  mode:my_custom_mode  constants:tau:6.2831853072

config: {
    "mode": "my_custom_mode",
    "has_gpu": False,
    "constants": {
        "pi": 3,
        "tau": 6.2831853072,
    },
}
unused_args: ["arg1"]

Example 4

Again but with really complicated arguments:
(each argument is parsed as yaml)

python ./run.py arg1 --  mode:my_custom_mode  'constants: { tau: 6.2831853072, pi: 3.1415926, reserved_letters: [ "C", "K", "i" ] }'

prints:

config: {
    "mode": "my_custom_mode", 
    "has_gpu": False, 
    "constants": {
        "pi": 3.1415926, 
        "tau": 6.2831853072, 
        "reserved_letters": ["C", "K", "i", ], 
    }, 
}
unused_args: ["arg1"]