quik-config 1.9.1

Project config files

What is this?

A config system that doesn't waste your time

• all values in the hierarchy can be overridden with CLI args
• select multiple profiles from (ex: GPU & DEV or UNIX & GPU & PROD) from the CLI, profile settings get merged recursively
• configs supports secrets (unsynced file along side git-committed config)
• provides a consistent central way to handle filepaths
• profiles can inherit other profiles
• default works along side argparse, but also can replace it
• can combine/import multiple config files

How do I use this?

pip install quik-config

In a config.py:

from quik_config import find_and_load

info = find_and_load(
    "info.yaml", # walks up folders until it finds a file with this name
    cd_to_filepath=True, # helpful if using relative paths
    fully_parse_args=True, # if you already have argparse, use parse_args=True instead
    show_help_for_no_args=False, # change if you want
)
# info.path_to
# info.absolute_path_to
# info.unused_args
# info.secrets
# info.available_profiles
# info.selected_profiles
# info.root_path
# info.project
# info.local_data
# info.as_dict
print(info.config) # dictionary

Create info.yaml with a structure like this:

# names in parentheses are special, all other names are not!
# (e.g. add/extend this with any custom fields)
(project):
    # the local_data file will be auto-generated
    # (its for machine-specific data)
    # so probably git-ignore whatever path you pick
    (local_data): ./local_data.ignore.yaml
    
    # example profiles
    (profiles):
        (default):
            blah: "blah blah blah"
            mode: development # or production. Same thing really
            has_gpu: maybe
            constants:
                pi: 3 # its 'bout 3 
        
        PROFILE1:
            constants:
                e: 2.7182818285
        
        PROD:
            mode: production
            constants:
                pi: 3.1415926536
                problems: true

Then run it:

python ./config.py

Which will print out this config:

{
    "blah": "blah blah blah", # from (default)
    "mode": "development",    # from (default)
    "has_gpu": "maybe",       # from (default)
    "constants": {
        "pi": 3,              # from (default)
    },
}

Features

Builtin Help

python ./config.py --help --profiles

available profiles:
    - DEV
    - GPU
    - PROD

as cli argument:
   -- --profiles='["DEV"]'
   -- --profiles='["GPU"]'
   -- --profiles='["PROD"]'

    ---------------------------------------------------------------------------------
    QUIK CONFIG HELP
    ---------------------------------------------------------------------------------
    
    open the file below and look for "(profiles)" for more information:
        $PWD/info.yaml
    
    examples:
        python3 ./ur_file.py   --  --help --profiles
        python3 ./ur_file.py   --  --help key1
        python3 ./ur_file.py   --  --help key1:subKey
        python3 ./ur_file.py   --  --help key1:subKey key2
        python3 ./ur_file.py   --  --profiles='[YOUR_PROFILE, YOUR_OTHER_PROFILE]'
        python3 ./ur_file.py   --  thing1:"Im a new value"          part2:"10000"
        python3 ./ur_file.py   --  thing1:"I : cause errors"        part2:10000
        python3 ./ur_file.py   --  'thing1:"I : dont cause errors"  part2:10000
        python3 ./ur_file.py   --  'thing1:["Im in a list"]'
        python3 ./ur_file.py   --  'thing1:part_A:"Im nested"'
        python3 ./ur_file.py "I get sent to ./ur_file.py" --  part2:"new value"
        python3 ./ur_file.py "I get ignored" "me too"  --  part2:10000
    
    how it works:
        - the "--" is a required argument, quik config only looks after the --
        - given "thing1:10", "thing1" is the key, "10" is the value
        - All values are parsed as json/yaml
            - "true" is boolean true
            - "10" is a number
            - '"10"' is a string (JSON escaping)
            - '"10\n"' is a string with a newline
            - '[10,11,hello]' is a list with two numbers and an unquoted string
            - '{"thing": 10}' is a map/object
            - "blah blah" is an un-quoted string with a space. Yes its valid YAML
            - multiline values are valid, you can dump an whole JSON doc as 1 arg
        - "thing1:10" overrides the "thing1" in the (profiles) of the info.yaml
        - "thing:subThing:10" is shorthand, 10 is the value, the others are keys
          it will only override the subThing (and will create it if necessary)
        - '{"thing": {"subThing":10} }' is long-hand for "thing:subThing:10"
        - '"thing:subThing":10' will currently not work for shorthand (parse error)
    
    options:
        --help
        --profiles
    
    ---------------------------------------------------------------------------------
    
    your default top-level keys:
        - mode
        - has_gpu
        - constants
    your local defaults file:
        ./local_data.ignore.yaml
    your default profiles:
        - DEV
    
    ---------------------------------------------------------------------------------

Select Profiles from CLI

python ./config.py @PROFILE1

prints:

{
    "blah": "blah blah blah", # from (default)
    "mode": "development",    # from (default)
    "has_gpu": "maybe",       # from (default)
    "constants": {
        "pi": 3.1415926536,   # from (default)
        "e": 2.7182818285,    # from PROFILE1
    },
}

python ./config.py @PROFILE1 @PROD

prints:

{
    "blah": "blah blah blah", # from (default)
    "mode": "production",     # from PROD
    "has_gpu": "maybe",       # from (default)
    "constants": {
        "pi": 3.1415926536,   # from (default)
        "e": 2.7182818285,    # from PROFILE1
        "problems": True,     # from PROD
    },
}

Override Values from CLI

python ./config.py @PROFILE1 mode:custom constants:problems:99

prints:

{
    "blah": "blah blah blah", # from (default)
    "mode": "custom",         # from CLI
    "has_gpu": "maybe",       # from (default)
    "constants": {
        "pi": 3.1415926536,   # from (default)
        "e": 2.7182818285,    # from PROFILE1
        "problems": 99,       # from CLI
    },
}

Again but with really complicated arguments:
(each argument is parsed as yaml)

python ./run.py arg1 --  mode:my_custom_mode  'constants: { tau: 6.2831853072, pi: 3.1415926, reserved_letters: [ "C", "K", "i" ] }'

prints:

config: {
    "mode": "my_custom_mode", 
    "has_gpu": False, 
    "constants": {
        "pi": 3.1415926, 
        "tau": 6.2831853072, 
        "reserved_letters": ["C", "K", "i", ], 
    }, 
}
unused_args: ["arg1"]

Working Alongside Argparse (quick)

Remove fully_parse_args and replace it with just parse_args

info = find_and_load(
    "info.yaml",
    parse_args=True, # <- will only parse after -- 
)

Everthing in the CLI is the same, but it waits for -- For example:

# quik_config ignores arg1 --arg2 arg3, so argparse can do its thing with them
python ./config.py arg1 --arg2 arg3 -- @PROD

Working Alongside Argparse (advanced)

Arguments can simply be passed as a list of strings, which can be useful for running many combinations of configs.

info = find_and_load(
    "info.yaml",
    args=[ "@PROD" ],
)

Relative and absolute paths

Add them to the info.yaml

(project):
    (local_data): ./local_data.ignore.yaml
    
    # filepaths (relative to location of info.yaml)
    (path_to):
        this_file:       "./info.yaml"
        blah_file:       "./data/results.txt"
    
    # example profiles
    (profiles):
        (default):
            blah: "blah blah blah"

Access them in python

info = find_and_load("info.yaml")
info.path_to.blah_file
info.absolute_path_to.blah_file # nice when then PWD != folder of the info file

Import other yaml files

You can import multiple profiles by specifying profile sources.
NOTE: the last profile source will override (merge keys) with the previous ones, but the main config will take the priority over any/all profile sources.

(project):
    (profile_sources):
        - ./comments.yaml
        - ./camera_profiles.yaml
    
    (profiles):
        # you can also load a single profile as a file
        (GPU): !load_yaml_file ./profiles/gpu.yaml

Different Profiles For Different Machines

Lets say you've several machines and an info.yaml like this:

(project):
    (profiles):
        DEV:
            cores: 1
            database_ip: 192.168.10.10
            mode: dev
        LAPTOP:
            cores: 2
        DESKTOP:
            cores: 8
        UNIX:
            line_endings: "\n"
        WINDOWS:
            line_endings: "\r\n"
        PROD:
            database_ip: 183.177.10.83
            mode: prod
            cores: 32

And lets say you have a config.py like this:

from quik_config import find_and_load
info = find_and_load(
    "info.yaml",
    defaults_for_local_data=["DEV", ],
    # if the ./local_data.ignore.yaml doesnt exist,
    # => create it and add DEV as the default no-argument choice
)

Run the code once to get a ./local_data.ignore.yaml file.

Each machine gets to pick the profiles it defaults to.
So, on your Macbook you can edit the ./local_data.ignore.yaml to include something like the following:

(selected_profiles):
    - LAPTOP # the cores:2 will be used (instead of cores:1 from DEV)
    - UNIX   #     because LAPTOP is higher in the list than DEV
    - DEV

On your Windows laptop you can edit it and put:

(selected_profiles):
    - LAPTOP
    - WINDOWS
    - DEV

Command Line Arguments

If you have run.py like this:

from quik_config import find_and_load

info = find_and_load("info.yaml", parse_args=True)

print("config:",      info.config     )
print("unused_args:", info.unused_args)

# 
# call some other function you've got
# 
#from your_code import run
#run(*info.unused_args)

Example 0

Using the python file and config file above

python ./run.py

Running that will output:

config: {
    "mode": "development",
    "has_gpu": False,
    "constants": {
        "pi": 3
    }
}
unused_args: []

Example 1

Show help. This output can be overridden in the info.yaml by setting (help): under the (project): key.

python ./run.py -- --help

Note the -- is needed in front of the help.

You can also add show_help_for_no_args=True if you want that behavior.
Ex:

from quik_config import find_and_load
info = find_and_load(
    "info.yaml",
    show_help_for_no_args=True
    parse_args=True,
)

Example 2

Again but selecting some profiles

python ./run.py arg1 -- --profiles='[PROD]'
# or
python ./run.py arg1 -- @PROD

Output:

config: {
    "mode": "production",
    "has_gpu": False,
    "constants": {
        "pi": 3.1415926536,
        "problems": True,
    },
}
unused_args: ["arg1"]

Example 3

Again but with custom arguments:

python ./run.py arg1 --  mode:my_custom_mode  constants:tau:6.2831853072

config: {
    "mode": "my_custom_mode",
    "has_gpu": False,
    "constants": {
        "pi": 3,
        "tau": 6.2831853072,
    },
}
unused_args: ["arg1"]

Example 4

Again but with really complicated arguments:
(each argument is parsed as yaml)

python ./run.py arg1 --  mode:my_custom_mode  'constants: { tau: 6.2831853072, pi: 3.1415926, reserved_letters: [ "C", "K", "i" ] }'

prints:

config: {
    "mode": "my_custom_mode", 
    "has_gpu": False, 
    "constants": {
        "pi": 3.1415926, 
        "tau": 6.2831853072, 
        "reserved_letters": ["C", "K", "i", ], 
    }, 
}
unused_args: ["arg1"]

Auto Generate Config-Specific Log Folders

If you add path_from_config_to_log_folder="../logs", as an argument to find_and_load

• The config is hashed
• A folder will be generated for each config
• A "run_index" is created (incrementes by 1 for each run with the same config)
• A nested folder will be created for each run
• info.unique_run_path is probably what you care about the most; the absolute path to log folder for this run
• info.unique_config_path is an absolute path to the unique config folder (summary stats for a specific config)

By default:

• config path is ./logs/[created_date]__[hash_of_config]/specific_config.yaml
• run path is ./logs/[created_date]__[hash_of_config]/[run_index]__[date]__[git_commit]/
  • run index is padded with zeros (ex: 0001)
  • If you don't have git, the __[git_commit] won't be there
• The ./logs/[created_date]__[hash_of_config]/running_data.yaml will look like this:

# guaranteed one line per run, each line is JSON
- {"run_index": 0, "git_commit": "1a69c85b61dc52eac7b1edbe13dd78ebbe46ece5", "start_time": "2024-07-01T15:22:09.327494", "inital_run_name": "run/hi2_0000"}
- {"run_index": 1, "git_commit": "1a69c85b61dc52eac7b1edbe13dd78ebbe46ece5", "start_time": "2024-07-01T15:22:09.341325", "inital_run_name": "0001__2024-07-01--15-22__1a69c8"}
- {"run_index": 2, "git_commit": "1a69c85b61dc52eac7b1edbe13dd78ebbe46ece5", "start_time": "2024-07-01T15:22:09.541018", "inital_run_name": "0002__2024-07-01--15-22__1a69c8"}

You can customize almost everything:

• You can rename* any of the folders manually  (and it'll keep working)
  • *the config path NEEDS to end with the hash of the config, but thats the only constraint
  • Ex: ./logs/your_name[hash_of_config]/
• padding of the run index (ex: 0001) with run_index_padding=4,
• the length of the config hash (ex: 6) with config_hash_length=6,
• the length of the git commit (ex: 6) with default_git_commit_length=6,
• using config_renamer, run_namer, and config_initial_namer like so:

from quik_config import find_and_load

config_renamer1 = lambda **_: "blahblah_"
config_renamer2 = lambda config, **_: f"{config.experiment_name}_"
config_renamer3 = lambda config, time, **_: f"{config.experiment_name}_{time}_"
config_renamer4 = lambda config, date, time, **_: f"{config.experiment_name}_{date}_{time}_"
config_renamer5 = lambda config, date, **_: f"{config.experiment_name}_{date}_"
config_renamer6 = lambda run_index, **_: f"{time}_{run_index}__"
# NOTE: run_index is a string, because its padded-out with zeros

run_namer1 = lambda **_: "blahblah"
run_namer2 = lambda run_index, **_: f"{run_index}"
run_namer3 = lambda run_index, **_: f"runs/{run_index}"
run_namer4 = lambda info, run_index, date, time, **_: f"runs_{date}/{run_index}"
# NOTE: for "run_namer", its okay to return "thing/{run_index}"
#       The sub-folders will be created

# only use this if you want to manually rename these folders
# (config_renamer would undo your manual rename)
config_initial_namer1 = lambda **_: "blahblah_"
config_initial_namer2 = lambda info, time, **_: f"0_{time}_"

# 
# basic example
# 
info = find_and_load(
    "info.yaml",
    path_from_config_to_log_folder="../logs",
    run_index_padding=4,
    config_hash_length=6,
    default_git_commit_length=6,
    config_renamer: lambda date, **_: f"{date}_",
    run_namer: lambda info, run_index, date, time, **_: f"{run_index}_{date}_{time}_",
    config_initial_namer: lambda info, time, **_: f"{time}_",
    config_renamer: lambda info, **_: f"blahblah_", # can "update" the name
)
print(info.this_run.run_index) # starts at 0 if this is a new/unique config
print(info.this_run.git_commit) # full 40-character git commit hash
print(info.this_run.start_time) # datetime.datetime.now() object (not a string)

print(info.unique_run_path)
print(info.unique_config_path)
print(info.log_folder)

# 
# lambda arg options
# 
    lambda date,                   **_: "blah"    # string: "2024-07-01"
    lambda time,                   **_: "blah"    # string: "17-52" (NOTE: "-" because colon is not valid filename character)
    lambda datetime,               **_: "blah"    # string: '2024-06-28--17-26'
    lambda now,                    **_: "blah"    # datetime.datetime object
    lambda config_hash,            **_: "blah"    # string: "lak4fa"
    lambda git_commit,             **_: "blah"    # 40-character git commit hash as a string
    lambda time_with_seconds,      **_: "blah"    # string: "17:52:00"
    lambda time_with_milliseconds, **_: "blah"    # string: "17:52:00.000"
    lambda year,                   **_: "blah"    # int
    lambda month,                  **_: "blah"    # int
    lambda day,                    **_: "blah"    # int
    lambda hour,                   **_: "blah"    # int
    lambda minute,                 **_: "blah"    # int
    lambda second,                 **_: "blah"    # int
    lambda unix_seconds,           **_: "blah"    # int
    lambda unix_milliseconds,      **_: "blah"    # int