kappaconfig 1.0.2

flexible yaml configuration framework

KappaConfig

KappaConfig is a configuration framework that allows you to define a full fletched configuration in yaml. Basic yaml and many yaml configuration frameworks are restrictive in how a yaml is processed. KappaConfig provides a rich extension to parsing yamls into primitive types.

With support for many use-cases out-of-the-box (which you can use, but don't have to):

• reuse defined dict/list/primities via cross reference or templating
• write python expressions in yaml
• use yamls from multiple sources to compose one large yaml during program execution

Install

pip install kappaconfig

Examples

This directory contains various examples on how KappaConfig can be used to create flexible and compact yaml files. <file>.yaml is the compact representation of a yaml that is resolved to <file>.result.yaml during runtime and subsequently used by the application. <file>.yaml abstracts away non-vital fields (for easy configuration design), while the resolved file (<file>.result.yaml) contains every detail for reproducability.

TODO example

TODO title

Lots of configurations consist of a small part that varies between different configurations and a large part that stays the same or only few variables of it change. For example: when running machine learning experiments the core of a dataset configuration (e.g. normalization, splits) stays largely the same but things like preprocessing might change between different experiments.

cifar10:
  train:
    split: train # use train split
    normalization: range # normalize to range [-1;1]
    filter: # use 45.000 samples for training
      index_from: 0
      index_to: 45000
  valid:
    split: train  # use train split (most datasets don't have a dedicated validation split)
    normalization: range
    filter: # use remaining 5.000 samples for validation
      index_from: 45000
      index_to: 50000
  test:
    split: test # use test split
    normalization: range

TODO continue example

Usage

import kappaconfig as kc

# load yaml from file
kc_hp = kc.from_file_uri("hyperparams.yaml")
# initialize default resolver
resolver = kc.DefaultResolver()
# resolve to primitive types
hp = resolver.resolve(hp)

Examples

Reference existing nodes

Inspired by OmegaConf/Hydra nodes can reference other nodes.

# input
batch_size: 64
train_loader:
  batch_size: ${batch_size}
test_loader:
  batch_size: ${batch_size}
---
# resolved
batch_size: 64
train_loader:
  batch_size: 64
test_loader:
  batch_size: 64

Write python code in yaml

# input 
seeds: ${eval:list(range(5))}
---
# resolved
seeds: [0, 1, 2, 3, 4] 

Parameterize templates

# warmup_cosine_schedule.yaml
vars: # vars is a special node that is removed after resolving a template
  # default values
  epochs: 100
  warmup_factor: 0.05
kind: sequential_schedule
sub_schedules:
- kind: warmup_schedule
  epochs: ${eval:${vars.epochs}*${vars.warmup_factor}}
- kind: cosine_schedule
  epochs: ${eval:${vars.epochs}*${eval:1-${vars.warmup_factor}}}
---
# template_default_params.yaml
optimizer:
  kind: SGD
  schedule:
    template: ${yaml:warmup_cosine_schedule.yaml}
---
# template_default_params.yaml resolved
optimizer:
  kind: SGD
  schedule:
    kind: sequential_schedule
    sub_schedules:
    - kind: warmup_schedule
      epochs: 5
    - kind: cosine_schedule
      epochs: 95
---
# template_custom_params.yaml
optimizer:
  kind: SGD
  schedule:
    template: ${yaml:warmup_cosine_schedule.yaml}
    template.vars.epochs: 200
---
# template_custom_params.yaml resolved
optimizer:
  kind: SGD
  schedule:
    kind: sequential_schedule
    sub_schedules:
    - kind: warmup_schedule
      epochs: 10
    - kind: cosine_schedule
      epochs: 190
---