risc-lasagna 2.0.0rc1

A package that makes it easy to load configuration variables from multiple sources and build a single config which depends on them.

[![coverage report](https://gitdma.risc-software.at/risc_ds/lasagna/badges/master/coverage.svg)](https://gitdma.risc-software.at/risc_ds/lasagna/-/commits/master)
[![pipeline status](https://gitdma.risc-software.at/risc_ds/lasagna/badges/master/pipeline.svg)](https://gitdma.risc-software.at/risc_ds/lasagna/-/commits/master) 

# Lasagna

A package that makes it easy to load configuration variables from 
multiple sources and build a single config data class.

## The Idea
The idea is to 'stack' the sources like lasagna and read them from the 
bottom up. This means the layer at the bottom can set defaults and any 
layers on top of it overwrite these defaults. Which allows to build 
configurations from multiple sources with one clear end result.

The configuration must be wrapped up in a single data class. 
This enables the use of type hints as well as taking advantage of any other standard tooling
which operates on data classes. The library also supports nested data classes. This results in 
multiple *advantages*:

- When refactoring, for example adding a required field to the dataclass, 
without adding it to the default config, results in a error on startup.
- Type annotations can support the developer. No more guessing what is read.
- Automatically parsing simple types from configuration files and the environment
- No boilerplate for accessing dicts / ...
- Easily combine multiple sources for the environment **(data class instances, .env files, environment)**

## Setup

No additional packages are currently required to use `lasagna`. Simply install it.
```shell script
pip install RISC-SPEC-LASAGNA
```

## Development

For development simply clone the repository. To release a version change the version specifier
in the `setup.cfg` and commit the changes with a tag. The tag name should be `v<VERSION_IN_SETUP_CFG>` (e.g. v0.1.2).

## Tests

See task in `pyproject.toml`.

## Examples

The following section contains a few examples whiche illustrate how to use the package.

### Simple Example
To build your config you need to first specify a dataclass which 
encompasses the necessary values. The next step is to specify where the
values come from. A simple example is to use the `EnvironmentLayer`. Which 
reads all values from the environment and loads them in the dataclass.

```python
from dataclasses import dataclass
import os  # for testing purposes

import lasagna
import lasagna.layer as layer


@dataclass
class ApplicationConfig:
    app_host: str
    app_port: int


def load_application_config() -> ApplicationConfig:
    # add environment for testing, this should be set by your deployment tool
    os.environ['APP_HOST'] = 'localhost'
    os.environ['APP_PORT'] = '12345'
    return lasagna.build(ApplicationConfig, layer.EnvironmentLayer())
```
Executing this example results in a simple parsed application config
```python
>>> load_application_config()
ApplicationConfig(app_host='localhost', app_port=12345)
```

### Simple Example with default layer
Sometimes it is easier to provide 'defaults' for every dataclass. This can be accomplished by providing
a default layer using your dataclass. The same example as before can be rewritten to use a default 
layers using the `DataClassDefaultLayer` and the `AggregationLayer`. The `AggregationLayer` 
Simply combines the layers from top to bottom. The bottom most layer is overwritten
by every the top layers. 

```python
from dataclasses import dataclass
import os  # for testing purposes

import lasagna
import lasagna.layer as layer


@dataclass
class ApplicationConfig:
    app_host: str
    app_port: int


def load_application_config() -> ApplicationConfig:
    os.environ['APP_HOST'] = 'remote_host'
    return lasagna.build(ApplicationConfig, 
        [
            layer.EnvironmentLayer(),
            layer.DataClassDefaultLayer(
                ApplicationConfig('localhost', 12345)
            ),
        ],
    )
```
Running the example results in the `EnvironmentLayer` overwriting the `APP_HOST` variable of the
`DataClassDefaultLayer`.
```python
>>> load_application_config()
ApplicationConfig(app_host='remote_host', app_port=12345)
```

### Example with nested dataclass
For larger configurations it might be easier or convenient 
to assemble it from multiple dataclasses. This is possible by 
prefixing the existing variables in the environment with the name of 
the property. Here is an example:
```python
from dataclasses import dataclass
import os  # for testing purposes
from typing import Optional

import lasagna
import lasagna.layer as layer


@dataclass
class DatabaseConfig:
    host: str
    port: int
    user: Optional[str] = None
    password: Optional[str] = None


@dataclass
class ApplicationConfig:
    app_host: str
    app_port: int
    database_config: DatabaseConfig  # the name of this variable is used as a prefix


def load_application_config() -> ApplicationConfig:
    # set the environment variable with a prefix
    os.environ['DATABASE_CONFIG_HOST'] = 'db-host'
    return lasagna.build(
        ApplicationConfig,
        [
            layer.EnvironmentLayer(),
            layer.DataClassDefaultLayer(
                ApplicationConfig('localhost', 12345, DatabaseConfig('localhost', 5432))
            ),
        ],
    )
```

The example results in the `EnvironmentLayer` overwriting the defaults:

```python
>>> load_application_config()
ApplicationConfig(app_host='localhost', app_port=12345, database_config=DatabaseConfig(host='db-host', port=5432, user=None, password=None))
```

### Example with YAML and lists
For more complex configuration YAML is a better option. Yaml also supports lists and objects inside lists.
>HINT: If you use lists with nested dataclasses be aware that only a single type per list is supported.

```python
import tempfile  # for testing purposes
from dataclasses import dataclass, field
from pathlib import Path
from typing import Optional, List

import lasagna
import lasagna.layer as layer


@dataclass
class DatabaseConfig:
    host: str
    port: int
    user: Optional[str] = None
    password: Optional[str] = None


@dataclass
class HealthCheck:
    rate_in_ms: int
    url: str


@dataclass
class ApplicationConfig:
    host: str
    port: int
    elements: List[str]
    health_check: Optional[HealthCheck] = field(default=None)
    protocol: str = field(default='https')
    database_configs: Optional[List[DatabaseConfig]] = field(default=None)


__example_yaml = """
host: remote_host
health_check:
  rate_in_ms: 30
  url: localhost:6543
elements:
  - abc
  - def
  - ghi
database_configs:
  - host: remote_db
    port: 1234
  - host: remote_db
    port: 1234
    user: postgres
    password: super_safe_password
"""


def load_application_config() -> ApplicationConfig:
    # set the environment variable with a prefix
    with tempfile.NamedTemporaryFile() as temp:
        temp.write(bytes(__example_yaml, encoding='utf-8'))
        temp.flush()
        return lasagna.build(
            ApplicationConfig,
            [
                layer.YamlLayer(Path(temp.name)),
                layer.DataClassDefaultLayer(
                    ApplicationConfig(
                        'localhost', 12345, ['a', 'b', 'c'],
                        database_configs=[DatabaseConfig('localhost', 5432)]
                    )
                ),
            ],
        )
```

Executing the example results in this dataclass:
```
>>> load_application_config() 
ApplicationConfig(host='remote_host', port=12345, elements=['abc', 'def', 'ghi'], health_check=HealthCheck(rate_in_ms=30, url='localhost:6543'), protocol='https', database_configs=[DatabaseConfig(host='remote_db', port=1234, user=None, password=None), DatabaseConfig(host='remote_db', port=1234, user='postgres', password='super_safe_password')])
```

### FAQ

#### What happens if not enough variables are present to instantiate the data class?
In this case python raises an Error when the constructor is called ('... missing required attribute ...'). There are
two options to avoid the error:
1. Add the necessary variable to the environment
2. Make the attribute optional

#### I have a global prefix ('RISCSW') which is prepended before any variable, do i need to add this to my variable names?
No. The `build` methods defines a `variable_prefix` parameter which can be used to deal with such cases.
Other layers may also include a `variable_prefix` which is then applied to all of the variables in the layer.

#### I want to read .env files. Is this possible?
Yes. You can use the `DotEnvLayer` for this. This layer also supports comment in the .env file.
The file path has to be provided - Either make it fixed or write it to another environment variable, to set it
dynamically.

#### I want to read .yml files. Is this possible?
Yes. Use the YamlLayer. This has a few limitations (directly nested lists are not supported). 
But works nicely for most cases. 

#### I want to read complex types (such as np.array, Unions)
Currently only simple types are supported. If you want to contribute contact the maintainer. Optional is the only 
supported type.