paramflow 0.2.1

No project description provided

paramflow

A parameter and configuration management library motivated by training machine learning models and managing configuration for applications that require profiles and layered parameters. paramflow is designed for flexibility and ease of use, enabling seamless parameter merging from multiple sources. It also auto-generates a command-line argument parser and allows for easy parameter overrides.

Features

• Layered configuration: Merge parameters from files, environment variables, and command-line arguments.
• Immutable dictionary: Provides a read-only dictionary with attribute-style access.
• Profile support: Manage multiple sets of parameters. Layer the chosen profile on top of the default profile.
• Layered meta-parameters: paramflow loads its own configuration using layered approach.
• Convert types: Convert types during merging using target parameters as a reference for type conversions.
• Generate argument parser: Use parameters defined in files as a reference for generating argparse parser.

Usage

Install:

pip install paramflow

Install with .env support:

pip install "paramflow[dotenv]"

params.toml

[default]
learning_rate = 0.001
batch_size = 64

app.py

import paramflow as pf
params = pf.load('params.toml')
print(params.learning_rate)

python app.py --help
usage: app.py [-h] [--learning_rate LEARNING_RATE] [--batch_size BATCH_SIZE]

options:
  -h, --help            show this help message and exit
  --learning_rate LEARNING_RATE
                        learning_rate = 0.001
  --batch_size BATCH_SIZE
                        batch_size = 64

Meta-parameter Layering

Meta-parameter layering controls how paramflow.load reads its own configuration.

Layering order:

1. paramflow.load arguments.
2. Environment variables (default prefix P_).
3. Command-line arguments (via argparse).

Activate profile using command-line arguments:

python print_params.py --profile dqn-adam

Activate profile using environment variable:

P_PROFILE=dqn-adam python print_params.py

Parameter Layering

Parameter layering merges parameters from multiple sources.

Layering order:

1. Configuration files (.toml, .yaml, .ini, .json).
2. .env file.
3. Environment variables (default prefix P_).
4. Command-line arguments (via argparse).

Layering order can be customized via source argument to param.flow.

params = pf.load(['params.toml', 'env', '.env', 'args'])

Overwrite parameter value:

python print_params.py --profile dqn-adam --learning_rate 0.0002

ML hyper-parameters profiles

params.toml

[default]
learning_rate = 0.00025
batch_size = 32
optimizer_class = 'torch.optim.RMSprop'
optimizer_kwargs = { momentum = 0.95 }
random_seed = 13

[adam]
learning_rate = 1e-4
optimizer_class = 'torch.optim.Adam'
optimizer_kwargs = {}

Activating adam profile

python app.py --profile adam

will result in overwriting default learning rate with 1e-4, default optimizer class with torch.optim.Adam and default optimizer arguments with and empty dict.

Development stages profiles

Profiles can be used to manage software development stages. params.toml:

[default]
debug = true
database_url = "mysql://localhost:3306/myapp"

[dev]
database_url = "mysql://dev:3306/myapp"

[prod]
debug = false
database_url = "mysql://prod:3306/myapp"

Activate prod profile:

export P_PROFILE=dev
python app.py