Skip to main content

Python configuration file and command line executable to run a script with

Project description

Python Configuration and a command line Runner

CircleCI codecov Documentation Status image

Python configuration file and command line executable to run a script with.

Why a python file as configuration?

  • Configuration of any complexity
  • No need to serialize the configuration
  • No need other meta-languages for the configuration

Usage

In the code

Configuration file (e.g. config.py):

num_epochs = 100
batch_size = 256

model = resnet18(10)
train_loader = get_train_loader("/path/to/dataset", batch_size=batch_size)
unsup_dataloader = get_train_unsup_loader("/path/to/unsup_dataset", batch_size=batch_size)
...

Script file (e.g. training.py):

from torch.utils.data import DataLoader
from py_config_runner import ConfigObject, TrainvalConfigSchema


class SSLTrainvalConfigSchema(TrainvalConfigSchema):

    unsup_dataloader: DataLoader



def training(config):
    # ...
    print(config.config_filepath)
    print(config.output_path)
    print(config.num_epochs)
    print(config.model)
    print(len(config.train_loader))


def main():

    config_filepath = "/path/to/config.py"
    config = ConfigObject(config_filepath)

    SSLTrainvalConfigSchema.validate(config)

    # Add more things at runtime    
    config.output_path = "/tmp/output"

    training(config)

With launcher

cd /path/to/my/project
py_config_runner scripts/training.py configs/train/baseline.py

or

cd /path/to/my/project
python -u -m py_config_runner scripts/training.py configs/train/baseline.py

or if your specific launcher requires only python script files:

cd /path/to/my/project
python -m special_launcher `py_config_runner_script` scripts/training.py configs/train/baseline.py

The only condition on the script file is it should contain run(config, **kwargs) callable method. Additionally, argument kwargs contains logger (e.g. kwargs['logger']) and local_rank (e.g. kwargs['logger']) for distributed computations.

No restrictions are applied on the configuration file. It is user's responsibility to provide the script file that can consume given configuration file. Provided configuration file is loaded as python module and exposed into the script as the module named config.

Examples

Installation

pip install py-config-runner

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

py_config_runner-0.3.2.tar.gz (11.2 kB view hashes)

Uploaded source

Built Distribution

py_config_runner-0.3.2-py2.py3-none-any.whl (11.9 kB view hashes)

Uploaded py2 py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page