No project description provided
Project description
ConfigSys
Very simple yet powerful configuration system for machine learning projects. The whole thing is around 170 lines of code, give it a look!
Provides a simple way to use nested dataclasses to specify an arbitrary complex configuration. Configs can be easily serialized to and from yaml files and back to nested dataclasses.
see configsys/example.py
For a more complex example see this repo.
Core Beliefs
Specifying configs in python is way better than raw yaml text files.
- You get all the help of the IDE (e.g. autocomplete, type checking, refactoring, etc.)
- You can use python logic (for loops, string replacement, functions, itertools, etc.) to decide your hyper-parameters and experiments.
The CLI is a terrible way to specify hyper-parameters or complex configs
- CLI configuration is basically an attempt at taking a 1 line string and serializing it into a python nested datastructure
- It is much easier to just create a python file which configures and runs (or launches) your jobs.
Install
pip install configsys
Example Usage
from dataclasses import dataclass
from configsys.config import ConfigMixin
# We recommend using kw_only=True to make sure Configs are specified explicitly
class ImageDataLoader:
@dataclass(kw_only=True)
class Config(ConfigMixin):
_target_ = "configsys.example.ImageDataLoader"
batch_size: int
shuffle: bool
def __init__(self, config: Config):
self.config = config
def get_data(self):
return dict(x=[2, 3], y=[4, 5])
class TextDataLoader:
@dataclass(kw_only=True)
class Config(ConfigMixin):
_target_ = "configsys.example.TextDataLoader"
batch_size: int
shuffle: bool
def __init__(self, config: Config):
self.config = config
def get_data(self, x):
return dict(x=['hello', 'world'])
class MLP:
@dataclass(kw_only=True)
class Config(ConfigMixin):
_target_ = "configsys.example.MLP"
n_layers: int
n_hidden: int
def __init__(self, config: Config):
self.config = config
def forward(self, x):
return x
class Transformer:
@dataclass(kw_only=True)
class Config(ConfigMixin):
_target_ = "configsys.example.Transformer"
n_heads: int
def __init__(self, config: Config):
self.config = config
def forward(self, x):
return x
# Configs do not necessarily have to be an attribute of their target class
# This is usually the case when the target is class that you did not write
@dataclass(kw_only=True)
class TrainerConfig(ConfigMixin):
_target_ = "configsys.example.Trainer"
dataloader: ImageDataLoader.Config | TextDataLoader.Config
model: MLP.Config | Transformer.Config
learning_rate: float
steps: int
class Trainer:
def __init__(self, config: TrainerConfig):
self.config = config
def train(self):
print("{self} training...")
print(self.config.learning_rate)
print(self.config.steps)
def main(outdir):
for model_config in [MLP.Config(n_layers=3, n_hidden=128),
Transformer.Config(n_heads=6)]:
config = TrainerConfig(
dataloader=ImageDataLoader.Config(batch_size=32, shuffle=True),
model=model_config,
learning_rate=0.1,
steps=100,
)
model = config.model.instantiate_target()
dataloader = config.dataloader.instantiate_target()
trainer = config.instantiate_target() # equivalent to trainer = Trainer(config=trainer_config)
trainer.train()
config.to_yaml_file(f"{outdir}/trainer_config.yaml")
trainer_config2 = TrainerConfig.from_yaml_file(f"{outdir}/trainer_config.yaml")
assert config == trainer_config2
Publishing
poetry build
poetry publish
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file configsys-0.3.0.tar.gz
.
File metadata
- Download URL: configsys-0.3.0.tar.gz
- Upload date:
- Size: 5.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.6.1 CPython/3.11.6 Darwin/22.6.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 423919281fe84e72d7cf7613c273c090ae9d29649ada6a0ecea53502e4f732c4 |
|
MD5 | 2439673cb45cd82f2d543ef81a304619 |
|
BLAKE2b-256 | 80aebb98a5a6ebba45df244be39cb0e528186dd7971cfdf1eaf318e99580267e |
File details
Details for the file configsys-0.3.0-py3-none-any.whl
.
File metadata
- Download URL: configsys-0.3.0-py3-none-any.whl
- Upload date:
- Size: 7.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.6.1 CPython/3.11.6 Darwin/22.6.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6129d8522741e211d36f9cddc5b3202814d1ea4759d3511f7766422036d62dfc |
|
MD5 | da0d2efa4d4fd785c64f3f96e585642d |
|
BLAKE2b-256 | fb8b863299d412e6e4f3dda08abecbaec850a60bac81a77a643f8c6a57643bbd |