nshconfig 0.30.1.dev0

Fully typed configuration management, powered by Pydantic

nshconfig

Fully typed configuration management, powered by Pydantic

Table of Contents

• Motivation
• Installation
• Usage
• Features
  • Draft Configs
    • Motivation
    • Usage Guide
  • Configuration Formats
    • JSON Support
    • YAML Support
    • Dictionary Support
    • Schema References
  • Dynamic Type Registry
    • Basic Usage
    • Plugin System Support
    • Key Features
    • When to Use
  • Configuration Codegen
    • Basic Usage
    • Features
      • Type-Safe Exports
      • TypedDict Generation
      • JSON Schema Generation
    • Command Line Options
    • Use Cases
  • MISSING Constant
    • Motivation
  • Seamless Integration with PyTorch Lightning
• Credit
• Contributing
• License

Motivation

As a machine learning researcher, I often found myself running numerous training jobs with various hyperparameters for the models I was working on. Keeping track of these parameters in a fully typed manner became increasingly important. While the excellent pydantic library provided most of the functionality I needed, I wanted to add a few extra features to streamline my workflow. This led to the creation of nshconfig.

Installation

You can install nshconfig via pip:

pip install nshconfig

Usage

While the primary use case for nshconfig is in machine learning projects, it can be used in any Python project where you need to store configurations in a fully typed manner.

Here's a basic example of how to use nshconfig:

import nshconfig as C

class MyConfig(C.Config):
    field1: int
    field2: str
    field3: C.AllowMissing[float] = C.MISSING

config = MyConfig.draft()
config.field1 = 42
config.field2 = "hello"
final_config = config.finalize()

print(final_config)

For more advanced usage and examples, please refer to the documentation.

Features

• Draft configs for a more Pythonic configuration creation experience
• Dynamic type registry for building extensible, plugin-based systems
• MISSING constant for better handling of optional fields
• Seamless integration with PyTorch Lightning

Draft Configs

Draft configs allow for a nicer API when creating configurations. Instead of relying on JSON or YAML files, you can create your configs using pure Python:

config = MyConfig.draft()

# Set some values
config.a = 10
config.b = "hello"

# Finalize the config
config = config.finalize()

This approach enables a more intuitive and expressive way of defining your configurations.

Motivation

The primary motivation behind draft configs is to provide a cleaner and more Pythonic way of creating configurations. By leveraging the power of Python, you can define your configs in a more readable and maintainable manner.

Usage Guide

1. Create a draft config using the draft() class method:

   config = MyConfig.draft()
   

2. Set the desired values on the draft config:

   config.field1 = value1
   config.field2 = value2
   

3. Finalize the draft config to obtain the validated configuration:

   final_config = config.finalize()
   

Based on your code and its functionality, I'll write a new section for the README that showcases the Registry feature. Here's my suggested addition:

Configuration Formats

nshconfig supports multiple formats for creating and serializing configurations, making it flexible for different use cases:

Python File/Module Support

nshconfig supports loading configurations directly from Python files and modules, offering a more dynamic way to create configurations:

from mypackage import ModelConfig

# Load from Python file
config1 = ModelConfig.from_python_file("model_config.py")

# Load from Python module
config2 = ModelConfig.from_python_module("myapp.configs.model")

The Python file or module should export either:

• A __config__ variable containing the configuration
• A __create_config__ function that returns the configuration

Let's assume that mypackage is a Python package with a ModelConfig configuration class with the following definition:

# mypackage/__init__.py

class ModelConfig(C.Config):
    hidden_size: int
    num_layers: int

Then you can load the configuration from the Python file like this:

# model_config.py

from mypackage import ModelConfig

# Option 1: Using __config__ variable
__config__ = ModelConfig(
    hidden_size=256,
    num_layers=4
)

# You can also return a dictionary instead of an instance:
__config__ = {
    "hidden_size": 256,
    "num_layers": 4
}

# Option 2: Using __create_config__ function
def __create_config__():
    return ModelConfig(
        hidden_size=256,
        num_layers=4
    )

You can also load multiple configurations from a single file using the list variants:

# Load multiple configs from Python file
configs = ModelConfig.from_python_file_list("model_configs.py")

# Load multiple configs from Python module
configs = ModelConfig.from_python_module_list("myapp.configs.models")

The Python file should export either:

• A __configs__ variable containing a list of configurations
• A __create_configs__ function that returns an iterable of configurations

Example multi-configuration file:

# model_configs.py

from mypackage import ModelConfig

# Option 1: Using __configs__ variable
__configs__ = [
    ModelConfig(hidden_size=256, num_layers=4),
    ModelConfig(hidden_size=512, num_layers=8)
]

# Option 2: Using __create_configs__ function
def __create_configs__():
    yield ModelConfig(hidden_size=256, num_layers=4)
    yield ModelConfig(hidden_size=512, num_layers=8)

The configurations can be provided either as instances of the configuration class or as dictionaries. This makes Python files/modules a flexible way to create configurations, especially when you need to compute values dynamically or reuse configuration components.

JSON Support

You can create and save configurations using JSON:

import nshconfig as C

class ModelConfig(C.Config):
    hidden_size: int
    num_layers: int

# Create from JSON string
config1 = ModelConfig.from_json_str('{"hidden_size": 256, "num_layers": 4}')

# Load from JSON file
config2 = ModelConfig.from_json_file("model_config.json")

# Save to JSON string
json_str = config1.to_json_str()

# Save to JSON file
config1.to_json_file("model_config.json")

YAML Support

YAML support requires installing the pydantic-yaml package, either by installing the yaml extra:

pip install "nshconfig[extra]" # Installs all extras
pip install "nshconfig[yaml]" # Installs only the YAML extra

or by installing the package directly:

pip install pydantic-yaml

Then you can work with YAML formats:

class ModelConfig(C.Config):
    hidden_size: int
    num_layers: int

# Create from YAML string
config1 = ModelConfig.from_yaml_str("""
hidden_size: 256
num_layers: 4
""")

# Load from YAML file
config2 = ModelConfig.from_yaml("model_config.yaml")

# Save to YAML string
yaml_str = config1.to_yaml_str()

# Save to YAML file
config1.to_yaml_file("model_config.yaml")

Dictionary Support

You can also create configurations directly from Python dictionaries:

class ModelConfig(C.Config):
    hidden_size: int
    num_layers: int

# Create from dictionary
config = ModelConfig.from_dict({
    "hidden_size": 256,
    "num_layers": 4
})

# Convert to dictionary
config_dict = config.to_dict()

Schema References

When saving to JSON or YAML, you can include schema references that enable better IDE support:

# Include schema reference in JSON
config.to_json_file("config.json", with_schema=True)

# Include schema reference in YAML
config.to_yaml_file("config.yaml", with_schema=True)

The schema references help IDEs provide autocompletion and validation when editing the configuration files.

Dynamic Type Registry

The Registry system enables dynamic registration of subtypes, allowing you to create extensible configurations that can be enhanced at runtime. This is particularly useful for plugin systems or any scenario where you want to allow users to add new types to your configuration schema.

Basic Usage

Here's a simple example of using the Registry system:

import nshconfig as C
from abc import ABC, abstractmethod
from typing import Literal, Annotated

# Define your base configuration
class AnimalConfig(C.Config, ABC):
    @abstractmethod
    def make_sound(self) -> str: ...

# Create a registry for animal types
animal_registry = C.Registry(
    AnimalConfig,
    discriminator="type"  # Discriminator field to determine the type of the config
)

# Register some implementations
@animal_registry.register
class DogConfig(AnimalConfig):
    type: Literal["dog"] = "dog"
    name: str

    def make_sound(self) -> str:
        return "Woof!"

@animal_registry.register
class CatConfig(AnimalConfig):
    type: Literal["cat"] = "cat"
    name: str

    def make_sound(self) -> str:
        return "Meow!"

# Create a config that uses the registry
@animal_registry.rebuild_on_registers
class ProgramConfig(C.Config):
    animal: Annotated[AnimalConfig, animal_registry.DynamicResolution()]

# Use it!
def main(program_config: ProgramConfig):
    print(program_config.animal.make_sound())

main(ProgramConfig(animal=DogConfig(name="Buddy")))  # Output: Woof!
main(ProgramConfig(animal=CatConfig(name="Whiskers")))  # Output: Meow!

Plugin System Support

The real power of the Registry system comes when building extensible applications. Other packages can register new types with your registry:

# In a separate plugin package:
@animal_registry.register
class BirdConfig(AnimalConfig):
    type: Literal["bird"] = "bird"
    name: str
    wingspan: float

    def make_sound(self) -> str:
        return "Tweet!"

# This works automatically, even though BirdConfig was registered after ProgramConfig was defined
main(ProgramConfig(animal=BirdConfig(name="Tweety", wingspan=1.2)))  # Output: Tweet!

Key Features

1. Type Safety: Full type checking support with discriminated unions
2. Runtime Extensibility: Register new types even after config classes are defined
3. Validation: Automatic validation of discriminator fields and type matching
4. Plugin Support: Perfect for building extensible applications
5. Pydantic Integration: Seamless integration with Pydantic's validation system

When to Use

The Registry system is particularly useful when:

• Building plugin systems that need configuration support
• Creating extensible applications where users can add new types
• Working with configurations that need to handle different variants of a base type
• Implementing pattern matching or strategy patterns with configuration support

Configuration Codegen

The configuration codegen feature provides tools to generate clean, importable interfaces and type definitions for your configurations. This is particularly useful for:

1. Creating type-safe client libraries from your configuration definitions
2. Generating TypeScript-like type definitions for better IDE support
3. Building plugin systems with strong type guarantees
4. Generating JSON schemas for configuration validation

Basic Usage

You can use the configuration codegen feature via the command line:

nshconfig-export my_module -o exported_configs

This will:

1. Find all configuration classes in my_module
2. Generate a clean export hierarchy in the exported_configs directory
3. Optionally generate TypedDict definitions and JSON schemas

Features

Type-Safe Exports

The codegen tool automatically creates a clean export hierarchy that maintains your module structure:

# Original: my_module/configs/model.py
class ModelConfig(Config):
    hidden_size: int
    num_layers: int

# Generated: exported_configs/configs/model.py
from my_module.configs.model import ModelConfig

# Your code can now import from the generated interface:
from exported_configs.configs.model import ModelConfig

TypedDict Generation

With the --generate-typed-dicts flag, nshconfig generates TypedDict versions of your configurations along with type-safe creator functions:

nshconfig-export my_module -o exported_configs --generate-typed-dicts

This creates TypedDict definitions that mirror your Config classes:

# Original Config
class ModelConfig(Config):
    hidden_size: int
    num_layers: int

# Generated TypedDict
class ModelConfigTypedDict(TypedDict):
    hidden_size: int
    num_layers: int

# Generated creator function
def CreateModelConfig(
    dict: ModelConfigTypedDict, /  # Positional only dict argument
) -> ModelConfig: ...

def CreateModelConfig(
    **dict: Unpack[ModelConfigTypedDict]  # Keyword arguments
) -> ModelConfig: ...

You can use these definitions in several ways:

from exported_configs.configs.model import ModelConfig, ModelConfigTypedDict, CreateModelConfig

# Use the TypedDict for type-safe dictionaries
config_dict: ModelConfigTypedDict = {
    "hidden_size": 256,
    "num_layers": 4
}

# Create configs from dictionaries
config1 = CreateModelConfig(config_dict)
config2 = CreateModelConfig(hidden_size=256, num_layers=4)

# Both are equivalent to:
config3 = ModelConfig(hidden_size=256, num_layers=4)

JSON Schema Generation

With the --generate-json-schema flag, nshconfig generates JSON schemas for your configurations:

nshconfig-export my_module -o exported_configs --generate-json-schema

This creates .schema.json files that can be used for:

• Configuration validation in any language
• API documentation
• IDE support for JSON/YAML files
• Integration with other tools

Command Line Options

nshconfig-export [-h] -o OUTPUT [--remove-existing | --no-remove-existing]
                [--recursive | --no-recursive] [--verbose | --no-verbose]
                [--ignore-module IGNORE_MODULE] [--ignore-abc | --no-ignore-abc]
                [--export-generics | --no-export-generics]
                [--generate-typed-dicts | --no-generate-typed-dicts]
                [--generate-json-schema | --no-generate-json-schema]
                module

Key options:

• --recursive: Recursively process all submodules (default: True)
• --ignore-abc: Skip abstract base classes
• --ignore-module: Ignore specific modules
• --export-generics: Include generic type definitions
• --generate-typed-dicts: Generate TypedDict definitions
• --generate-json-schema: Generate JSON schemas

Use Cases

1. Client Libraries: Generate clean, minimal interfaces for your configurations that clients can depend on without pulling in your entire codebase.

2. Plugin Systems: Use TypedDict definitions to allow plugins to work with your configurations without depending on your core library:

# Plugin code can use TypedDicts instead of importing your Config classes
from my_library_export import ModelConfigTypedDict

def process_config(config_dict: ModelConfigTypedDict) -> None:
    print(f"Processing model with {config_dict['num_layers']} layers")

3. IDE Support: Get better IDE completion and type checking when working with configuration dictionaries:

from my_library_export import ModelConfigTypedDict

def create_model_config() -> ModelConfigTypedDict:
    return {
        "hidden_size": 256,  # IDE knows this needs to be an int
        "num_layers": 4      # IDE provides completion for field names
    }

4. Schema Validation: Use generated JSON schemas to validate configurations in any environment:

import json
from jsonschema import validate

# Load the generated schema
with open("exported_configs/model/ModelConfig.schema.json") as f:
    schema = json.load(f)

# Validate a configuration
config = {"hidden_size": 256, "num_layers": 4}
validate(instance=config, schema=schema)

MISSING Constant

The MISSING constant is similar to None, but with a key difference. While None has the type NoneType and can only be assigned to fields of type T | None, the MISSING constant has the type Any and can be assigned to fields of any type.

Motivation

The MISSING constant addresses a common issue when working with optional fields in configurations. Consider the following example:

import nshconfig as C

# Without MISSING:
class MyConfigWithoutMissing(C.Config):
    age: int
    age_str: str | None = None

    def __post_init__(self):
        if self.age_str is None:
            self.age_str = str(self.age)

config = MyConfigWithoutMissing(age=10)
age_str_lower = config.age_str.lower()
# ^ The above line is valid code, but the type-checker will complain because `age_str` could be `None`.

In the above code, the type-checker will raise a complaint because age_str could be None. This is where the MISSING constant comes in handy:

# With MISSING:
class MyConfigWithMissing(C.Config):
    age: int
    age_str: C.AllowMissing[str] = C.MISSING

    def __post_init__(self):
        if self.age_str is C.MISSING:
            self.age_str = str(self.age)

config = MyConfigWithMissing(age=10)
age_str_lower = config.age_str.lower()
# ^ No more type-checker complaints!

By using the MISSING constant, you can indicate that a field is not set during construction, and the type-checker will not raise any complaints.

Seamless Integration with PyTorch Lightning

nshconfig seamlessly integrates with PyTorch Lightning by implementing the Mapping interface. This allows you to use your configs directly as the hparams argument in your Lightning modules without any additional effort.

Credit

nshconfig is built on top of the incredible pydantic library. Massive credit goes to the pydantic team for creating such a powerful and flexible tool for data validation and settings management.

Contributing

Contributions are welcome! If you find any issues or have suggestions for improvement, please open an issue or submit a pull request on the GitHub repository.

License

nshconfig is open-source software licensed under the MIT License.