Onion Config (Python Config)
onion_config is a Python package designed for easy configuration management. It supports loading and validating configuration data from environment variables and configuration files in JSON and YAML formats. It is a Pydantic based custom configuration package for Python projects.
✨ Features
- Main config based on Pydantic schema - https://pypi.org/project/pydantic
- Load environment variables - https://pypi.org/project/python-dotenv
- Load from multiple configs directories
- Load configs from YAML and JSON files
- Update the default config with additional configurations (
extra_dirdirectory) - Pre-load hook function to modify config data before loading and validation
- Validate config values with Pydantic validators
- Config as dictionary or Pydantic model (with type hints)
- Pre-defined base config schema for common config (
BaseConfig) - Base for custom config loader (
ConfigLoader) - Support Pydantic-v1 and Pydantic-v2
🛠 Installation
1. 🚧 Prerequisites
- Install Python (>= v3.9) and pip (>= 23):
- [RECOMMENDED] Miniconda (v3)
- [arm64/aarch64] Miniforge (v3)
- [Python virutal environment] venv
[OPTIONAL] For DEVELOPMENT environment:
- Install git
- Setup an SSH key (video tutorial)
2. 📥 Download or clone the repository
2.1. Prepare projects directory (if not exists):
# Create projects directory:
mkdir -pv ~/workspaces/projects
# Enter into projects directory:
cd ~/workspaces/projects
2.2. Follow one of the below options [A], [B] or [C]:
OPTION A. Clone the repository:
git clone https://github.com/bybatkhuu/module.python-config.git && \
cd module.python-config
OPTION B. Clone the repository (for DEVELOPMENT: git + ssh key):
git clone git@github.com:bybatkhuu/module.python-config.git && \
cd module.python-config
OPTION C. Download source code:
- Download archived zip file from releases.
- Extract it into the projects directory.
3. 📦 Install the package
OPTION A. [RECOMMENDED] Install from PyPi:
pip uninstall -y pydantic-settings
pip uninstall -y pydantic-core
# Then install with Pydantic-v1:
pip install -U onion-config[pydantic-v1]
# Install with pydantic-settings for Pydantic-v2:
pip install -U onion-config[pydantic-settings]
OPTION B. Install latest version directly from GitHub repository:
# Pydantic-v1:
pip install git+https://github.com/bybatkhuu/module.python-config.git[pydantic-v1]
# Pydantic-v2:
pip install git+https://github.com/bybatkhuu/module.python-config.git[pydantic-settings]
OPTION C. Install from the downloaded source code:
# Install directly from the source code:
# Pydantic-v1:
pip install .[pydantic-v1]
# Pydantic-v2:
pip install .[pydantic-settings]
# Or install with editable mode (for DEVELOPMENT):
# Pydantic-v1:
pip install -e .[pydantic-v1]
# Pydantic-v2:
pip install -e .[pydantic-settings]
OPTION D. Install from pre-built release files:
- Download
.whlor.tar.gzfile from releases - Install with pip:
# Pydantic-v1:
# Install from .whl file:
pip install ./onion_config-[VERSION]-py3-none-any.whl[pydantic-v1]
# Or install from .tar.gz file:
pip install ./onion_config-[VERSION].tar.gz[pydantic-v1]
# Pydantic-v2:
# Install from .whl file:
pip install ./onion_config-[VERSION]-py3-none-any.whl[pydantic-settings]
# Or install from .tar.gz file:
pip install ./onion_config-[VERSION].tar.gz[pydantic-settings]
OPTION E. Copy the module into the project directory (for testing):
# Install python dependencies:
pip install -r ./requirements/requirements.core.txt
# Pydantic-v1:
pip install -r ./requirements/requirements.pydantic-v1.txt
# Pydantic-v2:
pip install -r ./requirements/requirements.pydantic-settings.txt
# Copy the module source code into the project:
cp -r ./src/onion_config [PROJECT_DIR]
# For example:
cp -r ./src/onion_config /some/path/project/
🚸 Usage/Examples
Simple
ENV=production
examples/simple/configs/1.base.yml:
env: test
app:
name: "My App"
version: "0.0.1"
nested:
key: "value"
examples/simple/configs/2.extra.yml:
app:
name: "New App"
nested:
some: "value"
description: "Description of my app."
another_val:
extra: 1
import pprint
from loguru import logger
try:
import pydantic_settings
_has_pydantic_settings = True
except ImportError:
_has_pydantic_settings = False
from onion_config import ConfigLoader, BaseConfig
class ConfigSchema(BaseConfig):
env: str = "local"
try:
config: ConfigSchema = ConfigLoader(config_schema=ConfigSchema).load()
except Exception:
logger.exception("Failed to load config:")
exit(2)
if __name__ == "__main__":
logger.info(f"All: {config}")
logger.info(f"App name: {config.app['name']}")
if _has_pydantic_settings:
# Pydantic-v2:
logger.info(f"Config:\n{pprint.pformat(config.model_dump())}\n")
else:
# Pydantic-v1:
logger.info(f"Config:\n{pprint.pformat(config.dict())}\n")
Run the examples/simple:
cd ./examples/simple
python ./main.py
Output:
2023-09-01 00:00:00.000 | INFO | __main__:<module>:29 - All: env='production' another_val={'extra': 1} app={'name': 'New App', 'version': '0.0.1', 'nested': {'key': 'value', 'some': 'value'}, 'description': 'Description of my app.'}
2023-09-01 00:00:00.000 | INFO | __main__:<module>:30 - App name: New App
2023-09-01 00:00:00.000 | INFO | __main__:<module>:35 - Config:
{'another_val': {'extra': 1},
'app': {'description': 'Description of my app.',
'name': 'New App',
'nested': {'key': 'value', 'some': 'value'},
'version': '0.0.1'},
'env': 'production'}
Advanced
ENV=development
DEBUG=true
APP_NAME="Old App"
ONION_CONFIG_EXTRA_DIR="extra_configs"
ENV=production
APP_NAME="New App"
APP_SECRET="my_secret"
examples/advanced/configs/config.yml:
env: local
app:
name: "My App"
port: 9000
bind_host: "0.0.0.0"
version: "0.0.1"
ignore_val: "Ignore me"
logger:
output: "file"
examples/advanced/configs/logger.json:
{
"logger": {
"level": "info",
"output": "stdout"
}
}
examples/advanced/configs_2/config.yml:
extra:
config:
key1: 1
examples/advanced/configs_2/config_2.yml:
extra:
config:
key2: 2
examples/advanced/extra_configs/extra.json:
{
"extra": {
"type": "json"
}
}
from enum import Enum
from typing import Union
import pydantic
from pydantic import Field, SecretStr
_has_pydantic_settings = False
if "2.0.0" <= pydantic.__version__:
try:
from pydantic_settings import SettingsConfigDict
_has_pydantic_settings = True
except ImportError:
pass
from onion_config import BaseConfig
# Environments as Enum:
class EnvEnum(str, Enum):
LOCAL = "local"
DEVELOPMENT = "development"
TEST = "test"
DEMO = "demo"
STAGING = "staging"
PRODUCTION = "production"
# App config schema:
class AppConfig(BaseConfig):
name: str = Field("App", min_length=2, max_length=32)
bind_host: str = Field("localhost", min_length=2, max_length=128)
port: int = Field(8000, ge=80, lt=65536)
secret: SecretStr = Field(..., min_length=8, max_length=64)
version: str = Field(..., min_length=5, max_length=16)
description: Union[str, None] = Field(None, min_length=4, max_length=64)
if _has_pydantic_settings:
# Pydantic-v2:
model_config = SettingsConfigDict(extra="ignore", env_prefix="APP_")
else:
# Pydantic-v1:
class Config:
extra = "ignore"
env_prefix = "APP_"
# Main config schema:
class ConfigSchema(BaseConfig):
env: EnvEnum = Field(EnvEnum.LOCAL)
debug: bool = Field(False)
app: AppConfig = Field(...)
from loguru import logger
from onion_config import ConfigLoader
from schema import ConfigSchema
# Pre-load function to modify config data before loading and validation:
def _pre_load_hook(config_data: dict) -> dict:
config_data["app"]["port"] = "80"
config_data["extra_val"] = "Something extra!"
return config_data
config = None
try:
_config_loader = ConfigLoader(
config_schema=ConfigSchema,
configs_dirs=["configs", "configs_2", "/not_exists/path/configs_3"],
env_file_paths=[".env", ".env.base", ".env.prod"],
pre_load_hook=_pre_load_hook,
config_data={"base": "start_value"},
warn_mode="ALWAYS",
)
# Main config object:
config: ConfigSchema = _config_loader.load()
except Exception:
logger.exception("Failed to load config:")
exit(2)
import pprint
from loguru import logger
try:
import pydantic_settings
_has_pydantic_settings = True
except ImportError:
_has_pydantic_settings = False
from config import config
if __name__ == "__main__":
logger.info(f"All: {config}")
logger.info(f"ENV: {config.env}")
logger.info(f"DEBUG: {config.debug}")
logger.info(f"Extra: {config.extra_val}")
logger.info(f"Logger: {config.logger}")
logger.info(f"App: {config.app}")
logger.info(f"Secret: '{config.app.secret.get_secret_value()}'\n")
if _has_pydantic_settings:
# Pydantic-v2:
logger.info(f"Config:\n{pprint.pformat(config.model_dump())}\n")
else:
# Pydantic-v1:
logger.info(f"Config:\n{pprint.pformat(config.dict())}\n")
try:
# This will raise ValidationError
config.app.port = 8443
except Exception as e:
logger.error(f"{e}\n")
Run the examples/advanced:
cd ./examples/advanced
python ./main.py
Output:
2023-09-01 00:00:00.000 | INFO | onion_config._base:load:143 - Loading all configs...
2023-09-01 00:00:00.000 | WARNING | onion_config._base:_load_dotenv_file:201 - '/home/user/workspaces/projects/onion_config/examples/advanced/.env' file is not exist!
2023-09-01 00:00:00.000 | WARNING | onion_config._base:_load_configs_dir:257 - '/not_exists/path/configs_3' directory is not exist!
2023-09-01 00:00:00.000 | SUCCESS | onion_config._base:load:171 - Successfully loaded all configs!
2023-09-01 00:00:00.000 | INFO | __main__:<module>:19 - All: env=<EnvEnum.PRODUCTION: 'production'> debug=True app=AppConfig(name='New App', bind_host='0.0.0.0', port=80, secret=SecretStr('**********'), version='0.0.1', description=None) extra={'config': {'key1': 1, 'key2': 2}, 'type': 'json'} extra_val='Something extra!' logger={'output': 'stdout', 'level': 'info'} base='start_value'
2023-09-01 00:00:00.000 | INFO | __main__:<module>:20 - ENV: production
2023-09-01 00:00:00.000 | INFO | __main__:<module>:21 - DEBUG: True
2023-09-01 00:00:00.000 | INFO | __main__:<module>:22 - Extra: Something extra!
2023-09-01 00:00:00.000 | INFO | __main__:<module>:23 - Logger: {'output': 'stdout', 'level': 'info'}
2023-09-01 00:00:00.000 | INFO | __main__:<module>:24 - App: name='New App' bind_host='0.0.0.0' port=80 secret=SecretStr('**********') version='0.0.1' description=None
2023-09-01 00:00:00.000 | INFO | __main__:<module>:25 - Secret: 'my_secret'
2023-09-01 00:00:00.000 | INFO | __main__:<module>:30 - Config:
{'app': {'bind_host': '0.0.0.0',
'description': None,
'name': 'New App',
'port': 80,
'secret': SecretStr('**********'),
'version': '0.0.1'},
'base': 'start_value',
'debug': True,
'env': <EnvEnum.PRODUCTION: 'production'>,
'extra': {'config': {'key1': 1, 'key2': 2}, 'type': 'json'},
'extra_val': 'Something extra!',
'logger': {'level': 'info', 'output': 'stdout'}}
2023-09-01 00:00:00.000 | ERROR | __main__:<module>:36 - "AppConfig" is immutable and does not support item assignment
👍
🌎 Environment Variables
# ENV=development
# DEBUG=true
ONION_CONFIG_EXTRA_DIR="./extra_configs"
🧪 Running Tests
To run tests, run the following command:
# Install core dependencies:
pip install -r ./requirements/requirements.core.txt
# Pydantic-v1:
pip install -r ./requirements/requirements.pydantic-v1.txt
# Pydantic-v2:
pip install -r ./requirements/requirements.pydantic-settings.txt
# Install python test dependencies:
pip install -r ./requirements.test.txt
# Run tests:
python -m pytest -sv -o log_cli=true
# Or use the test script:
./scripts/test.sh -l -v -c
🏗️ Build Package
To build the python package, run the following command:
# Install python build dependencies:
pip install -r ./requirements/requirements.build.txt
# Build python package:
python -m build
# Or use the build script:
./scripts/build.sh
📝 Generate Docs
To build the documentation, run the following command:
# Install python documentation dependencies:
pip install -r ./requirements/requirements.docs.txt
# Serve documentation locally (for development):
mkdocs serve
# Or use the docs script:
./scripts/docs.sh
# Or build documentation:
mkdocs build
# Or use the docs script:
./scripts/docs.sh -b
📚 Documentation
Getting Started
API Documentation
Development
Release Notes
About
📑 References
- https://docs.pydantic.dev
- https://github.com/pydantic/pydantic
- https://docs.pydantic.dev/latest/usage/pydantic_settings
- https://github.com/pydantic/pydantic-settings
- https://saurabh-kumar.com/python-dotenv
- https://github.com/theskumar/python-dotenv
- https://packaging.python.org/tutorials/packaging-projects
Metadata
Release files for onion-config 5.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| onion_config-5.1.1.tar.gz | 19.7 kB | Without provenance | Details for onion_config-5.1.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| onion_config-5.1.1-py3-none-any.whl | Python 3 | none | any | Details for onion_config-5.1.1-py3-none-any.whl |
Total release size: 34.0 kB
Release files / onion_config-5.1.1.tar.gz
| Download URL | onion_config-5.1.1.tar.gz |
|---|---|
| Size | 19.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
cf9296643a1b612ff596dd13fb38c36ffe20b2a3a7c0c5b40fda066919f26b6a
|
|
BLAKE2b-256 checksum How to use checksums |
a4358941d940646faa1162887826d7e063492ba11f3643f253f6179b9057575e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.9.21
|
Release files / onion_config-5.1.1-py3-none-any.whl
| Download URL | onion_config-5.1.1-py3-none-any.whl |
|---|---|
| Size | 14.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
af8f7cf42fbc52440c389f5cedb40190f67201de436b71eed4645c9313443533
|
|
BLAKE2b-256 checksum How to use checksums |
b2098e8220539489c0361289d427bee890051d70887afaa004742c004de8f47b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.9.21
|
Security for onion-config 5.1.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.