Find and read your project configuration files easily
Project description
Welcome to Stela
Stela were the "configuration files" of ancient times. This library aims to simplify your project configurations, from json, yaml, ini or toml files in a single Python dictionary, which values can be easily be overridden by environment variables.
Index
- Install
- Basic Use
- Environment Variables from Shell
- Using layered environments
- Customize Stela
- Advanced Use
- Using with Pydantic
- How Stela find the configuration files?
- How Stela find the Environment Variables?
- How Stela handle more complex cases?
- Full Lifecycle example
- When Stela read the data?
- Refreshing Stela settings
- How Stela read the dictionary values?
- Logging Data
- All Stela Configuration Options
- Migrating from version 3.x
- Migrating from version 2.x
- Migrating from version 1.x
Install
$ pip install stela
Basic Use
For your project settings, you need to create a pyproject.toml
file, and
under the table environment
add the key/values you need:
# pyproject.toml [environment] my_file_path = "/foo/bar" my_api_url = "https://foo.bar" db.url = "user@password:db"
In Python you will access these settings importing stela like this:
from stela import settings API_URL = settings["my_api_url"] # API_URL = "https://foo.bar"
Environment Variables from Shell
Stela will check first, for the requested key, his SCREAMING_SNAKE_CASE
format in environment memory or .env
file. If no data is found, Stela
will return the value from pyproject.toml
tables.
Some examples:
Variable in toml | Environment Variable Name |
---|---|
debug | DEBUG |
3rd_party_api_url | 3_RD_PARTY_API_URL |
db.password | DB_PASSWORD |
For example, for db.url
Stela will look for
DB_URL
in the following order:
- Import
.env
data, if exists, in python environment, using python-dotenv library. - Check for
DB_URL
inos.environ
- Fallback to
db.url
inpyproject.toml
If data does not exist, Stela will raise a KeyError
, like a dictionary.
# pyproject.toml db.url = "fake@credentials"
# .env file DB_URL=real@credentials # will override info from pyproject.toml
from stela import settings DATABASE_URL = settings["db.url"] # DATABASE_URL = "real@credentials"
Using layered environments
For now, Stela are looking only for the environment
table in your
pyproject.toml
. But you may need to define
your settings per environment layer (i.e. development, tests, staging,
production, etc.)
First add use_environment_layers
in Stela options:
# pyproject.toml [tool.stela] use_environment_layers = true
Then define environment
sub-tables in
pyproject.toml
:
# pyproject.toml [tool.stela] use_environment_layers = true [environment] # now is a shared data between environments my_api_timeout = 30 [environment.local] my_api_url = "http://localhost:8000" [environment.production] my_api_url = "https://foo.bar"
When you add use_environment_layers = true
in config, Stela will now
always try to find the current environment looking for the ENVIRONMENT
variable. If this variable is not defined, Stela will use the default
environment, if available. If not, will raise a
StelaEnvironmentNotFoundError
.
To add a default environment, use the default_environment
option:
# pyproject.toml [tool.stela] use_environment_layers = true default_environment = "local" [environment] # now is a shared data between environments my_api_timeout = 30 [environment.local] my_api_url = "http://localhost:8000" [environment.production] my_api_url = "https://foo.bar"
To use different environment variable, use the environment_variable_name
option:
# pyproject.toml [tool.stela] use_environment_layers = true default_environment = "local" environment_variable_name = "ENV" [environment] # now is a shared data between environments my_api_timeout = 30 [environment.local] my_api_url = "http://localhost:8000" [environment.production] my_api_url = "https://foo.bar"
Customize Stela
Use the following variables to customize Stela behavior:
# You can also set these as environment variables too using STELA_ prefix. # For example, environment_variable_name can be override from STELA_ENVIRONMENT_VARIABLE_NAME [tool.stela] environment_variable_name = "ENVIRONMENT" # The Environment variable default_environment = "" # The default value for Environment variable env_table = "environment" # The main environment table in pyproject.toml use_environment_layers = false # Use environment layers env_file = ".env" # dotenv file name config_file_path = "." # relative path for configuration files do_not_read_dotenv = false # If True, will load dotenv file in os.environ dotenv_overwrites_memory = true # If True, will not overwrite keys from dotenv file if they exists on environ show_logs = false # Show Stela Logs log_filtered_value = true # Show Value from Stela settings in logs filtered. Use with caution.
Example:
# shell $ export DJANGO_ENV=production
# pyproject.toml [tool.stela] use_environment_layers = true environment_variable_name = "DJANGO_ENV" env_table = "my_project.config" [my_project.config] project.debug = true [my_project.config.production] project.debug = false
# settings.py from stela import settings DEBUG = settings["project.debug"] # False from pyproject.toml or from PROJECT_DEBUG in environment
Advanced Use
For very large projects, you can use separate environment layer per config files
(like development.ini
, staging.ini
, etc.)
Also, you can use .yaml
, .ini
, .json
and .toml
files.
In all files, Stela will always respect nested data. Suppose a file
called development.ini
which contains:
[foo] bar = value
As we know, Stela will convert data into a python dictionary:
{ "foo": { "bar": "value" } }
You can use the settings like this:
from stela import settings my_conf = settings["foo.bar"] # my_conf = "value"
This is possible because Stela uses under the hood the Scalpl library.
Using with Pydantic
Pydantic is a very popular library for data validation and settings management. Stela can enhance pydantic experience, by being a additional settings for it. You can import stela settings as a dictionary and import in pydantic settings, as per Customize settings sources documentation:
# pyproject.toml [tool.stela] use_environment_layers = true default_environment = "local" environment_variable_name = "ENV" [environment] # Shared data between environments my_api_timeout = 30 db.host = "localhost" [environment.local] my_api_url = "http://localhost:8000" [environment.production] my_api_url = "https://foo.bar"
from pydantic import BaseSettings, Extra from stela.pydantic import stela_settings class DbSettings(BaseSettings): host: str port: int = 5432 class Settings(BaseSettings): my_api_url: str my_api_timeout: int db: DbSettings class Config: # If you need to debug stela dictionary, # you can use this option to log stela settings dictionary log_stela_settings = True # If Stela dictionary has extra fields, # you can use this option to ignore them extra = Extra.ignore @classmethod def customise_sources( cls, init_settings, env_settings, file_secret_settings, ): return ( init_settings, stela_settings, # Add stela settings before env_settings env_settings, file_secret_settings, ) print(Settings()) # for ENV=local #> my_api_url='http://localhost:8000', my_api_timeout=30, db=DbSettings(host='localhost', port=5432)
Both config options (log_stela_settings
and extra
) are optional.
Please check file test_pydantic.py
in tests folder for additional info.
How Stela find the configuration files?
By default, Stela will use the value from ENVIRONMENT
environment
variable to find correspondent INI file. For example, if you set
ENVIRONMENT=development
Stela will look for development.ini
file in
project root.
You can change this behaviour inside pyproject.toml
file:
[tools.stela] environment_variable_name = "ENVIRONMENT" # Default Enviroment variable name config_file_extension = "INI" # YAML, TOML, JSON config_file_prefix = "" # You can add a prefix before name - ex.: env_development.ini config_file_suffix = "" # You can add a suffix after name - ex.: development_v1.ini default_environment = "development" # use this if you do not want to define the ENVIRONMENT key config_file_path = "." # relative path from project root for configuration files
How Stela find the Environment Variables?
We know Stela will try to find an environment variable using his
SCREAMING_SNAKE_CASE version. In this case, when the key is foo.bar
,
Stela will search for an env called FOO_BAR
in os.environ
files, before returning his dict value. Also, you can add a
prefix/suffix in this name (ex.: MYPROJECT_FOO_BAR
). To do this,
define them in pyproject.toml
:
[tools.stela] environment_prefix = "MYPROJECT_" environment_suffix = ""
from stela import settings my_conf = settings["foo.bar"] # my_conf = "hello world" from MYPROJECT_FOO_BAR
Also, you can define Stela to never get values from shell and/or dotenv, only from dictionary:
[tools.stela] do_not_read_environment = true do_not_read_dotenv = true
How Stela handle more complex cases?
Stela uses this lifecycle to handle the settings load, using the pre_load
, custom_load
and post_load
decorators
to modified data received from files and pyproject.toml, before importing settings.
To use this:
- The decorated function must return a valid python dictionary.
- Just one decorated function per phase.
- Import the decorated functions before import stela settings for the first time. If you create the file
conf_stela.py
at project root, decorated functions in there will be imported automatically.
The Pre-Load Phase (optional)
If defined, will always be the first step. To setup, create a
conf_stela.py
file on project root and use the pre_load
decorator
for your code.
Pre-Load Example:
# conf_stela.py at project root import plaster from stela.decorators import pre_load from stela.stela_options import StelaOptions from typing import Dict, Any @pre_load def get_from_plaster(options: StelaOptions) -> Dict[Any, Any]: """Get data from plaster. Must return a valid Python dictionary. :param options: StelaOptions instance :return dict """ env = options.current_environment # get current environment during lifecycle config_uri = f"{env}.ini#myapp" settings = plaster.get_settings(config_uri, 'my-settings') return settings
The Load Phase
In this phase Stela will run 2 default loaders and a third optional custom loader (if defined) on this default order:
- Runs
embed
loader (retrieve data frompyproject.toml
, if exists) - Runs
file
loader (retrieve data from config files, if exists) - Runs
custom
loader if defined (fromcustom_load
decorator, if exists)
Each step updates data received from the previous step. You can change
this order, modifying the load_order
in config:
# Or STELA_LOAD_ORDER env [tool.stela] # Default value is ["embed", "file", "custom"] load_order = ["custom"]
Custom Load Example:
# conf_stela.py at project root from stela.decorators import custom_load from stela.stela_options import StelaOptions from typing import Dict, Any @custom_load def remove_bad_data(data: Dict[Any, Any], options: StelaOptions) -> Dict[Any, Any]: """Remove bad data. Must return a valid Python dictionary. :param options: StelaOptions instance :return dict """ can_remove_bad_data = options.dotenv_data.get("REMOVE_BAD_DATA") # reading dotenv during lifecycle if can_remove_bad_data: data.pop("bad_key", None) return data
The Post-Load Phase
This is, always, the last phase.
# conf_stela.py at project root import boto3 from stela.decorators import post_load from stela import StelaOptions from typing import Dict, Any @post_load def add_ssm_parameters(data: dict, options: StelaOptions) -> Dict[Any, Any]: """Load settings from SSM to current Stela data. Data returned must be a Python Dictionary. :param data (dict): Data parsed from previous phases :param options (obj): Stela Options from pyproject.toml :return Dict[Any, Any] """ ssm = boto3.client('ssm') environment = options.current_environment response = ssm.get_parameters_by_path( Path=f'/foo/bar/{environment}', WithDecryption=True ) return {parameter["Name"]: parameter["Value"] for parameter in response["Parameters"]}
Full Lifecycle example
# pyproject.toml [environment] test = true
# conf_stela.py at project root from stela.decorators import pre_load, custom_load, post_load from typing import Dict, Any from stela import StelaOptions @pre_load def pre_load(options: StelaOptions) -> Dict[Any, Any]: # data value is: {} # Stela Options are available in options object. return {"foo": "bar"} @custom_load def load(data: dict, options: StelaOptions) -> Dict[Any, Any]: # data value is: {"foo": "bar", "test": True} from from pre_load and pyproject.toml return {"has_dogs": True} @post_load def post_load(data: dict, options: StelaOptions) -> Dict[Any, Any]: # data value is: {"foo": "bar", "has_dogs": True, "test": True} return {"number_of_dogs": 1}
from stela import settings print(settings) # {"foo": "bar", "has_dogs": True, "number_of_dogs": 1, "test": True}
When Stela read the data?
Stela are imported once, at module level - project settings are load and immediately available:
from stela import settings from flask import Flask app = Flask(__name__) app.config.update( SECRET_KEY=settings["my_app_secret"] # will read from dict or MY_APP_SECRET value, after run all lifecycle ) @app.route("/") def hello(): return f"Hello, Environment is {settings.stela_options.current_environment}"
Refreshing Stela settings
If you need to reload settings, use the stela.stela_reload
function.
Check the unit tests for additional input.
How Stela read the dictionary values?
Stela will respect the file format limitations. For example, INI files always return values as string, TOML files returning datetime objects, etc...
For environment variables, Stela will return value as string, by
default. For example: NUMBER_OF_CATS=3
will return a string.
You can set Stela to evaluate these values, as per
ast.literal_eval
rules. To do this, add in pyproject.toml
:
[tool.stela] evaluate_data = true
Logging data
Stela use the loguru package for logging, using INFO
for general messages
and DEBUG
for key/values retrieved in toml, environment keys, decorators, etc...
You can use the logs to debug data during Stela lifecycle.
By default, log are disabled. You can modify this behavior globally with the following configurations:
[tool.stela] show_logs = true log_filtered_value = true
Also, you can use decorators for fine-tuning logging:
from stela import settings from stela.decorators import stela_enable_logs, stela_disable_logs @stela_enable_logs def my_bugged_code(): return settings["my_api_url"] @stela_disable_logs def my_sensible_code(): return settings["my_credentials"]
The log level can be defined using the LOGURU_LOG_LEVEL
as per loguru documentation.
All Stela Configuration Options
All configuration files can be overridden using an environment variable,
using the STELA_
prefix. For example, default_environment
turns
STELA_DEFAULT_ENVIRONMENT
.
# All keys and default values available in pyproject.toml [tool.stela] config_file_extension = "INI" # or YAML, TOML, JSON config_file_path = "." # relative path from project root, for config files (i.e. .env, development.ini, etc.) config_file_prefix = "" # You can add a prefix before name - ex.: env_development.ini config_file_suffix = "" # You can add a suffix after name - ex.: development_v1.ini default_environment = "" # The default value for Environment variable do_not_read_environment = false # Do not read environment variables from shell do_not_read_dotenv = false # Do not read dotenv file env_file = ".env" # dotenv file name env_table = "environment" # The main environment table in pyproject.toml environment_prefix = "" # ex.: settings["foo.bar"] looks for MY_PREFIX_FOO_BAR environment_suffix = "" # ex.: settings["foo.bar"] looks for FOO_BAR_MY_SUFFIX environment_variable_name = "ENVIRONMENT" # The Environment variable evaluate_data = false # Evaluate data received from config files load_order = ["embed", "file", "custom"] # Default order for Loaders in Load Phase show_logs = false # As per loguru settings. log_filtered_value = true # When logging data, filter values from dict/env. use_environment_layers = false # Use environment layers dotenv_overwrites_memory = true # If True, values from os.environ will be override with dotenv values. If False, order is os.environ -> dotenv values -> files
Migrating from version 3.x
- Support for Python 3.7 was dropped
Migrating from version 2.x
- Logs are now disabled, by default. Too re-enable Stela logs:
[tool.stela] show_logs = true
Or set STELA_SHOW_LOGS=true
environment variable.
Migrating from version 1.x
- Support for Python 3.6 was dropped
- The
stela_reload
function now is imported now fromstela.utils
- The
@load
decorator now was renamed to@custom_load
- To mimic Stela load behavior from 1.x (Pre -> Load or Custom -> Post),
please configure the old lifecycle in
pyproject.toml
:
[tool.stela] use_environment_layers = true do_not_read_dotenv = true load_order = ["file"] # or ["custom"] with @custom_load decorator
Not working?
Don't panic. Get a towel and, please, open an issue.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.