Settings management using Pydantic, with ability to retrieve cloud secrets.
Project description
Settus
Settings management using Pydantic Settings and extended to secrets from Azure Keyvault, Databricks secrets [IN PROGRESS], AWS Secrets Manager and GCP Secrets Manager [IN PROGRESS]
Okube Company
Okube is committed to develop open source data engineering and ML engineering tools. Contributions are more than welcome.
Installation
Install using pip install -U settus[{cloud_provider}]
where {cloud_provider}
is the cloud provider(s) you want to fetch secrets from. Possible options are:
- azure
- aws
- gcp
- databricks
Getting started
Settus uses Pydantic Settings management as its foundation. In addition, it defines the settings sources mentioned below to fetch secrets from cloud providers. The default priority order is as follows:
- Init Settings
- Environment variables
- Azure KeyVault
- AWS Secrets Manager
- GCP Secrets Manager
- Databricks secrets
In other words, if a setting is not available from the initialization or from an environment variable, it wil sequentially lookup the field name (or aliases) in the other available sources.
Azure Key Vault
To use Azure Keyvault, log in using Azure CLI or set these environment variables:
AZURE_TENANT_ID
AZURE_CLIENT_ID
AZURE_CLIENT_SECRET
More logging in options are described here.
In addition, provide keyvault_url
either to SettingsConfigDict
or to a given field.
AWS Secrets Manager
To use AWS Secrets Manager, log in using AWS CLI or set these environment variables:
AWS_REGION
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
More logging in options are described here.
In addition, provide aws_secret_name
either to SettingsConfigDict
or to a given field.
GCP Secrets Manager
TODO
Databricks Secrets
TODO
A Simple Example
import os
from settus import BaseSettings
from settus import Field
KEYVAULT_URL = "https://my-keyvault.vault.azure.net/"
AWS_SECRET_NAME = "vault"
os.environ["MY_ENV"] = "my_value"
class Settings(BaseSettings):
# Value from environment variable "MY_ENV"
my_env: str = Field(default="undefined")
# Value from the Azure keyvault named `my-keyvault` with secret key `my-secret`
my_azure_secret: str = Field(default="undefined", alias="my-secret", keyvault_url=KEYVAULT_URL)
# Value from the secret named `vault` in AWS secrets manager and having the secret key `my-secret`
my_aws_secret: str = Field(default="undefined", alias="my-secret", aws_secret_name=AWS_SECRET_NAME)
settings = ()
print(settings.my_env)
#> my_value
print(settings.my_azure_secret)
#> secret_sauce
print(settings.my_aws_secret)
#> secret_sauce
Configuration Dict Example
When multiple settings share the same keyvault or aws secret, a global setting may be defined. In this case, Azure Keyvault will be called (assuming proper credentials are available) and if no value has been found, it will fall back on AWS Secrets. Changing the order of priorities is possible as described here.
import os
from settus import BaseSettings
from settus import Field
from settus import SettingsConfigDict
KEYVAULT_URL = "https://my-keyvault.vault.azure.net/"
AWS_SECRET_NAME = "vault"
os.environ["MY_ENV"] = "my_value"
class Settings(BaseSettings):
model_config = SettingsConfigDict(keyvault_url=KEYVAULT_URL, aws_secret_name=AWS_SECRET_NAME)
my_env: str = Field(default="undefined")
my_azure_secret: str = Field(default="undefined", alias="my-secret")
my_aws_secret: str = Field(default="undefined", alias="my-secret")
settings = ()
print(settings.my_env)
#> my_value
print(settings.my_azure_secret)
#> secret_sauce
print(settings.my_aws_secret)
#> secret_sauce
Contributing
TODO
Reporting a Security Vulnerability
TODO
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.