settus 0.0.6

Settings management using Pydantic, with ability to retrieve cloud secrets.

Settus

Settings management using Pydantic Settings and extended to secrets from Azure Keyvault, Databricks secrets [IN PROGRESS], AWS Secrets Manager and GCP Secrets Manager [IN PROGRESS]

Okube Company

Okube is committed to develop open source data engineering and ML engineering tools. Contributions are more than welcome.

Installation

Install using pip install -U settus[{cloud_provider}] where {cloud_provider} is the cloud provider(s) you want to fetch secrets from. Possible options are:

• azure
• aws
• gcp
• databricks

Getting started

Settus uses Pydantic Settings management as its foundation. In addition, it defines the settings sources mentioned below to fetch secrets from cloud providers. The default priority order is as follows:

• Init Settings
• Environment variables
• Azure KeyVault
• AWS Secrets Manager
• GCP Secrets Manager
• Databricks secrets

In other words, if a setting is not available from the initialization or from an environment variable, it wil sequentially lookup the field name (or aliases) in the other available sources.

Azure Key Vault

To use Azure Keyvault, log in using Azure CLI or set these environment variables:

• AZURE_TENANT_ID
• AZURE_CLIENT_ID
• AZURE_CLIENT_SECRET

More logging in options are described here.

In addition, provide keyvault_url either to SettingsConfigDict or to a given field.

AWS Secrets Manager

To use AWS Secrets Manager, log in using AWS CLI or set these environment variables:

• AWS_REGION
• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY

More logging in options are described here.

In addition, provide aws_secret_name either to SettingsConfigDict or to a given field.

GCP Secrets Manager

TODO

Databricks Secrets

TODO

A Simple Example

import os
from settus import BaseSettings
from settus import Field

KEYVAULT_URL = "https://my-keyvault.vault.azure.net/"
AWS_SECRET_NAME = "vault"

os.environ["MY_ENV"] = "my_value"

class Settings(BaseSettings):
    # Value from environment variable "MY_ENV"
    my_env: str = Field(default="undefined")
    
    # Value from the Azure keyvault named `my-keyvault` with secret key `my-secret` 
    my_azure_secret: str = Field(default="undefined", alias="my-secret", keyvault_url=KEYVAULT_URL)
    
    # Value from the secret named `vault` in AWS secrets manager and having the secret key `my-secret`
    my_aws_secret: str = Field(default="undefined", alias="my-secret", aws_secret_name=AWS_SECRET_NAME)

settings = ()
print(settings.my_env)
#> my_value
print(settings.my_azure_secret)
#> secret_sauce
print(settings.my_aws_secret)
#> secret_sauce

Configuration Dict Example

When multiple settings share the same keyvault or aws secret, a global setting may be defined. In this case, Azure Keyvault will be called (assuming proper credentials are available) and if no value has been found, it will fall back on AWS Secrets. Changing the order of priorities is possible as described here.

import os
from settus import BaseSettings
from settus import Field
from settus import SettingsConfigDict

KEYVAULT_URL = "https://my-keyvault.vault.azure.net/"
AWS_SECRET_NAME = "vault"

os.environ["MY_ENV"] = "my_value"

class Settings(BaseSettings):
    model_config = SettingsConfigDict(keyvault_url=KEYVAULT_URL, aws_secret_name=AWS_SECRET_NAME)
    my_env: str = Field(default="undefined")
    my_azure_secret: str = Field(default="undefined", alias="my-secret")
    my_aws_secret: str = Field(default="undefined", alias="my-secret")

settings = ()
print(settings.my_env)
#> my_value
print(settings.my_azure_secret)
#> secret_sauce
print(settings.my_aws_secret)
#> secret_sauce

Contributing

TODO

Reporting a Security Vulnerability

TODO