brickvar 0.0.5

Resolve Databricks config variables from literals, environment variables, and Azure Key Vault secrets, and substitute them into JSON config files.

brickvar

Resolve configuration variables for Databricks jobs from literals, environment variables, and Azure Key Vault secrets, and substitute them into JSON config files.

brickvar reads a JSON "variables" file in which each entry is one of:

• a literal string (which may reference other variables with ${VAR}),
• null, which substitutes a JSON null into the config file,
• an environment variable reference — {"env": "NAME"}, or
• a Databricks / Azure Key Vault secret — {"scope": ..., "key": ..., "base"?: ...}, read through the Databricks dbutils.secrets API.

Resolution is two-pass, so a secret's scope/key can themselves reference already-resolved literal or environment values. It can also substitute ${VAR} placeholders into any JSON file, leaving unknown placeholders intact.

A null variable substitutes a real JSON null. Because substitution is textual, its placeholder must be a complete string value — "${VAR}" becomes null (quotes and all). A null variable embedded in a larger string ("prefix-${VAR}") cannot become null and is left intact with a warning.

Install

pip install brickvar

Usage

from brickvar import configure_json

# Read a JSON file and substitute its ${VAR} placeholders from a variables file,
# in one call. dbutils is provided by the Databricks runtime and is required only
# when the variables file contains Key Vault secret entries.
spec = configure_json("spec.json", dbutils=dbutils, var_filepath="variables.json")

To merge several config files (and/or several variables files) in one call, use configure_jsons:

from brickvar import configure_jsons

# Variables files are merged *before* resolution, so a variable in one file may
# reference one defined in an earlier file. The config files are shallow-merged at
# the top level: a key defined by more than one file takes the last file's value
# (a conflict is logged as a warning).
spec = configure_jsons(
    ["base.json", "prod.json"],
    dbutils=dbutils,
    var_filepaths=["base.variables.json", "prod.variables.json"],
)

For finer-grained control, use the VariableResolver class directly:

from brickvar import VariableResolver

cfg = VariableResolver(dbutils=dbutils)

# Resolve a variables file to a dict.
variables = cfg.read_variables("variables.json")

# Read a JSON file and substitute its ${VAR} placeholders from a variables file.
spec = cfg.read_json("spec.json", "variables.json")

Example variables.json:

{
  "SECRET_SCOPE": { "env": "SECRET_SCOPE" },
  "HOST": "example.documents.azure.us",
  "PROXY": null,
  "CLIENT_ID": { "scope": "${SECRET_SCOPE}", "key": "SP-CLIENT-ID" },
  "STORAGE": { "scope": "kv", "key": "ACCOUNT", "base": "abfss://data@{}/curated" }
}

• HOST is a literal.
• SECRET_SCOPE comes from the SECRET_SCOPE environment variable.
• PROXY is null — a "${PROXY}" placeholder becomes a JSON null.
• CLIENT_ID is a secret whose scope is filled from the resolved SECRET_SCOPE.
• STORAGE is a secret wrapped by its base format string.

Development

python -m venv venv && source venv/bin/activate
pip install -e ".[dev]"
pytest

License

Apache-2.0