vdstudio-dbtfabric-notebook 0.1.0

Fabric notebook dbt job runner (clone + run + persist) on top of the official dbt-fabricspark adapter.

vdstudio-dbtfabric-notebook

A thin dbt job runner for Microsoft Fabric notebooks, built on top of the official dbt-fabricspark adapter.

A Fabric deploy notebook installs this package and makes a single call — run_dbt_job(config) — which clones the dbt project from Git, runs the dbt command(s), and persists logs and artifacts to the lakehouse.

Install

pip install vdstudio-dbtfabric-notebook

The Fabric notebook runtime must also have dbt-core and dbt-fabricspark available (the runner shells out to the dbt CLI). notebookutils is provided by the Fabric runtime and is imported lazily only when fetching GitHub App secrets from Key Vault.

Use

Inside a Fabric notebook:

from vdstudio_dbtfabric_notebook import (
    run_dbt_job,
    DbtJobConfig,
    RepoConfig,
    ConnectionConfig,
)

config = DbtJobConfig(
    command=["dbt deps", "dbt build --target prod"],
    repo=RepoConfig(
        url=repo_url,
        branch=repo_branch,
        github_app_id=github_app_id,            # Key Vault secret name
        github_installation_id=github_installation_id,  # Key Vault secret name
        github_pem_secret=github_pem_secret,    # Key Vault secret name
        vault_url=vault_url,
    ),
    connection=ConnectionConfig(
        lakehouse_name=lakehouse_name,
        lakehouse_id=lakehouse_id,
        workspace_id=workspace_id,
        workspace_name=workspace_name,
        schema_name=schema_name,
    ),
)

result = run_dbt_job(config)

What it does

run_dbt_job(config):

1. Environment setup — sets LAKEHOUSE, LAKEHOUSE_ID, SCHEMA, WORKSPACE_ID, WORKSPACE_NAME, DBT_JOB_NAME so the cloned project's profiles.yml resolves via env_var().
2. Clone — git clone --depth 1 -b <branch>, authenticated via a GitHub App. The app id, installation id, and PEM private key are Key Vault secret names, fetched at runtime through notebookutils. A pre-resolved token may be supplied to skip the GitHub App flow; public repos clone without auth.
3. Run — dbt deps, then the dbt command(s) in order. Logs are emitted as JSON; dbt.log, run_results.json, and manifest.json are persisted to /lakehouse/default/Files/logs/dbt/{YYYY}/{MM}/{DD}/{invocation_id}/.

Commands can be a single string or a list run sequentially; the returned DbtResult is from the last command.

Ephemeral validation

The runner consumes whatever ConnectionConfig and command it is given. A wrapper pipeline can override the lakehouse/workspace parameters and swap --target prod for an ephemeral target to validate against a throwaway lakehouse, without any code change in this package.

Security

GitHub App secrets and tokens are never baked into the notebook; the notebook passes Key Vault secret names, and the actual values are resolved at runtime inside the Fabric environment.

Development

pip install -e ".[dev]"
pytest

License

Apache-2.0