vdstudio-dbtfabric-notebook 0.2.0

Fabric notebook dbt job runner (clone + run + persist) on top of the official dbt-fabricspark adapter.

vdstudio-dbtfabric-notebook

A thin dbt job runner for Microsoft Fabric notebooks, built on top of the official dbt-fabricspark adapter.

A Fabric deploy notebook installs this package and makes a single call — run_dbt_job(config) — which clones the dbt project over SSH using a deploy key fetched from the attached lakehouse, runs the dbt command(s), and persists logs and artifacts to the lakehouse.

Install

pip install vdstudio-dbtfabric-notebook

The Fabric notebook runtime must also have dbt-core and dbt-fabricspark available (the runner shells out to the dbt CLI). No other runtime dependencies — the package has zero third-party Python deps.

Use

Inside a Fabric notebook:

from vdstudio_dbtfabric_notebook import (
    run_dbt_job,
    DbtJobConfig,
    RepoConfig,
    ConnectionConfig,
)

config = DbtJobConfig(
    command=["dbt deps", "dbt build --target prod"],
    repo=RepoConfig(
        url="[e-mail]:org/repo.git",
        branch="main",
        deploy_key_onelake_path="Files/_studio/<domain-slug>/deploy-keys/<repo>.id_ed25519",
    ),
    connection=ConnectionConfig(
        lakehouse_name=lakehouse_name,
        lakehouse_id=lakehouse_id,
        workspace_id=workspace_id,
        workspace_name=workspace_name,
        schema_name=schema_name,
    ),
)

result = run_dbt_job(config)

What it does

run_dbt_job(config):

1. Environment setup — sets LAKEHOUSE, LAKEHOUSE_ID, SCHEMA, WORKSPACE_ID, WORKSPACE_NAME, DBT_JOB_NAME so the cloned project's profiles.yml resolves via env_var().
2. Clone over SSH — reads the deploy key from /lakehouse/default/<deploy_key_onelake_path> (the notebook's attached default lakehouse is mounted there by Fabric), copies it to a 0600 temp file, and runs git clone --depth 1 --branch <branch> with GIT_SSH_COMMAND configured to use that key. The temp key is deleted immediately after, even on failure.
3. Run — dbt deps, then the dbt command(s) in order. Logs are emitted as JSON; dbt.log, run_results.json, and manifest.json are persisted to /lakehouse/default/Files/logs/dbt/{YYYY}/{MM}/{DD}/{invocation_id}/.

Commands can be a single string or a list run sequentially; the returned DbtResult is from the last command.

OneLake deploy-key convention

<lakehouse>/Files/_studio/<domain-slug>/deploy-keys/<repo-name>.id_ed25519

• One read-only deploy key per repo, generated and uploaded by studio at provisioning time.
• The notebook never sees the GitHub App PEM. It only reads back the per-repo private key from the lakehouse it's attached to.
• deploy_key_onelake_path is interpreted relative to /lakehouse/default/ unless an absolute path is supplied.

Ephemeral validation

The runner consumes whatever ConnectionConfig, branch, and command it is given. A wrapper pipeline can override the lakehouse/workspace parameters, the branch, and swap --target prod for an ephemeral target to validate against a throwaway lakehouse — no code change required.

Security

• No PEM and no Key Vault calls anywhere. The notebook's only secret exposure is the per-repo deploy key, scoped read-only.
• The staged temp key has 0600 permissions and is deleted in a finally block before the function returns.
• SSH host verification uses StrictHostKeyChecking=accept-new with a per-clone UserKnownHostsFile (/tmp/known_hosts_deploy). Pin the github.com fingerprint there if you want stricter guarantees.

Development

pip install -e ".[dev]"
pytest

License

Apache-2.0