A dbt-core plugin to import public nodes in multi-project deployments.
Project description
dbt-loom
dbt-loom is a dbt Core plugin that weaves together multi-project deployments. dbt-loom works by fetching public model definitions from your dbt artifacts, and injecting those models into your dbt project.
flowchart LR
classDef black fill:#f2f2ebff, stroke:#000, color:#000
classDef background fill:#f2f2ebff, stroke:#000, color:#000
classDef hidden fill:#BADC3F, stroke:#BADC3F, color:#BADC3F
style TOP fill:#BADC3F, stroke:#000
subgraph TOP[Your Infrastructure]
direction TB
dbt_runtime[dbt Core]:::background
proprietary_plugin[Open Source Metadata Plugin]:::background
files[Local Files]:::background
object_storage[Object Storage]:::background
discovery_api[dbt Cloud APIs]:::background
discovery_api --> proprietary_plugin
files --> proprietary_plugin
object_storage --> proprietary_plugin
proprietary_plugin --> dbt_runtime
end
Project:::black --> TOP --> Warehouse:::black
dbt-loom currently supports obtaining model definitions from:
- Local manifest files
- dbt Cloud
- S3-compatible object storage services [TODO]
:warning: dbt Core's plugin functionality is still in beta. Please note that this may break in the future as dbt Labs solidifies the dbt plugin API in future versions.
Getting Started
To being, install the dbt-loom
python package.
pip install dbt-loom
Next, create a dbt-loom
configuration file. This configuration file provides the paths for your
upstream project's manifest files.
manifests:
- name: project_name
type: file
config:
path: path/to/manifest.json
By default, dbt-loom
will look for dbt-loom.config.yml
in your working directory. You can also set the
DBT_LOOM_CONFIG_PATH
environment variable.
Using dbt Cloud as an artifact source
You can use dbt-loom to fetch model definitions from dbt Cloud by setting up a dbt-cloud
manifest in your dbt-loom
config, and setting the DBT_CLOUD_API_TOKEN
environment variable in your execution environment.
manifests:
- name: project_name
type: dbt_cloud
config:
account_id: <YOUR DBT CLOUD ACCOUNT ID>
# Job ID pertains to the job that you'd like to fetch artifacts from
job_id: <REFERENCE JOB ID>
api_endpoint: <DBT CLOUD ENDPOINT>
# dbt Cloud has multiple regions with different URLs. Update this to
# your appropriate dbt cloud endpoint.
step_id: <JOB STEP>
# If your job generates multiple artifacts, you can set the step from
# which to fetch artifacts. Defaults to the last step.
How does it work?
As of dbt-core 1.6.0-b8, there now exists a dbtPlugin
class which defines functions that can
be called by dbt-core's PluginManger
. During different parts of the dbt-core lifecycle (such as graph linking and
manifest writing), the PluginManger
will be called and all plugins registered with the appropriate hook will be executed.
dbt-loom implements a get_nodes
hook, and uses a configuration file to parse manifests, identify public models, and
inject those public models when called by dbt-core
.
Known Caveats
Cross-project dependencies are a relatively new development, and dbt-core plugins are still in beta. As such there are a number of caveats to be aware of when using this tool.
- dbt plugins are only supported in dbt-core version 1.6.0-b8 and newer. This means you must be using a dbt adapter compatible with this version.
PluginNodeArgs
are not fully-realized dbtManifestNode
s, so documentation generated bydbt docs generate
may be sparse when viewing injected models.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.