dbt-pal

dbt-pal is Python Adapter Layer

A dbt adapter for running Python models without Dataproc or BigQuery DataFrames.

• SQL models work the same as dbt-bigquery
• Python models are executed in the process running dbt, and the results are written to BigQuery
• The only supported data platform is BigQuery

Inspired by dbt-fal, but this is an unrelated project with no guaranteed compatibility.

Usage

Installation

pip install dbt-pal

Prerequisites

• Python >= 3.11
• dbt-core >= 1.11.0
• dbt-bigquery >= 1.11.0
• Authentication to BigQuery must be configured (e.g. gcloud auth application-default login)

profiles.yml Configuration

Create a target with type: pal and specify the target name of the actual BigQuery target in db_profile field.

my_project:
  target: pal
  outputs:
    pal:
      type: pal
      db_profile: bq
    bq:
      type: bigquery
      method: oauth
      project: my-project
      dataset: my_dataset
      location: asia-northeast1

Limitations

• SQL models support the table and view materializations
• Python models support only the table materialization
• Python models are executed in the process running dbt, so the data size that can be handled depends on the memory of that process

License

Apache License 2.0.
This project was created by modifying code from dbt-fal.