dbt-bq-sourcegen

Create or update dbt source YAML from BigQuery.

Overview

dbt-bq-sourcegen automatically generates dbt source YAML files from BigQuery schemas, preserving existing configurations during updates. It supports table filtering with wildcard patterns and maintains YAML formatting/comments. This tool streamlines the process of keeping dbt source definitions in sync with your BigQuery datasets.

Installation

pip install dbt-bq-sourcegen

Usage

Apply (create or update automatically)

dbt-bq-sourcegen apply \
  --project-id your-project \
  --schema your_dataset \
  --output models/staging/your_dataset/src_your_dataset.yml

With table filtering

# Only include tables matching pattern
dbt-bq-sourcegen apply \
  --project-id your-project \
  --schema your_dataset \
  --table-pattern "stg_*" \
  --output models/staging/your_dataset/src_your_dataset.yml

# Exclude specific tables
dbt-bq-sourcegen apply \
  --project-id your-project \
  --schema your_dataset \
  --exclude "temp" \
  --output models/staging/your_dataset/src_your_dataset.yml

Options

• --project-id: Google Cloud project ID (required)
• --schema, --dataset: BigQuery schema/dataset name (required)
• --output: Output YAML file path (required)
• --table-pattern: Table name pattern (e.g., 'stg_*')
• --exclude: Exclude tables containing this string

Features

• Automatically generates dbt source YAML from BigQuery schema
• Updates existing source YAML files while preserving custom configurations
• Supports table filtering with wildcard patterns
• Preserves YAML formatting and comments
• Pure Python implementation with clean separation of concerns

Development

# Install development dependencies
uv sync

# Run tests
uv run pytest

# Format code
uv run ruff format src/