dbt-risingwave

A RisingWave adapter plugin for dbt.

RisingWave is a cloud-native streaming database that uses SQL as the interface language. It is designed to reduce the complexity and cost of building real-time applications. See https://www.risingwave.com.

dbt enables data analysts and engineers to transform data using software engineering workflows. For the broader RisingWave integration guide, see https://docs.risingwave.com/integrations/other/dbt.

Getting Started

1. Install dbt-risingwave.

python3 -m pip install dbt-risingwave

2. Get RisingWave running by following the official guide: https://www.risingwave.dev/docs/current/get-started/.

3. Configure ~/.dbt/profiles.yml.

default:
  outputs:
    dev:
      type: risingwave
      host: 127.0.0.1
      user: root
      pass: ""
      dbname: dev
      port: 4566
      schema: public
  target: dev

4. Run dbt debug to verify the connection.

Common Features

Detailed reference: docs/.

Schema Authorization

Use schema_authorization when dbt should create schemas with a specific owner:

{{ config(materialized='table', schema_authorization='my_role') }}

See docs/configuration.md for model-level and dbt_project.yml examples.

RisingWave Native Model Configs

The adapter supports RisingWave session settings such as streaming_parallelism, streaming_parallelism_for_backfill, backfill_rate_limit, and enable_index_selection in profiles and native model configs.

See docs/configuration.md for the full configuration matrix.

Serverless Backfill

Use enable_serverless_backfill=true in a model config or profile to enable serverless backfills for streaming queries.

See docs/configuration.md for examples.

Backfill Order Control

Materialized-view models can use backfill_order to render RisingWave's WITH (backfill_order = FIXED(...)) option from a list of dependency edges. The model query should continue to use ref() or source() so dbt records the dependency graph.

See docs/configuration.md for examples, requirements, and RisingWave limitations.

Background DDL

background_ddl=true lets supported materializations submit background DDL while still preserving dbt semantics by issuing RisingWave WAIT before dbt continues.

See docs/configuration.md for supported materializations, examples, and the cluster-wide WAIT caveat.

Zero-Downtime Rebuilds

materialized_view and view support swap-based zero-downtime rebuilds. Supported adapter-managed sink models can use RisingWave REPLACE SINK for cut-over. Both paths require zero_downtime={'enabled': true} plus the runtime flag --vars 'zero_downtime: true'.

See docs/zero-downtime-rebuilds.md for requirements, current sink limitations, cleanup behavior, and helper commands.

Functions

dbt-risingwave now supports a first version of dbt function resources for RisingWave scalar UDFs.

Current contract:

• supported:
  • SQL scalar functions
  • JavaScript scalar functions via functions/*.sql plus config.language: javascript
  • external Python scalar functions via functions/*.sql plus config.language: python
    • with config.link: http://host:port
    • optional config.remote_name
    • optional config.always_retry_on_network_error
• materialization: CREATE FUNCTION IF NOT EXISTS
• JavaScript async options:
  • config.async: true -> WITH (async = true)
  • config.batch: true -> WITH (batch = true)
  • config.always_retry_on_network_error: true -> WITH (always_retry_on_network_error = true)
• supported volatility config:
  • deterministic -> IMMUTABLE
  • stable -> STABLE
  • non-deterministic -> VOLATILE

Current limits:

• no replace/update path for an existing function body
• no overload-family management
• no aggregate or table functions
• no default arguments
• upstream dbt-core function contracts do not yet map cleanly to RisingWave-native .js authoring or RisingWave external Python UDF authoring, so JavaScript and Python currently use adapter config on functions/*.sql

See docs/functions.md for the full first-version contract and example layout.

Indexes

RisingWave indexes support INCLUDE and DISTRIBUTED BY clauses beyond what the Postgres adapter exposes. Configure them in the model config:

{{ config(
    materialized='materialized_view',
    indexes=[
        {'columns': ['user_id'], 'include': ['name', 'email'], 'distributed_by': ['user_id']}
    ]
) }}

This generates:

CREATE INDEX IF NOT EXISTS "__dbt_index_mv_user_id"
  ON mv (user_id)
  INCLUDE (name, email)
  DISTRIBUTED BY (user_id);

Note: RisingWave does not support unique or type (index method) options from the Postgres adapter. These options are silently ignored.

dbt-risingwave also emits adapter validation warnings for known unsupported or ignored model patterns, such as retention_seconds on CREATE MATERIALIZED VIEW or complete DDL statements inside query-based materializations. See docs/configuration.md for the warning list and the risingwave_adapter_validation mode.

Materializations

The adapter follows standard dbt model workflows, with RisingWave-specific materializations and behaviors.

Typical usage:

{{ config(materialized='materialized_view') }}

select *
from {{ ref('events') }}

See docs/configuration.md for adapter-specific configuration examples, including streaming session settings, background DDL, sinks, and subscriptions.

Documentation

• docs/README.md: documentation index
• docs/configuration.md: profile options, model configs, sink settings, and background DDL usage
• docs/functions.md: first-version RisingWave scalar function support and limitations
• docs/zero-downtime-rebuilds.md: zero-downtime rebuilds for views and materialized views, plus supported sink cut-overs

dbt Run Behavior

• dbt run: creates models that do not already exist.
• dbt run --full-refresh: drops and recreates models so the deployed objects match the current dbt definitions.

Graph Operators

Graph operators are useful when you want to rebuild only part of a project.

Data Tests

dbt-risingwave extends dbt data-test failure storage to support materialized_view in addition to the upstream table and view options.

Example:

models:
  - name: my_model
    columns:
      - name: id
        tests:
          - not_null:
              config:
                store_failures: true
                store_failures_as: materialized_view

This is useful for realtime monitoring workflows where test failures should remain continuously queryable as a RisingWave materialized view.

dbt run --select "my_model+"   # select my_model and all children
dbt run --select "+my_model"   # select my_model and all parents
dbt run --select "+my_model+"  # select my_model, and all of its parents and children

Examples

• Official dbt example: jaffle_shop
• RisingWave example: dbt_rw_nexmark