Skip to main content

The RisingWave adapter plugin for dbt

Project description

dbt-risingwave

A RisingWave adapter plugin for dbt.

RisingWave is a cloud-native streaming database that uses SQL as the interface language. It is designed to reduce the complexity and cost of building real-time applications. https://www.risingwave.com

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications. Use dbt for data transformations in RisingWave

Getting started

The package has not been published to PyPI, please install it via git.

  1. Install dbt-risingwave
python3 -m pip install dbt-risingwave
  1. Get RisingWave running

Please follow this guide to setup a functional RisingWave instance.

  1. Configure dbt profile file

The profile file is located in ~/.dbt/profiles.yml. Here's an example of how to use it with RisingWave.

default:
  outputs:
    dev:
      type: risingwave
      host: 127.0.0.1
      user: root
      pass: ""
      dbname: dev
      port: 4566
      schema: public
  target: dev
  1. Run dbt debug to check whether configuration is correct.

Models

The dbt models for managing data transformations in RisingWave is similar to typical dbt sql models. The main differences are the materializations. We customized the materializations to fit the data processing model of RisingWave.

Materializations INFO
materialized_view Create a materialized view. This materialization is corresponding to the incremental one in dbt. To use this materialization, add {{ config(materialized='materialized_view') }} to your model SQL files.
materializedview (Deprecated) only for backward compatibility, use materialized_view instead
ephemeral This materialization uses common table expressions in RisingWave under the hood. To use this materialization, add {{ config(materialized='ephemeral') }} to your model SQL files.
table Create a table. To use this materialization, add {{ config(materialized='table') }} to your model SQL files.
view Create a view. To use this materialization, add {{ config(materialized='view') }} to your model SQL files.
incremental Use materialized_view instead if possible, since RisingWave is designed to use materialized view to manage data transformation in an incremental way. From v1.7.3, dbt-risingwave support incremental model to give users better control of when to update their model. This model will update table in a batch way incrementally.
source Define a source {{ config(materialized='source') }}. You need to provide your create source statement as a whole in this model.
table_with_connector Define a table with a connector {{ config(materialized='table_with_connector') }}. You need to provide your create table with connector statement as a whole in this model. Because dbt table has its own semantics, RisingWave use table_with_connector to distinguish itself from it.
sink Define a sink {{ config(materialized='sink') }}. You need to provide your create sink statement as a whole in this model.

To learn how to use, you can check RisingWave offical example dbt_rw_nexmark.

DBT RUN behavior

  • dbt run: only create new models (if not exists) without dropping any models.
  • dbt run --full-refresh: drop models and create the new ones. This command can make sure your streaming pipelines are consistent with what you define in dbt models.

Graph operators

Graph operators is useful when you want to only recreate a subset of your models.

dbt run --select "my_model+"         # select my_model and all children
dbt run --select "+my_model"         # select my_model and all parents
dbt run --select "+my_model+"         # select my_model, and all of its parents and children

Tests

All items below have been tested against the the latest RisingWave daily build verison.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dbt_risingwave-1.8.1.tar.gz (18.2 kB view details)

Uploaded Source

Built Distribution

dbt_risingwave-1.8.1-py3-none-any.whl (22.2 kB view details)

Uploaded Python 3

File details

Details for the file dbt_risingwave-1.8.1.tar.gz.

File metadata

  • Download URL: dbt_risingwave-1.8.1.tar.gz
  • Upload date:
  • Size: 18.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.10

File hashes

Hashes for dbt_risingwave-1.8.1.tar.gz
Algorithm Hash digest
SHA256 930ad6f44ab6958fb7649221633880cf9f2032da0c8d88b1eab4b19d6b0d1998
MD5 e77ce2b0be320954d5403757b7728d2d
BLAKE2b-256 873352143b4f7a021d59520f1ca8e546da1d4e8576964e09272118624d2d463f

See more details on using hashes here.

File details

Details for the file dbt_risingwave-1.8.1-py3-none-any.whl.

File metadata

File hashes

Hashes for dbt_risingwave-1.8.1-py3-none-any.whl
Algorithm Hash digest
SHA256 4f8815811c18edd3fb41b2f422f54f1a2c08a9db885f795ea86961fab123b96b
MD5 aee81c78a6ecc0eb3f8618e7507b0634
BLAKE2b-256 46c0a2225dd2b974d0b0470214ec73edfbea038958b7faba6abea9ada4ded68b

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page