Skip to main content

dbt-confluent

The dbt adapter for Confluent Cloud Flink SQL.

Build, test, and manage streaming data transformations on Confluent Cloud using dbt's familiar development workflow.

Overview

dbt-confluent lets you use dbt to define and run SQL transformations on Confluent Cloud's fully managed Apache Flink service. It supports both batch-style and streaming materializations, enabling continuous data pipelines defined as dbt models.

Features:

  • Standard dbt materializations (table, view, ephemeral) adapted for Flink SQL
  • Streaming-native materializations (streaming_table, streaming_source) for continuous data pipelines
  • A declarative materialized_table materialization built on Flink's CREATE OR ALTER MATERIALIZED TABLE
  • Integration with Confluent Cloud connectors (e.g., Datagen/Faker) via streaming_source
  • distributed_by config to control Kafka partitioning via the DISTRIBUTED BY HASH(...) INTO N BUCKETS clause
  • Schema drift detection on re-runs (columns, WITH options, distributed_by) — surfaces every violation in one error
  • Adopt existing tables and statements deployed outside dbt via the alias and statement_name configs
  • tableflow config to materialize a model's backing Kafka topic as an Iceberg/Delta table via Tableflow

See Materializations for the full list and details.

Installation

pip install dbt-confluent

or with uv:

uv add dbt-confluent

Requires Python 3.10–3.13.

Configuration

After installing, scaffold a new project with:

dbt init my_project

Select confluent as the adapter and fill in the prompts for your Confluent Cloud credentials.

You have several options for configuring your credentials:

  • Use OAuth (interactive Confluent Cloud sign-in, no API key)
    • Sets auth: oauth in your profile to use OAuth login (the default is auth: api_key).
    • Omit global_api_key/global_api_secret/flink_api_key/flink_api_secret when using auth: oauth.
  • Use a Global API key
    • Uses a single API key & secret pair that works against every Confluent Cloud route used by this adapter.
    • This is the simplest configuration for non-interactive use.
    • Sets global_api_key & global_api_secret.
  • Use a Flink-region API key
    • Flink-region API keys allow non-interactive use similar to global API keys, but are more restricted in scope.
    • Some routes used by the adapter will not work with a Flink-region API key.

dbt init will prompt you for a choice and ensure your profile is configured correctly.

Concept mapping

Confluent Cloud Flink uses different terminology than traditional databases. Here's how dbt concepts map to Flink and Confluent Cloud:

dbt concept Flink concept Confluent Cloud entity
database Catalog Environment
schema Database Kafka cluster

Schema configuration

Unlike most dbt adapters, dbt-confluent cannot create or drop schemas — a dbt schema maps to a Flink database (Kafka cluster) in Confluent Cloud, which is managed externally. Both the dbname in your profiles.yml and any model-level schema config must reference an existing Flink database by name:

# dbt_project.yml
models:
  my_project:
    +schema: my-kafka-cluster

Usage

Streaming table

A streaming table creates a table and runs a continuous INSERT query against it:

-- models/pageviews_enriched.sql
{{
  config(
    materialized='streaming_table',
    with={'changelog.mode': 'append'}
  )
}}

SELECT
  p.user_id,
  p.page_url,
  u.username
FROM {{ ref('pageviews') }} p
JOIN {{ ref('users') }} u ON p.user_id = u.user_id

Streaming source

A streaming source creates a connector-backed source table. The model SQL defines the column definitions:

-- models/datagen_users.sql
{{
  config(
    materialized='streaming_source',
    connector='faker',
    with={'rows-per-second': '10'}
  )
}}

`user_id` INT,
`username` STRING,
`email` STRING

Materialized table

A materialized table is maintained continuously by Flink. Each run re-asserts the query; Flink evolves it in place when it changes and no-ops when it doesn't:

-- models/orders_by_status.sql
{{
  config(
    materialized='materialized_table',
    distributed_by={'columns': ['status'], 'buckets': 6}
  )
}}

SELECT status, COUNT(*) AS orders, SUM(amount) AS total
FROM {{ ref('orders') }}
GROUP BY status

Two caveats worth knowing before changing a materialized table model:

  • Changing the definition of a stateful model (aggregations, joins, windows) evolves it in place but silently resets its results: Flink discards the processing state and resumes from current offsets, so totals restart from the change point and pre-change history is never reprocessed. Use --full-refresh to rebuild correct results.
  • --full-refresh drops the materialized table, permanently deleting its backing Kafka topic, all of its data, and the associated Schema Registry schema versions.

See Materializations for the full list and details.

Known Limitations

  • No schema management: Flink databases (Kafka clusters) cannot be created or dropped — they are managed in Confluent Cloud.
  • No table renames: ALTER TABLE RENAME is not supported; to effectively rename a model you must drop and recreate the underlying table, which for table, streaming_table, streaming_source, and materialized_table materializations requires running with --full-refresh.
  • No transactions: Flink SQL is non-transactional.
  • No snapshots: Flink SQL lacks the batch operations (MERGE, UPDATE) required by dbt snapshots.
  • No incremental: dbt's batch-incremental semantics does not map to Flink's continuous processing model. Use streaming_table instead.
  • Drift detection for WITH options: Schema drift detection only verifies that user-specified WITH options exist with correct values. It cannot detect when options are removed from the config (because connectors may add default options that cannot be distinguished from user-specified ones). Use --full-refresh to change or remove WITH options. Drift detection can be disabled per-model with config(on_schema_drift='ignore'). See Materializations for details.
  • Materialized table distribution: a materialized table's distributed_by (columns and buckets) is fixed at creation; changing it requires --full-refresh (drop and recreate). Column, WITH, and query-logic changes evolve in place. See Materializations.
  • Materialized table evolution resets state: evolving a stateful materialized table (aggregations, joins, windows) discards its processing state and resumes from current offsets — results silently restart from the change point and history is not reprocessed. Rebuild with --full-refresh. See Materializations.

Development

git clone https://github.com/confluentinc/dbt-confluent
cd dbt-confluent
uv sync --dev

See CONTRIBUTING.md for changelog and contribution guidelines.

Code quality

uv run ruff check dbt/ tests/
uv run ruff format --check dbt/ tests/

Running tests

Tests require a Confluent Cloud environment. Set the following environment variables (or add them to a test.env file):

export CONFLUENT_ENV_ID=env-xxxxxx
export CONFLUENT_ORG_ID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
export CONFLUENT_COMPUTE_POOL_ID=lfcp-xxxxx
export CONFLUENT_CLOUD_PROVIDER=aws
export CONFLUENT_CLOUD_REGION=us-west-6
export CONFLUENT_TEST_DBNAME=dbname
export CONFLUENT_FLINK_API_KEY=xxx
export CONFLUENT_FLINK_API_SECRET=xxx

# Optional: a second compute pool (same environment + region, different from
# CONFLUENT_COMPUTE_POOL_ID) used only by the per-model compute pool test.
# The test is skipped when this is unset or equal to CONFLUENT_COMPUTE_POOL_ID.
export CONFLUENT_COMPUTE_POOL_ID_2=lfcp-yyyyy

# A Global API key. The Tableflow and function (artifact upload) functional tests need one
# (those control-plane routes require it regardless of the Flink-region pair above -- see
# MATERIALIZATIONS.md#tableflow), and they are skipped if either of these is unset.
export CONFLUENT_GLOBAL_API_KEY=xxx
export CONFLUENT_GLOBAL_API_SECRET=xxx
uv run pytest

By default the functional tests run serially. They can run in parallel with pytest-xdist by passing -n. Tests are distributed per module (--dist=loadscope is already configured), so classes in the same module share a worker.

uv run pytest tests/functional -n 4

Parallel runs have been validated at -n 4.

All workers share one schema, so each test class gets its own namespace: its model relations are aliased with a dbt_test_<session>_<class hash>_ prefix and its statement names start with dbt-test-<session>-<class hash>- (see tests/functional/adapter/isolation.py). When a test refers to a relation by its model name, it must go through physical_name() to get the prefixed name. At the end of each class its relations and statements are dropped, along with any dbt_test_ / dbt-test- leftovers from earlier sessions that are more than two hours old.

Two things opt out of this:

  • A test class can set NAMESPACE_RELATIONS = False to keep its literal model names (the unit test materialization looks the tested model up by name, not alias, so test_unit_materialization.py does this). Those names must be unique across the whole suite, and they are not swept if a run is killed before teardown.

  • A test that can't run alongside others (for example, one that observes the whole shared environment) can be marked @pytest.mark.serial. It fails immediately under -n, so run the rest in parallel with:

    uv run pytest -m "not serial" ./tests/functional
    

    and the marked ones separately, without -n, with:

    uv run pytest -m serial ./tests/functional
    

    This is intended primarily for development purposes, and was used during the introduction of parallel test runs to enable porting modules individually more easily. We should not be committing new tests that cannot be run in parallel.

Versioning

This adapter follows semantic versioning and is versioned independently from dbt Core. Compatibility with dbt Core is declared via dependencies (currently requires dbt-core~=1.11).

License

Apache-2.0 — see LICENSE for details.

Metadata

Release files for dbt-confluent 0.5.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for dbt-confluent 0.5.0
File Size Uploaded Provenance Actions
dbt_confluent-0.5.0.tar.gz 318.9 kB Without provenance Details for dbt_confluent-0.5.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for dbt-confluent 0.5.0
File Interpreter ABI Platform Actions
dbt_confluent-0.5.0-py3-none-any.whl Python 3 none any Details for dbt_confluent-0.5.0-py3-none-any.whl

Total release size: 422.8 kB

Release files / dbt_confluent-0.5.0.tar.gz

Download URL dbt_confluent-0.5.0.tar.gz
Size 318.9 kB
Tags Source
SHA-256 checksum
How to use checksums
fd5b06d68d25487fd1be638d96ba86e35e8307e8d7963d9d80500e41cfab728a
BLAKE2b-256 checksum
How to use checksums
84cd660389f4e08a92a98b6416873259c03fff2faec102d290fe4aa08105d3be
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / dbt_confluent-0.5.0-py3-none-any.whl

Download URL dbt_confluent-0.5.0-py3-none-any.whl
Size 103.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
f7b2d63501a89d7d134d9b0e0d6c89e01e4023f22ec2946527a6ec7a914f361c
BLAKE2b-256 checksum
How to use checksums
e9e0b393718943f74a00a734095ac337f82aa62d651e560755d3dcdfe0a1d7a8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for dbt-confluent 0.5.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page