Skip to main content

The Flink adapter plugin for dbt

Project description

DBT-Flink Adapter

This is POC for DBT-FLINK adapter

rerequisities

  • Flink 1.16+
  • Flink SqlGateway

Usage

Before we start to play with dbt adapter, we need to setup Flink cluster with SqlGateway configured and running. To simplify that process we prepared Docker Compose configuration.

Install dbt-flink-adapter

cd project_example
python3 -m venv venv
source venv/bin/activate
pip install dbt-flink-adapter
dbt --version

dbt-flink should be enlisted among plugins

Configure DBT profile

Locate DBT profile on your machine. It should be in home directory under ~/.dbt/profiles.yml Add there below config:

flink_profile:
  target: dev
  outputs:
    dev:
      type: flink
      host: localhost
      port: 8083
      session_name: test_session

Launch Flink cluster

cd envs/kafka
docker compose up

cd envs/flink-1.16
docker compose up

Play with sample DBT project with dbt-flink adapter

dbt test
dbt run

FLink SQL tables should be created on Flink cluster

tear down Flink cluster

docker compose stop

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dbt-flink-adapter-1.3.3.tar.gz (19.7 kB view hashes)

Uploaded Source

Built Distribution

dbt_flink_adapter-1.3.3-py3-none-any.whl (23.9 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page