Skip to main content

The Databricks adapter plugin for dbt

Project description

databricks logo dbt logo

Unit Tests Badge Integration Tests Badge

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.

The Databricks Lakehouse provides one simple platform to unify all your data, analytics and AI workloads.

dbt-databricks

The dbt-databricks adapter contains all of the code enabling dbt to work with Databricks. This adapter is based off the amazing work done in dbt-spark. Some key features include:

  • Easy setup. No need to install an ODBC driver as the adapter uses pure Python APIs.
  • Open by default. For example, it uses the the open and performant Delta table format by default. This has many benefits, including letting you use MERGE as the the default incremental materialization strategy.
  • Support for Unity Catalog. dbt-databricks>=1.1.1 supports the 3-level namespace of Unity Catalog (catalog / schema / relations) so you can organize and secure your data the way you like.
  • Performance. The adapter generates SQL expressions that are automatically accelerated by the native, vectorized Photon execution engine.

Choosing between dbt-databricks and dbt-spark

If you are developing a dbt project on Databricks, we recommend using dbt-databricks for the reasons noted above.

dbt-spark is an actively developed adapter which works with Databricks as well as Apache Spark anywhere it is hosted e.g. on AWS EMR.

Getting started

Installation

Install using pip:

pip install dbt-databricks

Upgrade to the latest version

pip install --upgrade dbt-databricks

Profile Setup

your_profile_name:
  target: dev
  outputs:
    dev:
      type: databricks
      catalog: [optional catalog name, if you are using Unity Catalog, only available in dbt-databricks>=1.1.1]
      schema: [database/schema name]
      host: [your.databrickshost.com]
      http_path: [/sql/your/http/path]
      token: [dapiXXXXXXXXXXXXXXXXXXXXXXX]

Quick Starts

These following quick starts will get you up and running with the dbt-databricks adapter:

Compatibility

The dbt-databricks adapter has been tested:

  • with Python 3.7 or above.
  • against Databricks SQL and Databricks runtime releases 9.1 LTS and later.

Tips and Tricks

Choosing compute for a Python model

You can override the compute used for a specific Python model by setting the http_path property in model configuration. This can be useful if, for example, you want to run a Python model on an All Purpose cluster, while running SQL models on a SQL Warehouse. Note that this capability is only available for Python models.

def model(dbt, session):
    dbt.config(
      http_path="sql/protocolv1/..."
    )

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dbt_databricks-1.10.4.tar.gz (78.9 kB view details)

Uploaded Source

Built Distribution

dbt_databricks-1.10.4-py3-none-any.whl (126.9 kB view details)

Uploaded Python 3

File details

Details for the file dbt_databricks-1.10.4.tar.gz.

File metadata

  • Download URL: dbt_databricks-1.10.4.tar.gz
  • Upload date:
  • Size: 78.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: python-httpx/0.28.1

File hashes

Hashes for dbt_databricks-1.10.4.tar.gz
Algorithm Hash digest
SHA256 468be03dd9521630d2edc7973bbef719aed8dbc4b11345fdc03d1fc17a30d9e3
MD5 55738d1ad31edb253d06feaf82cccb0a
BLAKE2b-256 1ae17b1510f11517697d7cfdeeafb3db4f390e98deec47702f0b97e516991f1c

See more details on using hashes here.

File details

Details for the file dbt_databricks-1.10.4-py3-none-any.whl.

File metadata

File hashes

Hashes for dbt_databricks-1.10.4-py3-none-any.whl
Algorithm Hash digest
SHA256 55266a39e881a84f7ea4522a59d5e39779d14dc1b89a9347c74e5c087a9403be
MD5 9b7a3af9861069a2565d3be5a65d8b7c
BLAKE2b-256 e3ebc52b922352b0c5a0860125169690ff0200a36d7f80e134aebd17331a94f8

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page