dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
The Databricks Lakehouse provides one simple platform to unify all your data, analytics and AI workloads.
dbt-databricks
The dbt-databricks adapter contains all of the code enabling dbt to work with Databricks. This adapter is based off the amazing work done in dbt-spark. Some key features include:
- Easy setup. No need to install an ODBC driver as the adapter uses pure Python APIs.
- Open by default. For example, it uses the the open and performant Delta table format by default. This has many benefits, including letting you use
MERGEas the the default incremental materialization strategy. - Support for Unity Catalog. dbt-databricks supports the 3-level namespace of Unity Catalog (catalog / schema / relations) so you can organize and secure your data the way you like.
- Performance. The adapter generates SQL expressions that are automatically accelerated by the native, vectorized Photon execution engine.
Choosing between dbt-databricks and dbt-spark
If you are developing a dbt project on Databricks, we recommend using dbt-databricks for the reasons noted above.
dbt-spark is an actively developed adapter which works with Databricks as well as Apache Spark anywhere it is hosted e.g. on AWS EMR.
Getting started
Installation
Install using pip:
pip install dbt-databricks
Upgrade to the latest version
pip install --upgrade dbt-databricks
Profile Setup
your_profile_name:
target: dev
outputs:
dev:
type: databricks
catalog: [optional catalog name, if you are using Unity Catalog]
schema: [database/schema name]
host: [your.databrickshost.com]
http_path: [/sql/your/http/path]
token: [dapiXXXXXXXXXXXXXXXXXXXXXXX]
Documentation
For documentation on Databricks-specific features, configurations, and capabilities:
- Databricks configurations - Complete reference for all Databricks-specific model configurations, materializations, and incremental strategies
- Connect to Databricks - Setup and authentication guide
Quick Starts
These following quick starts will get you up and running with the dbt-databricks adapter:
- Set up your dbt project with Databricks
- Using dbt Cloud with Databricks (Azure | AWS)
- Submitting Python models as Databricks Workflows
- Running a dbt project as a Databricks job
- Using Unity Catalog with dbt-databricks
- Continuous integration in dbt
- Loading data from S3 into Delta using the databricks_copy_into macro
- Contribute to this repository
Compatibility
The dbt-databricks adapter has been tested:
- with Python 3.7 or above.
- against
Databricks SQLandDatabricks runtime releases 9.1 LTSand later.
Tips and Tricks
Choosing compute for a Python model
You can override the compute used for a specific Python model by setting the http_path property in model configuration. This can be useful if, for example, you want to run a Python model on an All Purpose cluster, while running SQL models on a SQL Warehouse. Note that this capability is only available for Python models.
def model(dbt, session):
dbt.config(
http_path="sql/protocolv1/..."
)
Python models and ANSI mode
When ANSI mode is enabled (spark.sql.ansi.enabled=true), there are limitations when using pandas DataFrames in Python models:
-
Regular pandas DataFrames: dbt-databricks will automatically handle conversion even when ANSI mode is enabled, falling back to
spark.createDataFrame()if needed. -
pandas-on-Spark DataFrames: If you create pandas-on-Spark DataFrames directly in your model (using
pyspark.pandasordatabricks.koalas), you may encounter errors with ANSI mode enabled. In this case, you have two options:- Disable ANSI mode for your session: Set
spark.sql.ansi.enabled=falsein your cluster or SQL warehouse configuration - Set the pandas-on-Spark option in your model code:
import pyspark.pandas as ps ps.set_option('compute.fail_on_ansi_mode', False)
Note: This may cause unexpected behavior as pandas-on-Spark follows pandas semantics (returning null/NaN for invalid operations) rather than ANSI SQL semantics (raising errors).
- Disable ANSI mode for your session: Set
For more information about ANSI mode and its implications, see the Spark documentation on ANSI compliance.
4 maintainers
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file dbt_databricks-1.12.4.tar.gz.
File metadata
- Download URL: dbt_databricks-1.12.4.tar.gz
- Upload date:
- Size: 121.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/7.0.0 CPython/3.10.20
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
55cf7fe840bf8f59c96ce2bed3f9ccb3c416d33e745e0c4e6c732726ef74396c
|
|
| MD5 |
6f45be565016b40fecd00b038e6f8a8f
|
|
| BLAKE2b-256 |
49aae90532b62bdbf173613b5ffb1eab795491c4337bdc27a40a9d4f1d14b386
|
File details
Details for the file dbt_databricks-1.12.4-py3-none-any.whl.
File metadata
- Download URL: dbt_databricks-1.12.4-py3-none-any.whl
- Upload date:
- Size: 179.1 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/7.0.0 CPython/3.10.20
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
c200d901394cec48972e73319ca41e8fbc3346f93c6d403f6c638ccf8ddf5a2d
|
|
| MD5 |
b18e1ce5d6bb0da94a4c627670793925
|
|
| BLAKE2b-256 |
69bb8b7db00f8998df1bdd5f25d00cb6972e01e8066550dc950f6f4735f83664
|
