Skip to main content

Databricks Warehouse

A Python library that wraps the Databricks SQL connector (with optional fallbacks to Databricks Connect) and provides a standard interface for querying SQL warehouses from outside of Databricks.

Overview

This library offers a simplified interface for running SQL queries on a Databricks cluster or warehouse and handling authentication using OAuth.

The goal of this library is to make querying Databricks "just work" in a standard way across projects for Data Scientists. This library is ideal for projects like external apps that sit outside of Databricks, run relatively small queries, and don't operate on PySpark DataFrames. Note that for many projects, especially those that leverage PySpark, the best way to interact with Databricks might be with Databricks Connect, not this library.

Installation

SQL Connector Only

To include databricks_warehouse in your project, add it as a dependency in your project's pyproject.toml:

[tool.poetry.dependencies]
...
databricks_warehouse = { path = "../../libraries/databricks_warehouse", develop = True }

Databricks Connect Fallback

The Databricks SQL Connector can only be used from compute that's running outside of Databricks. To support queries from Databricks jobs as well, the preferred method is to use Databricks Connect. All query methods have built-in fallbacks to Databricks Connect which will be run if we detect we're in a Databricks environment. To include this optional feature, make sure to include the connect extra (excluded by default to limit dependencies):

pip install databricks_warehouse[connect]
poetry add databricks_warehouse --extras connect

Primary Methods

The following methods can be imported from databricks_warehouse directly and are the primary ways that users should interact with this library:

  1. read_databricks -- Run a SQL query against an endpoint and return the results as a Pandas DataFrame.
  2. read_databricks_pl -- Run a SQL query against an endpoint and return the results as a Polars DataFrame.
  3. execute_databricks -- Execute a SQL statement and don't return results. This is useful for non-SELECT statements such as INSERT, CREATE, etc.

Configuration

Databricks Client Unified Authentication

This project uses the Databricks SDK for configuring credentials and cluster information. As such, it adheres to the Databricks Client Unified Authentication protocol: https://docs.databricks.com/en/dev-tools/auth/unified-auth.html. This means that any setup you already have for configuring Databricks connections (namely environment variables and your ~/.databrickscfg file) should directly apply here.

Due to security issues with Personal Access Tokens, this library only supports User-to-Machine (U2M) and Machine-to-Machine (M2M) OAuth.

Environment Variables

Set the following environment variables to configure the connection defaults:

Authentication

Always Required:

  • DATABRICKS_HOST: The URL of the Databricks workspace.

Optional:

  • DATABRICKS_CLIENT_ID and DATABRICKS_CLIENT_SECRET: For Machine-to-Machine OAuth using a service principal. Set this when running from automated jobs as a service principal.

If DATABRICKS_CLIENT_ID and DATABRICKS_CLIENT_SECRET are not provided, we fall back to User-to-Machine OAuth using the Databricks CLI instead. See this documentation for more information.

Cluster Specification

Either:

  • DATABRICKS_WAREHOUSE_ID: The ID of the SQL warehouse to run queries on. This should be the primary way the library is used, as SQL warehouses are highly optimized for queries. Cannot be set with DATABRICKS_CLUSTER_ID.

OR

  • DATABRICKS_CLUSTER_ID: The ID of the cluster you wish to connect to. Cannot be set with DATABRICKS_WAREHOUSE_ID.

Configuration Priority Ordering

As this library leverages the Databricks SDK for configuration, we have the same ability to read settings either from environment variables or from a Databricks Configuration File (~/.databrickscfg). In all cases, environment variables take precedence over configuration profiles. See Authentication order of evaluation for more information.

Examples

from databricks_warehouse import read_databricks, read_databricks_pl, execute_databricks

# By default, the target is determined from environment variables or ~/.databrickscfg settings.
# This will run on whatever $DATABRICKS_CLUSTER_ID / $DATABRICKS_WAREHOUSE_ID specify.
# If running remotely from outside of Databricks, this will use the Databricks SQL Connector.
# If running from a Databricks job, this will use Databricks Spark instead.
df = read_databricks("SELECT * FROM your_table LIMIT 10")

# Settings can also be overridden if you require more flexibility than environment variables allow
df = read_databricks(
    "SELECT * FROM your_table LIMIT 10",
    host="https://your-workspace.cloud.databricks.com",
    warehouse_id="your-warehouse-id",
)

# Same, but returns results as Polars DataFrame instead of Pandas
df_polars = read_databricks_pl("SELECT * FROM your_table LIMIT 10")

# For non-SELECT statements (CREATE, DELETE, INSERT, etc.), use `execute_databricks` instead
execute_databricks("CREATE TABLE dev.my_table (...)")

Metadata

Release files for databricks_warehouse 1.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for databricks_warehouse 1.0.0
File Size Uploaded Provenance Actions
databricks_warehouse-1.0.0.tar.gz 5.3 kB Without provenance Details for databricks_warehouse-1.0.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for databricks_warehouse 1.0.0
File Interpreter ABI Platform Actions
databricks_warehouse-1.0.0-py3-none-any.whl Python 3 none any Details for databricks_warehouse-1.0.0-py3-none-any.whl

Total release size: 10.9 kB

Release files / databricks_warehouse-1.0.0.tar.gz

Download URL databricks_warehouse-1.0.0.tar.gz
Size 5.3 kB
Tags Source
SHA-256 checksum
How to use checksums
cca6f9b71b0b30ae78ecf7789acffab6340fc9e1f2934f3ed1a2c74bd92bd737
BLAKE2b-256 checksum
How to use checksums
bc480545a9ea69e3542a5a81bc61cf231b719f46f9a51dbc449615986a6e3a51
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/1.8.3 CPython/3.12.5 Darwin/24.4.0

Release files / databricks_warehouse-1.0.0-py3-none-any.whl

Download URL databricks_warehouse-1.0.0-py3-none-any.whl
Size 5.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
ba3e821622342e52efd90f678516489a6bac7dc56481c7b95805697dad6c3f7f
BLAKE2b-256 checksum
How to use checksums
27b6948e1b2d1aa48a2e196240c69d08bd2abf68795b9f848055f9ae371eeadc
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/1.8.3 CPython/3.12.5 Darwin/24.4.0

Security for databricks_warehouse 1.0.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.0.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page