Databricks SQL Connector for Python
The Databricks SQL Connector for Python allows you to develop Python applications that connect to Databricks clusters and SQL warehouses. It is a Thrift-based client with no dependencies on ODBC or JDBC. It conforms to the Python DB API 2.0 specification.
This connector uses Arrow as the data-exchange format, and supports APIs (e.g. fetchmany_arrow) to directly fetch Arrow tables. Arrow tables are wrapped in the ArrowQueue class to provide a natural API to get several rows at a time. PyArrow is required to enable this and use these APIs, you can install it via pip install pyarrow or pip install databricks-sql-connector[pyarrow].
The connector includes built-in support for HTTP/HTTPS proxy servers with multiple authentication methods including basic authentication and Kerberos/Negotiate authentication. See docs/proxy.md and examples/proxy_authentication.py for details.
You are welcome to file an issue here for general use cases. You can also contact Databricks Support here.
Requirements
Python 3.9 or above is required.
Documentation
For the latest documentation, see
For a full reference of every sql.connect(...) keyword argument — type,
default, per-backend support (Thrift vs Kernel), and meaning — see
CONNECTION_PARAMETERS.md.
Quickstart
Installing the core library
Install using pip install databricks-sql-connector
Installing the core library with PyArrow
Install using pip install databricks-sql-connector[pyarrow]
Installing with the Rust kernel backend (use_kernel=True)
Install using pip install databricks-sql-connector[kernel]
This adds the optional databricks-sql-kernel
extension (a native Rust client core, exposed via PyO3). Pass
use_kernel=True to sql.connect(...) to route the connection through it
instead of the default Thrift backend:
connection = sql.connect(
server_hostname=host,
http_path=http_path,
access_token=token,
use_kernel=True,
)
Notes:
- Requires Python >= 3.10 (the kernel wheel is published as
cp310-abi3). On older interpreters the[kernel]extra installs nothing anduse_kernel=Trueraises anImportError. - The extra also pulls in PyArrow, which the kernel result path requires.
- Authentication supports PAT (
access_token), OAuth M2M/U2M, and SP-wide workload identity federation (identity_federation_client_id).
export DATABRICKS_HOST=********.databricks.com
export DATABRICKS_HTTP_PATH=/sql/1.0/endpoints/****************
Example usage:
import os
from databricks import sql
host = os.getenv("DATABRICKS_HOST")
http_path = os.getenv("DATABRICKS_HTTP_PATH")
connection = sql.connect(
server_hostname=host,
http_path=http_path)
cursor = connection.cursor()
cursor.execute('SELECT :param `p`, * FROM RANGE(10)', {"param": "foo"})
result = cursor.fetchall()
for row in result:
print(row)
cursor.close()
connection.close()
In the above example:
server-hostnameis the Databricks instance host name.http-pathis the HTTP Path either to a Databricks SQL endpoint (e.g. /sql/1.0/endpoints/1234567890abcdef), or to a Databricks Runtime interactive cluster (e.g. /sql/protocolv1/o/1234567890123456/1234-123456-slid123)
Note: This example uses Databricks OAuth U2M to authenticate the target Databricks user account and needs to open the browser for authentication. So it can only run on the user's machine.
Transaction Support
The connector supports multi-statement transactions with manual commit/rollback control. Set connection.autocommit = False to disable autocommit mode, then use connection.commit() and connection.rollback() to control transactions.
For detailed documentation, examples, and best practices, see TRANSACTIONS.md.
SQLAlchemy
Starting from databricks-sql-connector version 4.0.0 SQLAlchemy support has been extracted to a new library databricks-sqlalchemy.
- Github repository databricks-sqlalchemy github
- PyPI databricks-sqlalchemy pypi
Quick SQLAlchemy guide
Users can now choose between using the SQLAlchemy v1 or SQLAlchemy v2 dialects with the connector core
- Install the latest SQLAlchemy v1 using
pip install databricks-sqlalchemy~=1.0 - Install SQLAlchemy v2 using
pip install databricks-sqlalchemy
Contributing
See CONTRIBUTING.md
License
2 maintainers
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file databricks_sql_connector-4.5.0.tar.gz.
File metadata
- Download URL: databricks_sql_connector-4.5.0.tar.gz
- Upload date:
- Size: 239.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.12.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
69f7ac2cc35f77f78c62ffb1aa09dda3aeec3c5037b9d7eddca78ee2e81f046e
|
|
| MD5 |
efd359433914cdcb19d26096fcab78d9
|
|
| BLAKE2b-256 |
b4017cc496468e38485b5577eec978a72572d9631b15945912cb9f51fd9db241
|
File details
Details for the file databricks_sql_connector-4.5.0-py3-none-any.whl.
File metadata
- Download URL: databricks_sql_connector-4.5.0-py3-none-any.whl
- Upload date:
- Size: 265.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.12.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
2b73a5e3688621873cec90ef27ca367ad30a28a8c03c65472638003440a5408c
|
|
| MD5 |
9d3f802543674314e4fa7c7ab961b886
|
|
| BLAKE2b-256 |
17a27bd3b6f494fc68d4b54bc7ca41061efee078dc4d0ecbadcee2b05dfbfd1c
|
