dbt-spark-livy
The dbt-spark-livy adapter allows you to use dbt along with Apache spark-livy and Cloudera Data Platform with Livy server support. This code bases use the dbt-spark project (https://github.com/dbt-labs/dbt-spark), and provides a Livy connectivity support over it.
Getting started
- Install dbt
- Read the introduction and viewpoint
Running locally
A docker-compose environment starts a Spark Thrift server and a Postgres database as a Hive Metastore backend.
Note: dbt-spark now supports Spark 3.1.1 (formerly on Spark 2.x).
Python >= 3.8
dbt-core ~= 1.3.0
pyspark
sqlparams
requests_kerberos
requests-toolbelt
python-decouple
Installing dbt-spark-livy
pip install dbt-spark-livy
Profile Setup
demo_project:
target: dev
outputs:
dev:
type: spark_livy
method: livy
schema: my_db
host: https://spark-livy-gateway.my.org.com/dbt-spark/cdp-proxy-api/livy_for_spark3/
user: my_user
password: my_pass
Caveats
- While using livy , in the Livy UI if you notice sessions change state to dead from starting instead of idle, make sure there is a proper mapping for the user in the IDBroker mapping section
- Actions > Manage Access > IDBroker Mappings . Reference
- Also make sure the workload password is set either through UI or CLI. Reference
Supported features
Please see the original adapter documentation: https://github.com/dbt-labs/dbt-spark and https://docs.getdbt.com/reference/warehouse-profiles/spark-profile
4 maintainers
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file dbt-spark-livy-1.3.1.tar.gz.
File metadata
- Download URL: dbt-spark-livy-1.3.1.tar.gz
- Upload date:
- Size: 38.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/3.8.0 pkginfo/1.9.2 readme-renderer/34.0 requests/2.27.1 requests-toolbelt/0.10.1 urllib3/1.26.13 tqdm/4.64.1 importlib-metadata/4.8.3 keyring/23.4.1 rfc3986/1.5.0 colorama/0.4.5 CPython/3.6.4
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
094cd6a4c001905c8757cd74df61ab21bd84ff6c1eb219f13ea83afbb3a2154e
|
|
| MD5 |
ec166ab98352e8e5d6b44f59aa79fa22
|
|
| BLAKE2b-256 |
55f80e3e8dd35e8b81a08e54bdb3220e1bc7a18a5fea84b7301e06f3b502901d
|
File details
Details for the file dbt_spark_livy-1.3.1-py3-none-any.whl.
File metadata
- Download URL: dbt_spark_livy-1.3.1-py3-none-any.whl
- Upload date:
- Size: 49.8 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/3.8.0 pkginfo/1.9.2 readme-renderer/34.0 requests/2.27.1 requests-toolbelt/0.10.1 urllib3/1.26.13 tqdm/4.64.1 importlib-metadata/4.8.3 keyring/23.4.1 rfc3986/1.5.0 colorama/0.4.5 CPython/3.6.4
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
9b9fe3063361f1f2b0f50b5d9e59b30dc0a1323691ef5f1f402a9ecbb482295a
|
|
| MD5 |
998fa9273171e157bcbc79df7cfa70d8
|
|
| BLAKE2b-256 |
bbedc78fbef98b728e10b0cc47148cd2743d28b8f0b532fa9db447d9f50710e9
|
