The dbt-spark-livy adapter plugin for Spark in Cloudera DataHub with Livy interface
Project description
dbt-spark-livy
The dbt-spark-livy
adapter allows you to use dbt along with Apache spark-livy and Cloudera Data Platform with Livy server support. This code bases use the dbt-spark project (https://github.com/dbt-labs/dbt-spark), and provides a Livy connectivity support over it.
Getting started
- Install dbt
- Read the introduction and viewpoint
Running locally
A docker-compose
environment starts a Spark Thrift server and a Postgres database as a Hive Metastore backend.
Note: dbt-spark now supports Spark 3.1.1 (formerly on Spark 2.x).
Python >= 3.8
dbt-core ~= 1.3.0
pyspark
sqlparams
requests_kerberos
requests-toolbelt
python-decouple
Installing dbt-spark-livy
pip install dbt-spark-livy
Profile Setup
demo_project:
target: dev
outputs:
dev:
type: spark_livy
method: livy
schema: my_db
host: https://spark-livy-gateway.my.org.com/dbt-spark/cdp-proxy-api/livy_for_spark3/
user: my_user
password: my_pass
Caveats
- While using livy , in the Livy UI if you notice sessions change state to dead from starting instead of idle, make sure there is a proper mapping for the user in the IDBroker mapping section
- Actions > Manage Access > IDBroker Mappings . Reference
- Also make sure the workload password is set either through UI or CLI. Reference
Supported features
Please see the original adapter documentation: https://github.com/dbt-labs/dbt-spark and https://docs.getdbt.com/reference/warehouse-profiles/spark-profile
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for dbt_spark_livy-1.3.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9b9fe3063361f1f2b0f50b5d9e59b30dc0a1323691ef5f1f402a9ecbb482295a |
|
MD5 | 998fa9273171e157bcbc79df7cfa70d8 |
|
BLAKE2b-256 | bbedc78fbef98b728e10b0cc47148cd2743d28b8f0b532fa9db447d9f50710e9 |