dbt-spark-livy

The dbt-spark-livy adapter allows you to use dbt along with Apache spark-livy and Cloudera Data Platform with Livy server support. This code bases use the dbt-spark project (https://github.com/dbt-labs/dbt-spark), and provides a Livy connectivity support over it.

Getting started

Install dbt
Read the introduction and viewpoint

Running locally

A docker-compose environment starts a Spark Thrift server and a Postgres database as a Hive Metastore backend. Note: dbt-spark now supports Spark 3.1.1 (formerly on Spark 2.x).

Python >= 3.8

dbt-core ~= 1.3.0

pyspark

sqlparams

requests_kerberos

requests-toolbelt

python-decouple

Installing dbt-spark-livy

pip install dbt-spark-livy

Profile Setup

demo_project:
  target: dev
  outputs:
    dev:
     type: spark_livy
     method: livy
     schema: my_db
     host: https://spark-livy-gateway.my.org.com/dbt-spark/cdp-proxy-api/livy_for_spark3/
     user: my_user
     password: my_pass

Caveats

While using livy , in the Livy UI if you notice sessions change state to dead from starting instead of idle, make sure there is a proper mapping for the user in the IDBroker mapping section
Actions > Manage Access > IDBroker Mappings . Reference
Also make sure the workload password is set either through UI or CLI. Reference

Supported features

Please see the original adapter documentation: https://github.com/dbt-labs/dbt-spark and https://docs.getdbt.com/reference/warehouse-profiles/spark-profile

Metadata

Release files for dbt-spark-livy 1.3.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for dbt-spark-livy 1.3.1
File	Size	Uploaded
dbt-spark-livy-1.3.1.tar.gz	38.6 kB	Dec 5, 2022	Details

Built distribution (wheel)

Table of built distributions (wheels) for dbt-spark-livy 1.3.1
File	Interpreter	ABI	Platform	Reset
dbt_spark_livy-1.3.1-py3-none-any.whl 49.8 kB Dec 5, 2022	Python 3	none	any	Details

Total release size: 88.4 kB

Release files / dbt-spark-livy-1.3.1.tar.gz

Download URL	dbt-spark-livy-1.3.1.tar.gz
Size	38.6 kB
Tags	Source
SHA-256 checksum How to use checksums	`094cd6a4c001905c8757cd74df61ab21bd84ff6c1eb219f13ea83afbb3a2154e`
BLAKE2b-256 checksum How to use checksums	`55f80e3e8dd35e8b81a08e54bdb3220e1bc7a18a5fea84b7301e06f3b502901d`
Upload date	Dec 5, 2022
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/3.8.0 pkginfo/1.9.2 readme-renderer/34.0 requests/2.27.1 requests-toolbelt/0.10.1 urllib3/1.26.13 tqdm/4.64.1 importlib-metadata/4.8.3 keyring/23.4.1 rfc3986/1.5.0 colorama/0.4.5 CPython/3.6.4`

Release files / dbt_spark_livy-1.3.1-py3-none-any.whl

Download URL	dbt_spark_livy-1.3.1-py3-none-any.whl
Size	49.8 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`9b9fe3063361f1f2b0f50b5d9e59b30dc0a1323691ef5f1f402a9ecbb482295a`
BLAKE2b-256 checksum How to use checksums	`bbedc78fbef98b728e10b0cc47148cd2743d28b8f0b532fa9db447d9f50710e9`
Upload date	Dec 5, 2022
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/3.8.0 pkginfo/1.9.2 readme-renderer/34.0 requests/2.27.1 requests-toolbelt/0.10.1 urllib3/1.26.13 tqdm/4.64.1 importlib-metadata/4.8.3 keyring/23.4.1 rfc3986/1.5.0 colorama/0.4.5 CPython/3.6.4`