No project description provided
Project description
dbt-spark-cde
The dbt-spark-cde
adapter allows you to use dbt Cloudera Data Platform with CDE API server support. This code bases use the dbt-spark project (https://github.com/dbt-labs/dbt-spark), and provides CDE API based connectivity support over it.
Getting started
- Install dbt
- Read the introduction and viewpoint
Requirements
Python >= 3.9
dbt-core ~= 1.3.0
requests >= 2.28.1
requests-toolbelt >= 0.9.1
pyspark
requests_kerberos
requests-toolbelt
python-decouple
Installing dbt-spark-cde
pip install dbt-spark-cde
Profile Setup
demo_project:
target: dev
outputs:
dev:
type: spark_cde
method: cde
schema: my_db
auth_endpoint: https://service.spark-cde-gateway.my.org.com/
host: https://spark-cde-gateway.my.org.com/dex/api/v1/
user: my_user
password: my_pass
- To obtain auth_endpoint follow the steps here: (https://docs.cloudera.com/data-engineering/cloud/api-access/topics/cde-api-get-access-token.html)
Caveats
- While using cde , in the Livy UI if you notice sessions change state to dead from starting instead of idle, make sure there is a proper mapping for the user in the IDBroker mapping section
- Actions > Manage Access > IDBroker Mappings . Reference
- Also make sure the workload password is set either through UI or CLI. Reference
Supported features
Please see the original adapter documentation: https://github.com/dbt-labs/dbt-spark and https://docs.getdbt.com/reference/warehouse-profiles/spark-profile
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
dbt-spark-cde-1.3.0.tar.gz
(42.1 kB
view hashes)
Built Distribution
Close
Hashes for dbt_spark_cde-1.3.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a3d94f35c5f591b1f6185912c193c51e1a59169adafc1a4b5ba0d4f9ae3b356f |
|
MD5 | ba786f0545543dadf1e0106c8306ecd6 |
|
BLAKE2b-256 | 5fc8ab251f7b7f0b902ae18b86a160c266497f0d3c92148e78c9c9e58489fe46 |