DBT Databricks factory
Project description
dbt-databricks-factory
Creates dbt based GCP workflows.
Installation
Use the package manager pip to install dbt-databricks-factory for [dp (data-pipelines-cli)]:
pip install dbt-databricks-factory
Usage
To create a new dbt workflow json schema, run:
python -m dbt_databricks_factory.cli create-job \
--job-name '<job name>' \
--project-dir '<dbt project directory>' \
--profiles-dir '<path to profiles directory>' \
--git-provider '<git provider>' \
--git-url 'https://url.to/repo.git' \
--git-branch 'main' \
--job-cluster my-cluster-name @path/to/cluster_config.json \
--default-task-cluster my-cluster-name \
--library 'dbt-databricks>=1.0.0,<2.0.0' \
--library 'dbt-bigquery==1.3.0' \
--pretty \
path/to/dbt/manifest.json > workflow.json
This workflow will create a json file with the dbt workflow definition. You can then use it to create a new workflow in Databricks by for example post request like here:
curl --fail-with-body -X POST "${DATABRICKS_HOST}api/2.1/jobs/create" \
-H "Authorization: Bearer ${DATABRICKS_TOKEN}" \
-H "Content-Type: application/json" \
-d "@workflow.json" >job_id.json
echo "Job ID:"
cat job_id.json
curl --fail-with-body -X POST "${DATABRICKS_HOST}api/2.1/jobs/run-now" \
-H "Authorization: Bearer ${DATABRICKS_TOKEN}" \
-H "Content-Type: application/json" \
-d @job_id.json >run_id.json
echo "Run ID:"
cat run_id.json
curl --fail-with-body -X GET -G "${DATABRICKS_HOST}api/2.1/jobs/runs/get" \
-H "Authorization: Bearer ${DATABRICKS_TOKEN}" \
-d "run_id=$(jq -r '.run_id' < run_id.json)" >run_status.json
jq < run_status.json
To get more information about the command, run:
python -m dbt_databricks_factory.cli create-job --help
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file dbt-databricks-factory-0.1.1.tar.gz
.
File metadata
- Download URL: dbt-databricks-factory-0.1.1.tar.gz
- Upload date:
- Size: 16.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.11.5
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | d3fa854b246f2f093cca13de7bc9eaa6a3f841cafdddf4c97c871ccd8cf37b26 |
|
MD5 | 6e159ce7c76fab33d36c6b66273ab144 |
|
BLAKE2b-256 | 0a6741c6185baedd9180f02ceeb792c1f72ca75f962aaf1fbc0cef2718308bf3 |
File details
Details for the file dbt_databricks_factory-0.1.1-py3-none-any.whl
.
File metadata
- Download URL: dbt_databricks_factory-0.1.1-py3-none-any.whl
- Upload date:
- Size: 15.1 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.11.5
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2eee92673e854c11f94e51b5593964a3462af3cd2037b576feeee9b1186c2cb1 |
|
MD5 | 841dd1a7e262da77250637a0743122b0 |
|
BLAKE2b-256 | db484c1d01bd0f2b1dbf8c135b445429524b420d6fded82d124598821026fdfa |