Skip to main content

DBT Databricks factory

Project description

dbt-databricks-factory

Python Version PyPI Version Downloads

Creates dbt based GCP workflows.

Installation

Use the package manager pip to install dbt-databricks-factory for [dp (data-pipelines-cli)]:

pip install dbt-databricks-factory

Usage

To create a new dbt workflow json schema, run:

python -m dbt_databricks_factory.cli create-job \
    --job-name '<job name>' \
    --project-dir '<dbt project directory>' \
    --profiles-dir '<path to profiles directory>' \
    --git-provider '<git provider>' \
    --git-url 'https://url.to/repo.git' \
    --git-branch 'main' \
    --job-cluster my-cluster-name @path/to/cluster_config.json \
    --default-task-cluster my-cluster-name \
    --library 'dbt-databricks>=1.0.0,<2.0.0' \
    --library 'dbt-bigquery==1.3.0' \
    --pretty \
    path/to/dbt/manifest.json > workflow.json

This workflow will create a json file with the dbt workflow definition. You can then use it to create a new workflow in Databricks by for example post request like here:

curl --fail-with-body -X POST "${DATABRICKS_HOST}api/2.1/jobs/create" \
-H "Authorization: Bearer ${DATABRICKS_TOKEN}" \
-H "Content-Type: application/json" \
-d "@workflow.json" >job_id.json

echo "Job ID:"
cat job_id.json
curl --fail-with-body -X POST "${DATABRICKS_HOST}api/2.1/jobs/run-now" \
-H "Authorization: Bearer ${DATABRICKS_TOKEN}" \
-H "Content-Type: application/json" \
-d @job_id.json >run_id.json

echo "Run ID:"
cat run_id.json
curl --fail-with-body -X GET -G "${DATABRICKS_HOST}api/2.1/jobs/runs/get" \
-H "Authorization: Bearer ${DATABRICKS_TOKEN}" \
-d "run_id=$(jq -r '.run_id' < run_id.json)" >run_status.json

jq < run_status.json

To get more information about the command, run:

python -m dbt_databricks_factory.cli create-job --help

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dbt-databricks-factory-0.1.1.tar.gz (16.3 kB view details)

Uploaded Source

Built Distribution

dbt_databricks_factory-0.1.1-py3-none-any.whl (15.1 kB view details)

Uploaded Python 3

File details

Details for the file dbt-databricks-factory-0.1.1.tar.gz.

File metadata

File hashes

Hashes for dbt-databricks-factory-0.1.1.tar.gz
Algorithm Hash digest
SHA256 d3fa854b246f2f093cca13de7bc9eaa6a3f841cafdddf4c97c871ccd8cf37b26
MD5 6e159ce7c76fab33d36c6b66273ab144
BLAKE2b-256 0a6741c6185baedd9180f02ceeb792c1f72ca75f962aaf1fbc0cef2718308bf3

See more details on using hashes here.

File details

Details for the file dbt_databricks_factory-0.1.1-py3-none-any.whl.

File metadata

File hashes

Hashes for dbt_databricks_factory-0.1.1-py3-none-any.whl
Algorithm Hash digest
SHA256 2eee92673e854c11f94e51b5593964a3462af3cd2037b576feeee9b1186c2cb1
MD5 841dd1a7e262da77250637a0743122b0
BLAKE2b-256 db484c1d01bd0f2b1dbf8c135b445429524b420d6fded82d124598821026fdfa

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page