Version control your dbt Cloud jobs with YML.
Project description
dbt-cloud-jobs
Version control your dbt Cloud jobs with YML.
Installation
pip install dbt-cloud-jobs
Quickstart
-
Create an API token in dbt Cloud:
- Generate a service account token, make sure to grant the
Jobs Admin
permission set. - If you do not have access to create a service account token you can create a User API token. Note that the service account method is preferred.
- Generate a service account token, make sure to grant the
-
Set an environment variable with the value of the token:
export DBT_API_TOKEN="<VALUE_FROM_PREVIOUS_STEP>"
-
Set an environment variable for the region where your dbt Cloud account is hosted. The value must be one of "AU", "Europe" or "US" (see docs here):
export DBT_CLOUD_REGION="<REGION>"
-
Import your existing dbt Cloud jobs:
dbt_cloud_jobs --import --account-id 123456 --file dbt_cloud_jobs.yml
-
Edit the definition of your jobs in
dbt_cloud_jobs.yml
. -
Sync the updated definitions to dbt Cloud:
dbt_cloud_jobs --sync --file dbt_cloud_jobs.yml
Recommended usage in CI/CD
CI
In CI dbt-cloud-jobs
should be used to verify that the provided YML file is valid. For example:
- name: Install dbt_cloud_jobs
run: pip install dbt-cloud-jobs
- name: Validate `dbt_cloud_jobs.yml`
run: dbt_cloud_jobs --validate --file dbt_cloud_jobs.yml
CD
In CD dbt-cloud-jobs
should be used to sync the provided YML file to dbt Cloud. For example:
- name: Install dbt_cloud_jobs
run: pip install dbt-cloud-jobs
- name: Sync `dbt_cloud_jobs.yml`
run: dbt_cloud_jobs --sync --file dbt_cloud_jobs.yml
Example
For an example of how this package can be used, take a look at dbt-cloud-jobs-example-repo
, specifically:
- The
dbt_cloud_jobs.yml
file. - The CI pipeline.
- The CD pipeline.
- This CI pipeline run that validated the jobs definitions added in PR1.
- This CD pipeline run that updated the
Daily job
dbt Cloud job.
Limitations/Warnings
-
Service account tokens are created at the account level. This means that if you have multiple dbt Cloud accounts you will need to create different
dbt_cloud_jobs.yml
files for each account. If you try to usedbt-cloud-jobs
with a file that contains multipleaccount_id
values, an error will be raised. -
⚠️
dbt_cloud_jobs
expects to "own" all the jobs in the projects where it is used. This means that if you are runningdbt_cloud_jobs --sync --file dbt_cloud_jobs.yml
for the first time in a project, any pre-existing jobs not present in yourdbt_cloud_jobs.yml
file will be deleted.
Development
To setup your development environment, fork this repository and run:
poetry shell
poetry install
Set the following environment variables:
export DBT_ACCOUNT_ID=<DBT_ACCOUNT_ID>
export DBT_CLOUD_REGION="<DBT_CLOUD_REGION>"
export DBT_ENVIRONMENT_ID=<DBT_ENVIRONMENT_ID>
export DBT_PROJECT_ID=<DBT_PROJECT_ID>
export DBT_API_TOKEN="<DBT_API_TOKEN>"
It is highly recommended that a dedicated dbt Cloud environment be used for development.
All tests can be run via:
make test
Release
Trigger the Publish to PyPi
workflow, inputting the version to publish to PyPi. This workflow will:
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for dbt_cloud_jobs-0.1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3b8cac90e067e9290e196de68a9d42546c2ed58a293a6b9643214c4c464dc2dd |
|
MD5 | 8203b14d2213d50394e6fc9b7f7a3e97 |
|
BLAKE2b-256 | 12f090cd978928981d2db80b1e094d49263c9ea749d72313a24fb7c72bb5da19 |