Version control your dbt Cloud jobs with YML.
Project description
dbt-cloud-jobs
Version control your dbt Cloud jobs with YML.
Installation
pip install dbt-cloud-jobs
Quickstart
-
Create an API token in dbt Cloud:
- Generate a service account token, make sure to grant the
Jobs Admin
permission set. - If you do not have access to create a service account token you can create a User API token. Note that the service account method is preferred.
- Generate a service account token, make sure to grant the
-
Set an environment variable with the value of the token:
export DBT_API_TOKEN="<VALUE_FROM_PREVIOUS_STEP>"
-
Set an environment variable for the region where your dbt Cloud account is hosted. The value must be one of "AU", "Europe" or "US" (see docs here):
export DBT_CLOUD_REGION="<REGION>"
-
Import your existing dbt Cloud jobs:
dbt_cloud_jobs --import --account-id 123456 --file dbt_cloud_jobs.yml
-
Edit the definition of your jobs in
dbt_cloud_jobs.yml
. -
Sync the updated definitions to dbt Cloud:
dbt_cloud_jobs --sync --file dbt_cloud_jobs.yml
Recommended usage in CI/CD
CI
In CI dbt-cloud-jobs
should be used to verify that the provided YML file is valid. For example:
- name: Install dbt_cloud_jobs
run: pip install dbt-cloud-jobs
- name: Validate `dbt_cloud_jobs.yml`
run: dbt_cloud_jobs --validate --file dbt_cloud_jobs.yml
CD
In CD dbt-cloud-jobs
should be used to sync the provided YML file to dbt Cloud. For example:
- name: Install dbt_cloud_jobs
run: pip install dbt-cloud-jobs
- name: Sync `dbt_cloud_jobs.yml`
run: dbt_cloud_jobs --sync --file dbt_cloud_jobs.yml
Limitations/Warnings
-
Service account tokens are created at the account level. This means that if you have multiple dbt Cloud accounts you will need to create different
dbt_cloud_jobs.yml
files for each account. If you try to usedbt-cloud-jobs
with a file that contains multipleaccount_id
values, an error will be raised. -
⚠️
dbt_cloud_jobs
expects to "own" all the jobs in the projects where it is used. This means that if you are runningdbt_cloud_jobs --sync --file dbt_cloud_jobs.yml
for the first time in a project, any pre-existing jobs not present in yourdbt_cloud_jobs.yml
file will be deleted.
Development
To setup your development environment, fork this repository and run:
poetry shell
poetry install
Set the following environment variables:
export DBT_ACCOUNT_ID=<DBT_ACCOUNT_ID>
export DBT_CLOUD_REGION="<DBT_CLOUD_REGION>"
export DBT_ENVIRONMENT_ID=<DBT_ENVIRONMENT_ID>
export DBT_PROJECT_ID=<DBT_PROJECT_ID>
export DBT_API_TOKEN="<DBT_API_TOKEN>"
It is highly recommended that a dedicated dbt Cloud environment be used for development.
All tests can be run via:
make test
Release
Trigger the Publish to PyPi
workflow, inputting the version to publish to PyPi. This workflow will:
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for dbt_cloud_jobs-0.0.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2e7bf03e3ae03f98f26d04bc9946a4c6a9bdf5f2dd2af400d4337f260ee199d2 |
|
MD5 | fe9ccb41752c0af2d581f826028cede3 |
|
BLAKE2b-256 | a6240ef7abaddf9de0409811350531150ee32affff8b432b24995cab280f5331 |