dbt-remote project

This package provides

• dbt-remote, a drop-in replacement for the dbt CLI for analytics engineers.
• dbt-server, a Cloud Run API that will need to be deployed to perform the remote dbt runs (for data platform engineers: How to deploy?).

dbt-remote

This CLI runs dbt commands remotely on GCP-hosted server.

Requirements

• An initialized dbt core project. (dbt core quickstart)
• The gcloud CLI. (gcloud install guide)
• A deployed dbt-server. (dbt-server deployment guide)

Installation

python3 -m pip install gcp-dbt-remote --no-cache-dir

Refresh your shell/venv to enable the cli:

source venv/bin/activate

OR

conda activate

Setup your GCP project, gcloud CLI, and default credentials

export PROJECT_ID=<your-gcp-project-id>

gcloud auth login
gcloud auth application-default login
gcloud config set project $PROJECT_ID

Make sure your dbt project is properly setup locally.

dbt debug

> All checks passed!

Test the CLI installation (requires you to have deployed the dbt-server)

dbt-remote debug

> INFO    [dbt] All checks passed!
> INFO    [job] Command successfully executed

Use dbt-remote just like you would do with the regular dbt CLI

dbt-remote run

dbt-remote run --select my_first_dbt_model

View all dbt-remote options

dbt-remote --help

Schedule dbt runs

Use the --schedule option and a cron expression to schedule a run. Help with cron expressions.

dbt-remote run --schedule '0 8 * * *'

[...]
Sending request to server...
Job run scheduled at 0 8 * * * (At 08:00 AM) with uuid: e11f1085-8ad9-4dcd-b09f-d8a8369075b9

This will create a cloud scheduler that will call the dbt-server at the configured time.

You can also declare a run schedule using a config file. Be aware that this is authoritative and will replace any previously scheduled runs not in the file. It will also update the manifest for the scheduled runs.

# schedules.yaml
schedule-1:
   command: build --select my_first_dbt_model --project-dir=tests/dbt_project
   schedule: "2 3 4 5 6"

schedule-3:
   command: build --select my_first_dbt_model --project-dir=tests/dbt_project
   schedule: "2 3 4 5 6"

dbt-remote schedules set schedules.yaml

[...]
The following actions will be performed:
+ Add: schedule-1
- Delete: schedule-2
~ Redeploy: schedule-3
Do you want to continue? [y/N]: y

To check your scheduled runs, either go to the cloud scheduler UI of your project, or list them uting the cli:

dbt-remote schedules list

[...]
dbt-server-e11f1085-8ad9-4dcd-b09f-d8a8369075b9
   command: run
   schedule: 0 8 * * * (At 08:00 AM) UTC
   target: https://dbt-server-vo6sb27zvq-ew.a.run.app/schedule/e11f1085-8ad9-4dcd-b09f-d8a8369075b9/start

You can also delete them in the UI, or using the CLI:

dbt-remote schedules delete e11f1085-8ad9-4dcd-b09f-d8a8369075b9

[...]
Schedule dbt-server-e11f1085-8ad9-4dcd-b09f-d8a8369075b9 deleted

(optional) Set persistent configurations for dbt-remote using config command

dbt-remote config set server_url=http://myserver.com location=europe-west9

View all configuration options

dbt-remote config help