dbt-argo

A simple Python package to parse dbt dependencies to create an Argo Workflow job. It can also submit Argo files directly to the server.

Motivations

Simplify CI steps to create Argo workflow for dbt projects.

Limitations

• This version only support Google Cloud Storage as backend to store dbt artifacts to be used by Argo as artifacts.
• This version only support CronWorkflow and Workflow generation

Usage

By default, it assumes the dbt_project.yml file is located at the root folder, if not the DBT_PROJECT_DIR can be defined to locate the project path.

Usage: dbt-argo [OPTIONS] COMMAND [ARGS]...

  Test

Options:
  --help  Show this message and exit.

Commands:
  cp      copy local folder to GCS Bucket
  create  Create a dbt Argo Workflow
  cron    Manage cron workflow
  submit  dbt-argo deploy {FILE}

Inspirations

• Hera for Argo Python CLI + pydantic validation objects
• couler for easy cluster submission (like kubectl apply -f ...)
• dbt > 1.5 for easy manifest parsing directly in python scripts
• Argo CLI for cli options
• kubectl for simple CLI application
• cloudpathlib to deal with cloud file like pathlib