dbt-gen 0.1.7

Tool to generate dbt resources.

dbt-gen

Tool for generating dbt model, source files.

Commands

1. Generate dbt project from cookiecutter template

dbt_gen generate_dbt

2. Generate source

dbt_gen generate_source --help

usage: dbt_gen generate_source [-h] [--profile-path PROFILE_PATH]
                               [--profile-name PROFILE_NAME] [--target TARGET]
                               [--database DATABASE] [--threads THREADS]
                               source_folder

positional arguments:
  source_folder         Folder to write source YAML files

optional arguments:
  -h, --help            show this help message and exit
  --profile-path PROFILE_PATH
                        Path to dbt profile YAML. Default is
                        /home/tien/.dbt/profiles.yml
  --profile-name PROFILE_NAME
                        Dbt profile name. Default is `default`.
  --target TARGET       Dbt profile target. Default is `dev`.
  --database DATABASE   Database to inspect. Default is the database in
                        profile.
  --threads THREADS     Max threads. Default is your machine number of
                        threads.

3. Generate base models and tests

dbt_gen generate_base_model --help

usage: dbt-gen generate_base_model [-h] [--profile-path PROFILE_PATH]
                                   [--profile-name PROFILE_NAME]
                                   [--target TARGET] [--threads THREADS]
                                   [--template TEMPLATE]
                                   source_path output_folder

positional arguments:
  source_path           Path to dbt source YAML.
  output_folder         Folder to write base models.

optional arguments:
  -h, --help            show this help message and exit
  --profile-path PROFILE_PATH
                        Path to dbt profile YAML. Default is
                        /home/tien/.dbt/profiles.yml
  --profile-name PROFILE_NAME
                        Dbt profile name. Default is `default`.
  --target TARGET       Dbt profile target. Default is `dev`.
  --threads THREADS     Max threads. Default is your machine number of
                        threads.
  --template TEMPLATE   Path to custom Jinja template.

3.1 Generate base models using custom Jinja template

• Get default Jinja template by running

dbt-gen get_template > your_custom_template.sql

• Edit your custom template
• Run generating base models using the custom template

dbt-gen generate_base_models --template your_custom_template.sql path/to/source path/to/output

3.2. Jinja variables in rendering base models

• source_name
  • Type: String
• table_name
  • Type: String
• columns
  • Type: Array of objects
  • Item:
    • Type: object
    • Attributes:
      • database: String
      • schema: String
      • table: String
      • name: String
      • position: Integer
      • nullable: Boolean
      • dtype: String
      • extra: Dict

4. Generate base tests

dbt_gen generate_base_test --help

usage: dbt-gen generate_base_test [-h] [--profile-path PROFILE_PATH]
                                  [--profile-name PROFILE_NAME]
                                  [--target TARGET]
                                  source_path output_folder

positional arguments:
  source_path           Path to dbt source YAML.
  output_folder         Folder to write base models.

optional arguments:
  -h, --help            show this help message and exit
  --profile-path PROFILE_PATH
                        Path to dbt profile YAML. Default is
                        /home/tien/.dbt/profiles.yml
  --profile-name PROFILE_NAME
                        Dbt profile name. Default is `default`.
  --target TARGET       Dbt profile target. Default is `dev`.

Caveats

• For BigQuery sources, dbt-gen supports making tests only based on Stitch meta table _sdc_primary_keys. If there is no such table, test YAML will be empty.
• For Snowflake, dbt-gen makes tests based on Snowflake table constraints.

TODOs

• Add primary keys information into variables in rendering base model SQLs