dbt-contracts 1.0.7

Enforce standards for your dbt projects through automated checks and generators

dbt-contracts

Enforce standards for your dbt projects through automated checks and generators

• Validate that the metadata and properties of objects in your project match required standards
• Automatically generate properties in your project from their related database objects
• Apply complex filtering and validation rules setting for highly granular operations
• Execute these operations as pre-commit hooks for automatic project validation

Contents

• Installation
• Quick Start
• Pre-commit configuration
• Contracts Reference
  • Models
  • Model Columns
  • Sources
  • Source Columns
  • Macros
  • Macro Arguments
• Release History
• Contributing and Reporting Issues

Installation

Install through pip using one of the following commands:

pip install dbt-contracts

python -m pip install dbt-contracts

dbt-contracts is best utilised when used in conjunction with pre-commit hooks. Follow the installation guide for pre-commit to set this up if needed.

Quick Start

1. Create a contracts file. By default, the package will look for a file named contracts in the root of the repository. An example is provided below. For a full reference of the available configuration for this file, check out the documentation.

2. If configured, run dbt-generate to generate properties files from database objects. It can be useful to run this before validations if your validations require properties set which can be generated from database objects.

3. If configured, run dbt-validate to validate your contracts against the terms set in the configuration file.

4. Once you are satisfied with your configuration and the validations are passing, you may want to set pre-commit hooks to automatically validate your project when running git commands against it.

Example configuration

contracts:
  macros:
  - filter:
    - path:
        include: &id002
        - ^\w+\d+\s{1,3}$
        - include[_-]this
        exclude: &id001
        - ^\w+\d+\s{1,3}$
        - exclude[_-]this
        match_all: true
    validations:
    - has_properties
    arguments:
    - filter:
      - name:
          include: .*i\s+am\s+a\s+regex\s+pattern.*
          exclude: *id001
          match_all: true
      validations:
      - has_description
  sources:
  - filter:
    - tag:
        tags:
        - tag1
        - tag2
    - name:
        include: *id002
        exclude: .*i\s+am\s+a\s+regex\s+pattern.*
        match_all: false
    - path:
        include: .*i\s+am\s+a\s+regex\s+pattern.*
        exclude: *id001
        match_all: false
    validations:
    - has_tests:
        min_count: 2
        max_count: 6
    - has_required_tags:
        tags:
        - tag1
        - tag2
    - has_properties
    - has_required_meta_keys:
        keys:
        - key1
        - key2
    - has_allowed_tags:
        tags: &id003
        - tag1
        - tag2
    - exists
    generator:
      exclude:
      - columns
      - description
      filename: config.yml
      depth: 2
      description:
        overwrite: false
        terminator: __END__
      columns:
        overwrite: false
        add: true
        remove: false
        order: false
    columns:
    - filter:
      - meta:
          meta:
            key1: val1
            key2:
            - val2
            - val3
      - tag:
          tags: tag1
      validations:
      - has_matching_data_type:
          ignore_whitespace: true
          case_insensitive: false
          compare_start_only: false
      - has_description
      - has_matching_description:
          ignore_whitespace: false
          case_insensitive: true
          compare_start_only: false
      - has_allowed_meta_keys:
          keys: key1
      - has_matching_index:
          ignore_whitespace: true
          case_insensitive: false
          compare_start_only: true
      - has_data_type
      - has_tests:
          min_count: 2
          max_count: 4
      - has_allowed_tags:
          tags: *id003
      generator:
        exclude: data_type
        description:
          overwrite: true
          terminator: \n
        data_type:
          overwrite: false

Pre-commit configuration

This package is best utilised when used as in conjunction with pre-commit hooks. Follow the installation guide below to set this up if needed.

Each contract operation is set up to take a list files that have changed since the last commit as is required for pre-commit hooks to function as expected.

Set up and add the dbt-contracts operations to your .pre-commit-hooks.yaml file like the example below.

default_stages: [manual]

repos:
 - repo: meta
   hooks:
     - id: identity
       name: List files
       stages: [ manual, pre-commit ]
 - repo: https://github.com/geo-martino/dbt-contracts
   rev: v1.0.0
   hooks:
     - id: dbt-clean
       stages: [manual, pre-commit]
       additional_dependencies: [dbt-postgres]
     - id: dbt-deps
       stages: [manual]
       additional_dependencies: [dbt-postgres]
     - id: dbt-validate
       alias: dbt-validate-no-output
       name: Run models contracts
       stages: [pre-commit]
       args:
         - --contract
         - models
       additional_dependencies: [dbt-postgres]
     - id: dbt-validate
       alias: dbt-validate-no-output
       name: Run model columns contracts
       stages: [pre-commit]
       args:
         - --contract
         - models.columns
       additional_dependencies: [dbt-postgres]
     - id: dbt-validate
       alias: dbt-validate-no-output
       name: Run sources contracts
       stages: [pre-commit]
       args:
         - --contract
         - sources
       additional_dependencies: [dbt-postgres]
     - id: dbt-validate
       alias: dbt-validate-no-output
       name: Run source columns contracts
       stages: [pre-commit]
       args:
         - --contract
         - sources.columns
       additional_dependencies: [dbt-postgres]

     - id: dbt-validate
       alias: dbt-validate-output-annotations
       name: Run all contracts
       stages: [manual]
       args:
         - --format
         - github-annotations
         - --output
         - contracts_results.json
       additional_dependencies: [dbt-postgres]
     - id: dbt-generate
       name: Generate properties for all contracts
       stages: [manual]
       additional_dependencies: [dbt-postgres]

Contracts Reference

Below you will find a list of all available contracts grouped by the dbt object it operates on. Refer to this list to help when designing your contracts file.

Models

Filters

• name: Filter models based on their names.
• path: Filter models based on their paths.
• tag: Filter models based on their tags.
• meta: Filter models based on their meta values.
• is_materialized: Filter models taking only those which are not ephemeral.

Terms

• has_properties: Check whether the models have properties files defined.
• has_description: Check whether the models have descriptions defined in their properties.
• has_required_tags: Check whether the models have the expected set of required tags set.
• has_allowed_tags: Check whether the models have only tags set from a configured permitted list.
• has_required_meta_keys: Check whether the models have the expected set of required meta keys set.
• has_allowed_meta_keys: Check whether the models have only meta keys set from a configured permitted list.
• has_allowed_meta_values: Check whether the models have only meta values set from a configured permitted mapping of keys to values.
• exists: Check whether the models exist in the database.
• has_tests: Check whether models have an appropriate number of tests configured.
• has_all_columns: Check whether models have all columns set in their properties.
• has_expected_columns: Check whether models have the expected names of columns set in their properties.
• has_matching_description: Check whether the descriptions configured in models' properties match the descriptions in the database.
• has_contract: Check whether models have appropriate configuration for a contract in their properties.
• has_valid_ref_dependencies: Check whether models have an appropriate number of upstream dependencies
• has_valid_source_dependencies: Check whether models have an appropriate number of upstream dependencies for sources
• has_valid_macro_dependencies: Check whether models have an appropriate number of upstream dependencies for macros
• has_no_final_semicolon: Check if models have a final semicolon present in their queries.
• has_no_hardcoded_refs: Check if models have any hardcoded references to database objects in their queries.
• has_constraints: Check whether models have an appropriate number of constraints configured in their properties.

You may also configure a generator to automatically and dynamically generate properties files for these models from database objects.

Model Columns

Filters

• name: Filter model columns based on their names.
• tag: Filter model columns based on their tags.
• meta: Filter model columns based on their meta values.

Terms

• has_description: Check whether the model columns have descriptions defined in their properties.
• has_required_tags: Check whether the model columns have the expected set of required tags set.
• has_allowed_tags: Check whether the model columns have only tags set from a configured permitted list.
• has_required_meta_keys: Check whether the model columns have the expected set of required meta keys set.
• has_allowed_meta_keys: Check whether the model columns have only meta keys set from a configured permitted list.
• has_allowed_meta_values: Check whether the model columns have only meta values set from a configured permitted mapping of keys to values.
• exists: Check whether the columns exist in the database.
• has_tests: Check whether columns have an appropriate number of tests configured.
• has_expected_name: Check whether columns have an expected name based on their data type.
• has_data_type: Check whether columns have a data type configured in their properties.
• has_matching_description: Check whether the descriptions configured in columns' properties matches the descriptions in the database.
• has_matching_data_type: Check whether the data type configured in a column's properties matches the data type in the database.
• has_matching_index: Check whether the index position within the properties of a column's table

You may also configure a generator to automatically and dynamically generate properties files for these columns from database objects.

Sources

Filters

• name: Filter sources based on their names.
• path: Filter sources based on their paths.
• tag: Filter sources based on their tags.
• meta: Filter sources based on their meta values.
• is_enabled: Filter sources taking only those which are enabled.

Terms

• has_properties: Check whether the sources have properties files defined.
• has_description: Check whether the sources have descriptions defined in their properties.
• has_required_tags: Check whether the sources have the expected set of required tags set.
• has_allowed_tags: Check whether the sources have only tags set from a configured permitted list.
• has_required_meta_keys: Check whether the sources have the expected set of required meta keys set.
• has_allowed_meta_keys: Check whether the sources have only meta keys set from a configured permitted list.
• has_allowed_meta_values: Check whether the sources have only meta values set from a configured permitted mapping of keys to values.
• exists: Check whether the sources exist in the database.
• has_tests: Check whether sources have an appropriate number of tests configured.
• has_all_columns: Check whether sources have all columns set in their properties.
• has_expected_columns: Check whether sources have the expected names of columns set in their properties.
• has_matching_description: Check whether the descriptions configured in sources' properties match the descriptions in the database.
• has_loader: Check whether sources have appropriate configuration for a loader in their properties.
• has_freshness: Check whether sources have freshness configured in their properties.
• has_downstream_dependencies: Check whether sources have an appropriate number of downstream dependencies.

You may also configure a generator to automatically and dynamically generate properties files for these sources from database objects.

Source Columns

Filters

• name: Filter source columns based on their names.
• tag: Filter source columns based on their tags.
• meta: Filter source columns based on their meta values.

Terms

• has_description: Check whether the source columns have descriptions defined in their properties.
• has_required_tags: Check whether the source columns have the expected set of required tags set.
• has_allowed_tags: Check whether the source columns have only tags set from a configured permitted list.
• has_required_meta_keys: Check whether the source columns have the expected set of required meta keys set.
• has_allowed_meta_keys: Check whether the source columns have only meta keys set from a configured permitted list.
• has_allowed_meta_values: Check whether the source columns have only meta values set from a configured permitted mapping of keys to values.
• exists: Check whether the columns exist in the database.
• has_tests: Check whether columns have an appropriate number of tests configured.
• has_expected_name: Check whether columns have an expected name based on their data type.
• has_data_type: Check whether columns have a data type configured in their properties.
• has_matching_description: Check whether the descriptions configured in columns' properties matches the descriptions in the database.
• has_matching_data_type: Check whether the data type configured in a column's properties matches the data type in the database.
• has_matching_index: Check whether the index position within the properties of a column's table

You may also configure a generator to automatically and dynamically generate properties files for these columns from database objects.

Macros

Filters

• name: Filter macros based on their names.
• path: Filter macros based on their paths.

Terms

• has_properties: Check whether the macros have properties files defined.
• has_description: Check whether the macros have descriptions defined in their properties.

Macro Arguments

Filters

• name: Filter macro arguments based on their names.

Terms

• has_description: Check whether the macro arguments have descriptions defined in their properties.
• has_type: Check whether macro arguments have a data type configured in their properties.

Release History

For change and release history, check out the documentation.

Contributing and Reporting Issues

If you have any suggestions, wish to contribute, or have any issues to report, please do let me know via the issues tab or make a new pull request with your new feature for review.

For more info on how to contribute to dbt-contracts, check out the documentation.

I hope you enjoy using dbt-contracts!