diqu 0.2.0

Data Quality CLI for the Auto-Alerts

diqu

Automate and streamline the alerting/ notification process for dbt test results using this versatile CLI companion tool. Receive detailed alerts & test metadata seamlessly on various platforms, promoting improved collaboration on dbt project issues 🐞🚀.

Who is this for

This tool is designed for individuals or teams seeking to automate the management of their dbt project issues (test warnings, errors... etc) outside the dbt environment.

Features

• Automated alerts and notifications based on recorded dbt test results.
• Built-in support for dq-tools and custom query input.
• Auto-labels deprecated tests for quick & easy identification.
• Creates and updates Jira tickets with the latest tests' metadata.
• Sends succinct and informative messages to a dedicated Slack channel.

dbt alert rant 🟢 🟡 🔴 ⚫

"Run errors are red,
Test warnings are yellow,
Where's my bug alert,
And joint bugfix workflow?"

-- by diqu OG Contributors

Supported Modules

• Sources (DWH connections)
  • Snowflake
  • CSV file
• Package (parsing dbt test results)
  • 
  • Custom query
• Alert Modules (alert/ notification)
  • Jira
  • Slack

Installation

pip install diqu [--upgrade]

📓 NOTE: The required Data Warehouse (DWH) module should already be installed if you are using diqu in a working dbt project. If not, please perform additional steps to install these DWH modules. For example, if you're using Snowflake:

pip install "snowflake-connector-python[pandas]"
pip install "snowflake-connector-python[secure-local-storage]"

Usage

preflight

# define the query params
export ISSUE_DEPRECATED_WINDOW_IN_DAYS=your_issue_deprecation_time_in_day, default to "3"
export ISSUE_UPDATE_WINDOW_IN_DAYS=your_issue_historical_data_update_window_in_days, default to "14"

# build dq-tools log table
dbt run -s dq_tools

diqu alert --to slack --to jira

04:33:17  diqu: INFO - Run with diqu==1.0.0 🏃
04:33:19  diqu: INFO - Using dbt project at: /path/to/dbt/project
04:33:19  diqu: INFO - Using dbt profiles.yml at: ~/.dbt
04:33:19  diqu: INFO - Using snowflake connection
04:33:19  diqu: INFO - Looking for the query in: ./dq_tools__get_test_results.sql
04:33:23  diqu: INFO - Alerting to: SLACK
04:33:23  diqu: INFO - ✅ Done > Slack
04:33:23  diqu: INFO - Alerting to: JIRA
04:33:23  diqu: INFO - ✅ Done > JIRA

Alert Modules Configurations

Slack

• Use the environment variables to configure the Slack Channel:

  export SLACK_TOKEN=your_token
  export SLACK_CHANNEL=your_channel_name
  
  diqu alert --to slack
  

Jira Board

• Use the environment variables to configure the JIRA Board:

  export JIRA_SERVER=your_jira_server e.g. https://your_value.atlassian.net/
  export JIRA_AUTH_USER=your_service_account e.g. dqt_user@your_value.com
  export JIRA_AUTH_PASSWORD=your_service_token e.g. ATATTxxxxx
  export JIRA_PROJECT_ID=your_project_id e.g. 106413
  export JIRA_ISSUE_TYPE=your_issue_type, default to "Bug"
  export JIRA_OPEN_ISSUES_FILTER_BY_SUMMARY=your_issue_filter_on_title, default to "dq-tools"
  
  diqu alert --to jira
  

📖 For more details, please visit the documentation site.

Quick Demo

How to Contribute

This Auto Alert (diqu) tool is an open-source software. Whether you are a seasoned open-source contributor or a first-time committer, we welcome and encourage you to contribute code, documentation, ideas, or problem statements to this project.

👉 See CONTRIBUTING guideline for more details or check out CONTRIBUTING.md

🌟 And then, kudos to our beloved Contributors:

About Infinite Lambda

Infinite Lambda is a cloud and data consultancy. We build strategies, help organizations implement them, and pass on the expertise to look after the infrastructure.

We are an Elite Snowflake Partner, a Platinum dbt Partner, and two-times Fivetran Innovation Partner of the Year for EMEA.

Naturally, we love exploring innovative solutions and sharing knowledge, so go ahead and:

🔧 Take a look around our Git

✏️ Browse our tech blog

We are also chatty, so:

👀 Follow us on LinkedIn

👋🏼 Or just get in touch