Skip to main content

Open Source Data Quality Monitoring

Project description

Logo

Open Source Data Quality Monitoring.

License Versions coverage coverage Status

⭐️ If you like it, star the repo

| Documentations | Slack Community |

Why Data Monitoring?

APM (Application Performance Monitoring) tools are used to monitor the performance of applications. APM tools are mandatory part of dev stack. Without AMP tools, it is very difficult to monitor the performance of applications.

why_data_observability

But for Data products regular APM tools are not enough. We need a new kind of tools that can monitor the performance of Data applications. Data monitoring tools are used to monitor the data quality of databases and data pipelines. It identifies potential issues, including in the databases and data pipelines. It helps to identify the root cause of the data quality issues and helps to improve the data quality.

What is datachecks?

Datachecks is an open-source data monitoring tool that helps to monitor the data quality of databases and data pipelines. It identifies potential issues, including in the databases and data pipelines. It helps to identify the root cause of the data quality issues and helps to improve the data quality.

Datachecks can generate several reliability, uniqueness, completeness metrics from several data sources

Reports: Data Quality Visualisation

You can generate with just one command. It generates a beautiful data quality report with all the metrics. This html report can be shared with the team.

why_data_observability

CLI: Data Quality Visualisation in Bash

Data quality report can be generated in the terminal. It is very useful for debugging. All it takes is one command.

why_data_observability

Getting Started

Install datachecks with the command that is specific to the database.

Install Datachecks

To install all datachecks dependencies, use the below command.

pip install datachecks -U

Create the config file

With a simple config file, you can generate data quality reports for your data sources. Below is the sample config example. For more details, please visit the config guide

why_data_observability

Run from CLI

Generate Report in Terminal

datachecks inspect -C config.yaml

Generate HTML Report

datachecks inspect -C config.yaml  --html-report

Please visit the Quick Start Guide

Supported Data Sources

Datachecks supports sql and search data sources. Below are the list of supported data sources.

Data Source Type Supported
Postgres Transactional Database :thumbsup:
MySql Transactional Database :thumbsup:
MS SQL Server Transactional Database :thumbsup:
OpenSearch Search Engine :thumbsup:
Elasticsearch Search Engine :thumbsup:
GCP BigQuery Data Warehouse :thumbsup:
DataBricks Data Warehouse :thumbsup:
Snowflake Data Warehouse :thumbsup:
AWS RedShift Data Warehouse :x:

Metric Types

Metric Description
Reliability Metrics Reliability metrics detect whether tables/indices/collections are updating with timely data
Numeric Distribution Metrics Numeric Distribution metrics detect changes in the numeric distributions i.e. of values, variance, skew and more
Uniqueness Metrics Uniqueness metrics detect when data constraints are breached like duplicates, number of distinct values etc
Completeness Metrics Completeness metrics detect when there are missing values in datasets i.e. Null, empty value
Validity Metrics Validity metrics detect whether data is formatted correctly and represents a valid value

Overview

datacheck_architecture

What Datacheck does not do?

Community & Support

For additional information and help, you can use one of these channels:

  • Slack (Live chat with the team, support, discussions, etc.)
  • GitHub issues (Bug reports, feature requests)

Contributions

:raised_hands: We greatly appreciate contributions - be it a bug fix, new feature, or documentation!

Check out the contributions guide and open issues.

Datachecks contributors: :blue_heart:

Telemetry

Usage Analytics & Data Privacy

License

This project is licensed under the terms of the APACHE 2 License.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

datachecks-0.3.1.tar.gz (1.1 MB view details)

Uploaded Source

Built Distribution

datachecks-0.3.1-py3-none-any.whl (1.2 MB view details)

Uploaded Python 3

File details

Details for the file datachecks-0.3.1.tar.gz.

File metadata

  • Download URL: datachecks-0.3.1.tar.gz
  • Upload date:
  • Size: 1.1 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.4.1 CPython/3.9.11 Darwin/23.3.0

File hashes

Hashes for datachecks-0.3.1.tar.gz
Algorithm Hash digest
SHA256 914a15295d789cc08d26aa0851f8dcc261b27adceb462d2da0734383570468bf
MD5 e28358ec329fa15f4aec093906ff9d8a
BLAKE2b-256 2fc9cae49d6e074e9b9c95295562b9bde78565b44a74695a3dd08e2d0ebf90c6

See more details on using hashes here.

File details

Details for the file datachecks-0.3.1-py3-none-any.whl.

File metadata

  • Download URL: datachecks-0.3.1-py3-none-any.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.4.1 CPython/3.9.11 Darwin/23.3.0

File hashes

Hashes for datachecks-0.3.1-py3-none-any.whl
Algorithm Hash digest
SHA256 a03177a9305ce738efd14cc6c9406db8f0875fc0b44d7a6049e30d0ffde3000e
MD5 5aa8866e9319384b73c28d6e0aab49bd
BLAKE2b-256 abb05eb8624d0298df8d1bd6bfee21b28b492a234fe24da50a91909a83d5dea0

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page