Code review for data in dbt

PipeRider automatically compares your data to highlight the difference in impacted downstream dbt models so you can merge your Pull Requests with confidence.

How it works:

Easy to connect your datasource -> PipeRider leverages the connection profiles in your dbt project to connect to the data warehouse
Generate profiling statistics of your models to get a high-level overview of your data
Compare target branch changes with the main branch in a HTML report
Post a quick summary of the data changes to your PR, so others can be confident too

Core concepts

Easy to install: Leveraging dbt's configuration settings, PipeRider can be installed within 2 minutes
Fast comparison: by collecting profiling statistics (e.g. uniqueness, averages, quantiles, histogram) and metric queries, comparing downstream data impact takes little time, speeding up your team's review time
Valuable insights: various profiling statistics displayed in the HTML report give fast insights into your data

Quickstart

Install PipeRider

pip install piperider[<connector>]

PipeRider supports the following data connectors

connectors	install
snowflake	`pip install 'piperider[snowflake]'`
postgres	`pip install 'piperider[postgres]'`
bigquery	`pip install 'piperider[bigquery]'`
redshift	`pip install 'piperider[redshift]'`
parquet	`pip install 'piperider[parquet]'`
csv	`pip install 'piperider[csv]'`
duckdb	`pip install 'piperider[duckdb]'`

Initialize PipeRider: Go to your dbt project, and initialize PipeRider.
```
piperider init
```
Run PipeRider: Collect profiling statistics by using
```
dbt build
piperider run
```
Compare your changes: You then can compare the branch of your new Pull Request against the main branch and explore the impact of your changes by opening the generated HTML comparison report
```
git switch feature/pr-branch
dbt build
piperider run
piperider compare-reports --last
```
Post the markdown summary on the PR: You can post the markdown summary of the data changes to your Pull Request comment, so that your reviewer can merge with confidence.

Features

Use PipeRider for exploratory data analysis by doing piperider run to view the profiling statistics of a single data source, even in an environment that doesn't use dbt
Leverage dbt-defined metrics to have a quick overview of the impact on your most important metrics
Include PipeRider into your CI process via PipeRider Cloud or self-hosted to be confident of every PRs that is submitted
Benefit from dbt's features such as Slim CI, custom schema, custom database, node selection, dbt test result

Example Report Demo

We use the example project git-repo-analytics to demonstrate how to use piperider+dbt+duckdb to analyze dbt-core repository. Here is the generated result (daily update)

Run Report

Comparison Report

Comparison Summary in a PR

PipeRider Cloud (beta)

PipeRider Cloud offers a hosted version for HTML reports, including features such as alerts and historical trend watching. Get early beta access by signing up on our website: https://piperider.io

Development

See setup dev environment and the contributing guildlines to get started.

We love chatting with our users! Let us know if you have any questions, feedback, or need help trying out PipeRider! ❤️

Metadata

Release files for piperider 0.22.0a30601

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for piperider 0.22.0a30601
File	Size	Uploaded
piperider-0.22.0a30601.tar.gz	3.8 MB	Mar 23, 2023	Details

Built distribution (wheel)

Table of built distributions (wheels) for piperider 0.22.0a30601
File	Interpreter	ABI	Platform	Reset
piperider-0.22.0a30601-py3-none-any.whl 3.9 MB Mar 23, 2023	Python 3	none	any	Details

Total release size: 7.8 MB

Release files / piperider-0.22.0a30601.tar.gz

Download URL	piperider-0.22.0a30601.tar.gz
Size	3.8 MB
Tags	Source
SHA-256 checksum How to use checksums	`b1b98693e160798d85a3066b36d980147fbae37cf6729e298e0bddc483f7179b`
BLAKE2b-256 checksum How to use checksums	`2245e56a17565049a61fd87117f55afd2448ad1a1b3b99bf63a3319609de58db`
Upload date	Mar 23, 2023
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.2 CPython/3.8.16`

Release files / piperider-0.22.0a30601-py3-none-any.whl

Download URL	piperider-0.22.0a30601-py3-none-any.whl
Size	3.9 MB
Tags	Python 3
SHA-256 checksum How to use checksums	`8f3406e0d6a471beae86085e7319966f06ffb28283e02b56680ea0a5fea6c485`
BLAKE2b-256 checksum How to use checksums	`fb10deec7de1f2f85e7888e649f5f9c23f4feba69e65561cb7ac754905952609`
Upload date	Mar 23, 2023
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.2 CPython/3.8.16`