Skip to main content

ci-tests release pipy python downloads license InfuseAI Discord Invite

Docs | Roadmap | Discord | Blog

Code review for data in dbt

PipeRider automatically compares your data to highlight the difference in impacted downstream dbt models so you can merge your Pull Requests with confidence.

How it works:

  • Easy to connect your datasource -> PipeRider leverages the connection profiles in your dbt project to connect to the data warehouse
  • Generate profiling statistics of your models to get a high-level overview of your data
  • Compare target branch changes with the main branch in a HTML report
  • Post a quick summary of the data changes to your PR, so others can be confident too

Core concepts

  • Easy to install: Leveraging dbt's configuration settings, PipeRider can be installed within 2 minutes
  • Fast comparison: by collecting profiling statistics (e.g. uniqueness, averages, quantiles, histogram) and metric queries, comparing downstream data impact takes little time, speeding up your team's review time
  • Valuable insights: various profiling statistics displayed in the HTML report give fast insights into your data

Quickstart

  1. Install PipeRider

    pip install piperider[<connector>]
    

    PipeRider supports the following data connectors

    connectors install
    snowflake pip install 'piperider[snowflake]'
    postgres pip install 'piperider[postgres]'
    bigquery pip install 'piperider[bigquery]'
    redshift pip install 'piperider[redshift]'
    parquet pip install 'piperider[parquet]'
    csv pip install 'piperider[csv]'
    duckdb pip install 'piperider[duckdb]'
  2. Initialize PipeRider: Go to your dbt project, and initialize PipeRider.

    piperider init
    
  3. Run PipeRider: Collect profiling statistics by using

    dbt build
    piperider run
    
  4. Compare your changes: You then can compare the branch of your new Pull Request against the main branch and explore the impact of your changes by opening the generated HTML comparison report

    git switch feature/pr-branch
    dbt build
    piperider run
    piperider compare-reports --last
    
  5. Post the markdown summary on the PR: You can post the markdown summary of the data changes to your Pull Request comment, so that your reviewer can merge with confidence.

Features

  • Use PipeRider for exploratory data analysis by doing piperider run to view the profiling statistics of a single data source, even in an environment that doesn't use dbt
  • Leverage dbt-defined metrics to have a quick overview of the impact on your most important metrics
  • Include PipeRider into your CI process via PipeRider Cloud or self-hosted to be confident of every PRs that is submitted
  • Benefit from dbt's features such as Slim CI, custom schema, custom database, node selection, dbt test result

Example Report Demo

See Generated Single-Run Report

See Comparison Report

PipeRider Cloud (beta)

PipeRider Cloud offers a hosted version for HTML reports, including features such as alerts and historical trend watching. Get early beta access by signing up on our website: https://piperider.io

Development

See setup dev environment and the contributing guildlines to get started.

We love chatting with our users! Let us know if you have any questions, feedback, or need help trying out PipeRider! :heart:

Metadata

Release files for piperider-nightly 0.19.0.20230215

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for piperider-nightly 0.19.0.20230215
File Size Uploaded
piperider-nightly-0.19.0.20230215.tar.gz 3.6 MB Details

Built distribution (wheel)

Table of built distributions (wheels) for piperider-nightly 0.19.0.20230215
File Interpreter ABI Platform
piperider_nightly-0.19.0.20230215-py3-none-any.whl Python 3 none any Details

Total release size: 7.3 MB

Release files / piperider-nightly-0.19.0.20230215.tar.gz

Download URL piperider-nightly-0.19.0.20230215.tar.gz
Size 3.6 MB
Tags Source
SHA-256 checksum
How to use checksums
ea2fe9e00e05c25d98467f51e2342da62d67630053aa390cb18890757312dd12
BLAKE2b-256 checksum
How to use checksums
bb4b41df99d0d789a43a31d04359e72661259b2c19ae3c4b4d01a817f5794c3d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.8.16

Release files / piperider_nightly-0.19.0.20230215-py3-none-any.whl

Download URL piperider_nightly-0.19.0.20230215-py3-none-any.whl
Size 3.7 MB
Tags Python 3
SHA-256 checksum
How to use checksums
321bc5d2d6f5284c5f3e39ae37efd4dd0c12adadcc87cf8b56eb6976eb375791
BLAKE2b-256 checksum
How to use checksums
1fc891b1d9895e4fb689ad944f195f8179722b90d7b573903fcc0a314c9a4d27
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.8.16

Release history Release notifications | RSS feed

This release

0.19.0.20230215 This release

2 release files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page