Skip to main content

PiperRider CLI

Project description

ci-tests release pipy python downloads license InfuseAI Discord Invite

Docs | Roadmap | Discord | Blog

Code review for data in dbt

PipeRider automatically compares your data to highlight the difference in impacted downstream dbt models so you can merge your Pull Requests with confidence.

How it works:

  • Easy to connect your datasource -> PipeRider leverages the connection profiles in your dbt project to connect to the data warehouse
  • Generate profiling statistics of your models to get a high-level overview of your data
  • Compare target branch changes with the main branch in a HTML report
  • Post a quick summary of the data changes to your PR, so others can be confident too

Core concepts

  • Easy to install: Leveraging dbt's configuration settings, PipeRider can be installed within 2 minutes
  • Fast comparison: by collecting profiling statistics (e.g. uniqueness, averages, quantiles, histogram) and metric queries, comparing downstream data impact takes little time, speeding up your team's review time
  • Valuable insights: various profiling statistics displayed in the HTML report give fast insights into your data

Quickstart

  1. Install PipeRider

    pip install piperider[<connector>]
    

    PipeRider supports the following data connectors

    connectors install
    snowflake pip install 'piperider[snowflake]'
    postgres pip install 'piperider[postgres]'
    bigquery pip install 'piperider[bigquery]'
    redshift pip install 'piperider[redshift]'
    athena pip install 'piperider[athena]'
    parquet pip install 'piperider[parquet]'
    csv pip install 'piperider[csv]'
    duckdb pip install 'piperider[duckdb]'
  2. Initialize PipeRider: Go to your dbt project, and initialize PipeRider.

    piperider init
    
  3. Apply PipeRider tag: Apply piperider tag to the models you want to profile

  4. Run PipeRider: Collect profiling statistics by using

    dbt build
    piperider run
    
  5. Compare your changes: You then can compare the branch of your new Pull Request against the main branch and explore the impact of your changes by opening the generated HTML comparison report

    git switch feature/pr-branch
    dbt build
    piperider run
    piperider compare-reports --last
    
  6. Post the markdown summary on the PR: You can post the markdown summary of the data changes to your Pull Request comment, so that your reviewer can merge with confidence.

Features

  • Use PipeRider for exploratory data analysis by doing piperider run to view the profiling statistics of a single data source, even in an environment that doesn't use dbt
  • Leverage dbt-defined metrics to have a quick overview of the impact on your most important metrics
  • Include PipeRider into your CI process via PipeRider Cloud or self-hosted to be confident of every PRs that is submitted
  • Benefit from dbt's features such as Slim CI, custom schema, custom database, node selection, dbt test result

Example Report Demo

We use the example project git-repo-analytics to demonstrate how to use piperider+dbt+duckdb to analyze dbt-core repository. Here is the generated result (daily update)

Run Report

Comparison Report

Comparison Summary in a PR

PipeRider Cloud (beta)

PipeRider Cloud offers a hosted version for HTML reports, including features such as alerts and historical trend watching. Get early beta access by signing up on our website: https://piperider.io

Development

See setup dev environment and the contributing guildlines to get started.

We love chatting with our users! Let us know if you have any questions, feedback, or need help trying out PipeRider! :heart:

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

piperider-nightly-0.24.0.20230425.tar.gz (3.9 MB view details)

Uploaded Source

Built Distribution

File details

Details for the file piperider-nightly-0.24.0.20230425.tar.gz.

File metadata

File hashes

Hashes for piperider-nightly-0.24.0.20230425.tar.gz
Algorithm Hash digest
SHA256 cf020e3ae832161898435389d0de77e5b9cd61b55ca3bed50b6b7f972be45f07
MD5 7c8e2b600fbb59c2d54ebf6c21241992
BLAKE2b-256 395367211ce32d998e0a778806191411963c1f3c422e4daf3b2121ef039f1de3

See more details on using hashes here.

File details

Details for the file piperider_nightly-0.24.0.20230425-py3-none-any.whl.

File metadata

File hashes

Hashes for piperider_nightly-0.24.0.20230425-py3-none-any.whl
Algorithm Hash digest
SHA256 deb5eaba132826a31ed3fb4beda89671796d64fbfbd3a37be8bad4028f3d7f02
MD5 c23845c85a0b4a0ec319f8a7413788ac
BLAKE2b-256 9ac74cc02afdea16db71afb9fedfb622ba5edae1b8b77ccc1ef554ea00dd69c9

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page