Skip to main content

PiperRider CLI

Project description

ci-tests codecov release pipy python downloads license InfuseAI Discord Invite

Docs | Discord | Blog

Code review for data in dbt

PipeRider automatically compares your data to highlight the difference in impacted downstream dbt models so you can merge your Pull Requests with confidence.

How it works:

  • Easy to connect your datasource -> PipeRider leverages the connection profiles in your dbt project to connect to the data warehouse
  • Generate profiling statistics of your models to get a high-level overview of your data
  • Compare target branch changes with the main branch in a HTML report
  • Post a quick summary of the data changes to your PR, so others can be confident too

Core concepts

  • Easy to install: Leveraging dbt's configuration settings, PipeRider can be installed within 2 minutes
  • Fast comparison: by collecting profiling statistics (e.g. uniqueness, averages, quantiles, histogram) and metric queries, comparing downstream data impact takes little time, speeding up your team's review time
  • Valuable insights: various profiling statistics displayed in the HTML report give fast insights into your data

Quickstart

  1. Install PipeRider

    pip install piperider[<connector>]
    

    You can find all supported data source connectors here.

  2. Add PipeRider tag on your model: Go to your dbt project, and add the PipeRider tag on the model you want to profile.

    --models/staging/stg_customers.sql
    {{ config(
       tags=["piperider"]
    ) }}
    
    select ...
    

    and show the models would be run by piperider

     dbt list -s tag:piperider --resource-type model
    
  3. Run PipeRider

    piperider run
    

To see the full quick start guide, please refer to PipeRider documentation

Features

  • Model profiling: PipeRider can profile your dbt models and obtain information such as basic data composition, quantiles, histograms, text length, top categories, and more.
  • Metric queries: PipeRider can integrate with dbt metrics and present the time-series data of metrics in the report.
  • HTML report: PipeRider generates a static HTML report each time it runs, which can be viewed locally or shared.
  • Report comparison: You can compare two previously generated reports or use a single command to compare the differences between the current branch and the main branch. The latter is designed specifically for code review scenarios. In our pull requests on GitHub, we not only want to know which files have been changed, but also the impact of these changes on the data. PipeRider can easily generate comparison reports with a single command to provide this information.
  • CI integration: The key to CI is automation, and in the code review process, automating this workflow is even more meaningful. PipeRider can easily integrate into your CI process. When new commits are pushed to your PR branch, reports can be automatically generated to provide reviewers with more confidence in the changes made when reviewing.

Example Report Demo

We use the example project git-repo-analytics to demonstrate how to use piperider+dbt+duckdb to analyze dbt-core repository. Here is the generated result (daily update)

Run Report

Comparison Report

Comparison Summary in a PR

PipeRider Cloud (beta)

PipeRider Cloud allows you to upload reports and share them with your team members. For information on pricing plans, please refer to the pricing page.

PipeRider Compare Action

PipeRider provides the PipeRider Compare Action to quickly integrate into your Github Actions workflow. It has the following features:

  • Automatically generates a report comparing the PR branch to the main branch
  • Uploads the report to GitHub artifacts or PipeRider cloud
  • Adds a comment to the pull request with a comparison summary and a link to the report.

You can refer to example workflow yaml and the example pull request.

Development

See setup dev environment and the contributing guildlines to get started.

We love chatting with our users! Let us know if you have any questions, feedback, or need help trying out PipeRider! :heart:

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

piperider-nightly-0.42.0.20240201.tar.gz (4.1 MB view details)

Uploaded Source

Built Distribution

File details

Details for the file piperider-nightly-0.42.0.20240201.tar.gz.

File metadata

File hashes

Hashes for piperider-nightly-0.42.0.20240201.tar.gz
Algorithm Hash digest
SHA256 31ba6cdeaa730ef6aee242e7b487d8766f8e06796bbca6148adba59c97c83c5c
MD5 2cfb48590260dc78366e4d20139e4fea
BLAKE2b-256 1552d70c648b281967f0323c7febae3d7b38e770a6725bfc6631c643dd474195

See more details on using hashes here.

File details

Details for the file piperider_nightly-0.42.0.20240201-py3-none-any.whl.

File metadata

File hashes

Hashes for piperider_nightly-0.42.0.20240201-py3-none-any.whl
Algorithm Hash digest
SHA256 2dcf88bb2c2c1fb340fbdfd2bb27950286020f3bad4c68a59a08a9226d276f5e
MD5 002ba21f2696356e03b51833d96d6bd0
BLAKE2b-256 830ff53ed805639c53cda5e4544eddd67cda07340ed4c9e3b9f18b7e0211cecb

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page