piperider-nightly 0.10.0.20220925

PiperRider CLI

PipeRider: Data Reliability Toolkit

What's PipeRider?

PipeRider is a light-weight data reliability toolkit using warehouse-native profiling so you can have a full understanding of your data

We're in an early stage, so let us know if you have any questions, feedback, or need help installing PipeRider! :heart:

Profiling as a data reliability strategy

PipeRider will make your life easier by:

1. Building a data profile so you can easily understand your data
2. Creating test suggestions based on the profiling
3. Comparing data profile reports, so you track changes over time

Read how to implement a data quality strategy using profiling + testing

Made for the modern data team

For data engineers

• :zap: 2 min install & set-up
• :relieved: Non-intrusive & open-source: install and use locally
• :money_with_wings: Fast & cheap: 100M rows & 8 columns (or 50M & 16 columns) takes only 18s to profile
• :ledger: Cloud DataWarehouse native & auto-config for dbt

For data analysts

• :bowtie: Never waste time on analyzing wrong data: collects various metadata metrics such as freshness, uniqueness, distribution... check all metrics
• :speech_balloon: Communicate easily your data expectations by showing the report
• Zero-config dbt integration

Live Demo

Click here or on image to interact

Key features

• Generate an HTML Report featuring your data profile and data assertion test results (interactive sample)
• Compare two reports to understand how your data has changed over time (interactive sample)
• Test your data with data assertions:
  • Built-in data assertions
  • Extensible through custom assertions
  • Auto-generated data assertions
• Support various data sources like Snowflake, BigQuery, Redshift, Postgres, SQLite, DuckDB, CSV, Parquet and more.
• Zero-config support for dbt projects
• Automation through GitHub Actions, save reports in S3

Getting started

Get started quickly below, go to the docs, or check out this article on how to add data observability using PipeRider

Install PipeRider

pip install piperider

By default, PipeRider supports built-in SQLite connector, extra connectors are available:

connectors	install	supported since
snowflake	pip install 'piperider[snowflake]'
postgres	pip install 'piperider[postgres]'
bigquery	pip install 'piperider[bigquery]'	PipeRider v0.7.0
redshift	pip install 'piperider[redshift]'	PipeRider v0.7.0
parquet	pip install 'piperider[parquet]'	PipeRider v0.8.0
csv	pip install 'piperider[csv]'	PipeRider v0.8.0
duckdb	pip install 'piperider[duckdb]'	PipeRider v0.8.0

Use comma to install multiple connectors in one line:

pip install 'piperider[postgres,snowflake]'

You can follow the quick start guide to learn more about PipeRider.

Attach PipeRider to a dbt project

Initialize PipeRider inside a dbt project and your data source settings will be automatically configured

piperider init creates /.piperider under a dbt project root and generates necessary configurations.

Scan data quality from models

piperider run scans the models from data sources and creates assessment results in /.piperider/output

Generate reports

piperider generate-report generate a static HTML report.

Generate comparison view

You can use piperider compare-report to compare 2 reports.

Learn More

PipeRider Resources	Description
Documentation	PipeRider Main Doc Site
Sample_Project	Sample Project with with sqlite
dbt_Sample_Project	Sample Project with dbt
Roadmap	PipeRider Roadmap
Blog	How we got started

Get involved

Support

If you like what we are building, support us! Give us a :star: or get in touch. We'd love your feedback! Send us a message on piperider.io, join our Discord, or report an issue on GitHub

Contributions

We welcome contributions. See the Set up dev environment and the Contributing guildline to get started.