piperider-nightly 0.11.0.20221017

PiperRider CLI

Data reliability tool for profiling and testing your data

Docs | Roadmap | Discord | Blog

Data Reliability = Profiling + Testing

Piperider is a CLI tool that allows you to build data profiles and write assertion tests for easily evaluating and tracking your data's reliability over time.

Core Concepts

1. Profile Your Data to explore/understand what kind of dataset you're dealing with e.g. completeness, duplicates, missing values, distributions
2. Test Your Data to verify that your data is within acceptable range and formatted correctly
3. Observe & Monitor Your Data to keep an eye on how that data changes over time

Key Features

• SQL-based (additionally supports CSV)
• Data Profiling Characteristics
  • Provides rich data profiling metrics
  • e.g. missing, uniqueness, duplicate_rows, quantiles, histogram
• Test datasets with a mix of custom and built-in assertion definitions
• Auto-generates recommended assertions based on your single-run profiles
• Generates single-run reports to visualize your data profile and assertion test results (example)
• Generates comparison reports to visualize how your data has changed over time (example)
• Supported Datasources: Snowflake, BigQuery, Redshift, Postgres, SQLite, DuckDB, CSV, Parquet.

Quickstart

Installation

pip install piperider

By default, PipeRider supports built-in SQLite connector, extra connectors are available:

connectors	install
snowflake	pip install 'piperider[snowflake]'
postgres	pip install 'piperider[postgres]'
bigquery	pip install 'piperider[bigquery]'
redshift	pip install 'piperider[redshift]'
parquet	pip install 'piperider[parquet]'
csv	pip install 'piperider[csv]'
duckdb	pip install 'piperider[duckdb]'

Use comma to install multiple connectors in one line:

pip install 'piperider[postgres,snowflake]'

Initialize Project & Diagnose Settings

Once installed, initialize a new project with the following command.

piperider init        # initializes project config
piperider diagnose    # verifies your data source connection & project config

Profiling and Testing Your Data

Next, execute piperider run, which will do a number of things:

1. Create a single-run profile of your data source
2. Auto-generate recommended or template assertions files (first-run only)
3. Test that single-run profile against any available assertions, including custom and/or recommended assertions
4. Generate a static HTML report, which helps visualize the single-run profile and its assertion results.

Common Usages/Tips:

piperider run                           # profile all tables in the data source.

piperider run --table $TABLENAME        # profile a specific table

piperider generate-report -o $PATHNAME  # Specify the output location of the generated report

piperider generate-assertions           # To re-generate the recommended assertions after the first-run

Comparing Your Data Profiles

With at least two runs completed, you can then run piperider compare-reports, which will generate a comparison report that presents the changes between them (e.g. schema changes, column renaming, distributions).

Common Usages/Tips:

piperider compare-reports --last        # Compare the last two reports automatically using

For more details on the generated report, see the doc

Example Report Demo

See Generated Single-Run Report

See Comparison Report

Development

See setup dev environment and the contributing guildlines to get started.

We're in an early stage, so let us know if you have any questions, feedback, or need help trying out PipeRider! :heart: