Data reliability tool for profiling and testing your data
Docs | Roadmap | Discord | Blog
Data Reliability = Profiling + Testing
Piperider is a CLI tool that allows you to build data profiles and write assertion tests for easily evaluating and tracking your data's reliability over time.
Core Concepts
- Profile Your Data to explore/understand what kind of dataset you're dealing with e.g. completeness, duplicates, missing values, distributions
- Test Your Data to verify that your data is within acceptable range and formatted correctly
- Observe & Monitor Your Data to keep an eye on how that data changes over time
Key Features
- SQL-based (additionally supports CSV)
- Data Profiling Characteristics
- Provides rich data profiling metrics
- e.g.
missing,uniqueness,duplicate_rows,quantiles,histogram
- Test datasets with a mix of custom and built-in assertion definitions
- Auto-generates recommended assertions based on your single-run profiles
- Generates single-run reports to visualize your data profile and assertion test results (example)
- Generates comparison reports to visualize how your data has changed over time (example)
- Supported Datasources: Snowflake, BigQuery, Redshift, Postgres, SQLite, DuckDB, CSV, Parquet.
Quickstart
Installation
pip install piperider
By default, PipeRider supports built-in SQLite connector, extra connectors are available:
| connectors | install |
|---|---|
| snowflake | pip install 'piperider[snowflake]' |
| postgres | pip install 'piperider[postgres]' |
| bigquery | pip install 'piperider[bigquery]' |
| redshift | pip install 'piperider[redshift]' |
| parquet | pip install 'piperider[parquet]' |
| csv | pip install 'piperider[csv]' |
| duckdb | pip install 'piperider[duckdb]' |
Use comma to install multiple connectors in one line:
pip install 'piperider[postgres,snowflake]'
Initialize Project & Diagnose Settings
Once installed, initialize a new project with the following command.
piperider init # initializes project config
piperider diagnose # verifies your data source connection & project config
Profiling and Testing Your Data
Next, execute piperider run, which will do a number of things:
- Create a single-run profile of your data source
- Auto-generate recommended or template assertions files (first-run only)
- Test that single-run profile against any available assertions, including custom and/or recommended assertions
- Generate a static HTML report, which helps visualize the single-run profile and its assertion results.
Common Usages/Tips:
piperider run # profile all tables in the data source.
piperider run --table $TABLENAME # profile a specific table
piperider generate-report -o $PATHNAME # Specify the output location of the generated report
piperider generate-assertions # To re-generate the recommended assertions after the first-run
Comparing Your Data Profiles
With at least two runs completed, you can then run piperider compare-reports, which will generate a comparison report that presents the changes between them (e.g. schema changes, column renaming, distributions).
Common Usages/Tips:
piperider compare-reports --last # Compare the last two reports automatically using
For more details on the generated report, see the doc
Example Report Demo
See Generated Single-Run Report
Development
See setup dev environment and the contributing guildlines to get started.
We're in an early stage, so let us know if you have any questions, feedback, or need help trying out PipeRider! :heart:
1 maintainer
Metadata
1 maintainer
Release files for piperider-nightly 0.15.0.20221205
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| piperider-nightly-0.15.0.20221205.tar.gz | 3.6 MB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| piperider_nightly-0.15.0.20221205-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 7.3 MB
Release files / piperider-nightly-0.15.0.20221205.tar.gz
| Download URL | piperider-nightly-0.15.0.20221205.tar.gz |
|---|---|
| Size | 3.6 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
f260c2e5b88c9b0fbd09346ab2e2743bbcfa8082ae86ef6ff22e9694641f3df2
|
|
BLAKE2b-256 checksum How to use checksums |
3e230c3e537746b658c808fa75cc05504bb6f291bd41ebf048bf7eecda4fe84d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.8.14
|
Release files / piperider_nightly-0.15.0.20221205-py3-none-any.whl
| Download URL | piperider_nightly-0.15.0.20221205-py3-none-any.whl |
|---|---|
| Size | 3.7 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
c2ff18a04e2bc9f46d6e329564203abf563d0172f16735f87c89a8d944875b39
|
|
BLAKE2b-256 checksum How to use checksums |
16eadd487d90e3381ac2feeb8376d26f5892223ccf539275fdff2b56441c47d9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.8.14
|
