PipeRider: Data Reliability Toolkit
What's PipeRider?
PipeRider is a light-weight data reliability toolkit using warehouse-native profiling so you can have a full understanding of your data
We're in an early stage, so let us know if you have any questions, feedback, or need help installing PipeRider! :heart:
Profiling as a data reliability strategy
PipeRider will make your life easier by:
- Building a data profile so you can easily understand your data
- Creating test suggestions based on the profiling
- Comparing data profile reports, so you track changes over time
Read how to implement a data quality strategy using profiling + testing
Made for the modern data team
For data engineers
- :zap: 2 min install & set-up
- :relieved: Non-intrusive & open-source: install and use locally
- :money_with_wings: Fast & cheap: 100M rows & 8 columns (or 50M & 16 columns) takes only 18s to profile
- :ledger: Cloud DataWarehouse native & auto-config for dbt
For data analysts
- :bowtie: Never waste time on analyzing wrong data: collects various metadata metrics such as freshness, uniqueness, distribution... check all metrics
- :speech_balloon: Communicate easily your data expectations by showing the report
- Zero-config dbt integration
Live Demo
Click here or on image to interact
Key features
- Generate an HTML Report featuring your data profile and data assertion test results (interactive sample)
- Compare two reports to understand how your data has changed over time (interactive sample)
- Test your data with data assertions:
- Built-in data assertions
- Extensible through custom assertions
- Auto-generated data assertions
- Support various data sources like Snowflake, BigQuery, Redshift, Postgres, SQLite, DuckDB, CSV, Parquet and more.
- Zero-config support for dbt projects
- Automation through GitHub Actions, save reports in S3
Getting started
Get started quickly below, go to the docs, or check out this article on how to add data observability using PipeRider
Install PipeRider
pip install piperider
By default, PipeRider supports built-in SQLite connector, extra connectors are available:
| connectors | install | supported since |
|---|---|---|
| snowflake | pip install 'piperider[snowflake]' | |
| postgres | pip install 'piperider[postgres]' | |
| bigquery | pip install 'piperider[bigquery]' | PipeRider v0.7.0 |
| redshift | pip install 'piperider[redshift]' | PipeRider v0.7.0 |
| parquet | pip install 'piperider[parquet]' | PipeRider v0.8.0 |
| csv | pip install 'piperider[csv]' | PipeRider v0.8.0 |
| duckdb | pip install 'piperider[duckdb]' | PipeRider v0.8.0 |
Use comma to install multiple connectors in one line:
pip install 'piperider[postgres,snowflake]'
You can follow the quick start guide to learn more about PipeRider.
Attach PipeRider to a dbt project
piperider init creates /.piperider under a dbt project root and generates necessary configurations.
Scan data quality from models
piperider run runs profiling, and tests against assertions, and will render an HTML report /.piperider/outputs
You can do a run for a specific table using piperider run --table $TABLENAME
You can specifying the output location of a report using piperider generate-report -o $PATHNAME or specifying it in the config file.
All details on generating reports
Compare reports
You can use piperider compare-reports to compare 2 reports.
Learn More
| PipeRider Resources | Description |
|---|---|
| Documentation | PipeRider Main Doc Site |
| Sample_Project | Sample Project with with sqlite |
| dbt_Sample_Project | Sample Project with dbt |
| Roadmap | PipeRider Roadmap |
| Blog | How we got started |
Get involved
Support
If you like what we are building, support us! Give us a :star: or get in touch. We'd love your feedback! Send us a message on piperider.io, join our Discord, or report an issue on GitHub
Contributions
We welcome contributions. See the Set up dev environment and the Contributing guildline to get started.
1 maintainer
Metadata
1 maintainer
Release files for piperider-nightly 0.10.0.20220926
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| piperider-nightly-0.10.0.20220926.tar.gz | 3.6 MB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| piperider_nightly-0.10.0.20220926-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 7.2 MB
Release files / piperider-nightly-0.10.0.20220926.tar.gz
| Download URL | piperider-nightly-0.10.0.20220926.tar.gz |
|---|---|
| Size | 3.6 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7a29b16411d064b53f1e7e26757989633ca9c8638a0a530db10405fc00d635b2
|
|
BLAKE2b-256 checksum How to use checksums |
ecc8fb0b9cebf271da7fc2914c38bed7c4225a63b67a83279c31ab4f40c96680
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.8.14
|
Release files / piperider_nightly-0.10.0.20220926-py3-none-any.whl
| Download URL | piperider_nightly-0.10.0.20220926-py3-none-any.whl |
|---|---|
| Size | 3.6 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
46f7945a258232579911c7fe5dc6bf2d0b5ceb18911649732a6eced322dadc0e
|
|
BLAKE2b-256 checksum How to use checksums |
40ffb274b5746e2324ffa0a7e46b3526f5ce835201950e22fede22483e63ff2d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.8.14
|
