Recce
Recce is a PR review tool designed for DBT projects. It facilitates the comparison of results between two environments, such as development and production, and helps in identifying any differences.
Features
- Lineage diff
- Schema diff
- Row count diff
- Ad-Hoc Query and Query Diff
- Profile Diff
- Value Diff
- Checklist
Use cases
Recce is primarily designed for PR Review purposes. However, it can be expanded to the following use cases:
- During development, we can verify new results by contrasting them with those from production prior to pushing the changes.
- While reviewing PR, you can grasp the extent of the changes and their impact before merging.
- For troubleshooting, you can execute ad-hoc diff queries to pinpoint the root causes.
Usage
Prerequisites
You have to have at least two environments in your dbt project. For example, one is for development and another is for production. You can prepare two targets with separate schemas in your DBT profile. Here is profiles.yml example
jaffle_shop:
target: dev
outputs:
dev:
type: duckdb
path: jaffle_shop.duckdb
schema: dev
prod:
type: duckdb
path: jaffle_shop.duckdb
schema: main
Getting Started
5 minutes walkthrough by jaffle shop example
-
Installation
pip install recce
-
Go to your DBT project
cd your-dbt-project/
-
Prepare artifacts for base environment in
target-base/foldergit checkout main # Generate artifacts for base environment to 'target-base' dbt docs generate --target prod --target-path target-base/
-
Prepare artifacts for current working environment
git checkout feature/my-awesome-feature # Run dbt and generate artifacts for current working environments dbt run dbt docs generate
-
Run the recce server.
recce server # or with persistent state # recce server issue-123.json
Recce would diff environments between
target/andtarget-base/
Recce use dbt artifacts, which is generated when every invocation. You can find these files in the target/ folder.
| artifacts | DBT command |
|---|---|
| manifest.json | dbt docs generate, dbt run, .. |
| catalog.json | dbt docs generate |
Lineage Diff
Ad-Hoc Query and Query Diff
You can use any dbt macros installed in your project.
select * from {{ ref("mymodel") }}
In the query diff, which involves comparing the results from two different environments, the browser is required to pull all result data to the client side. Consequently, minimizing the data volume in the query results is essential for efficiency and performance.
Schema diff
To use schema diff, make sure that both environments have catalog.json.
Row count diff
Profile diff
Profile diff uses the get_profile from dbt_profiler. Make sure that this package is installed in your project.
packages:
- package: data-mie/dbt_profiler
version: <version>
Please reference [dbt-profiler] to understand the definition of each profiling stats.
Value diff
- Added: New added PKs.
- Removed: Removed PKs.
- Matched: For a column, the count of matched value of common PKs.
- Matched %: For a column, the ratio of matched over common PKs.
PK: Primary key
Value diff uses the compare_column_values from audit-helper. Make sure that this package is installed in your project.
packages:
- package: dbt-labs/audit_helper
version: <version>
Value diff requires to select a column as private key. The catalog.json is required to list the available columns.
Checklist
When you feel that your query is worth recording, you can add it to the checklist and name the title and leave a description. This is very helpful for subsequent posting in PR comments.
Q&A
Q: How recce connect to my data warehouse? Does recce support my data warehouse?
recce use the dbt adapter to connect to your warehouse. So it should work for your data warehouse.
Q: What credential does recce connect to the two environments?
Recce uses the same target in the profile to connect your warehouse. If you use the default target dev, it uses the credentials to connect to both environments. So please make sure that the credential can access both environments.
1 maintainer
Metadata
1 maintainer
Release files for recce-nightly 0.6.0.20240204
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| recce-nightly-0.6.0.20240204.tar.gz | 669.1 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| recce_nightly-0.6.0.20240204-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 1.4 MB
Release files / recce-nightly-0.6.0.20240204.tar.gz
| Download URL | recce-nightly-0.6.0.20240204.tar.gz |
|---|---|
| Size | 669.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
e9d9c87e829ff40f2646a460b894c1e1a419ea18354cca0ffb5ed808e3530289
|
|
BLAKE2b-256 checksum How to use checksums |
858bebc3654a61631a917e799587272a1a52d0da2a0658d83d2439793c79e943
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.10.13
|
Release files / recce_nightly-0.6.0.20240204-py3-none-any.whl
| Download URL | recce_nightly-0.6.0.20240204-py3-none-any.whl |
|---|---|
| Size | 681.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
4f5f10ba3c88040933d19463817b1f285049311e3c564552832cdf32085034a3
|
|
BLAKE2b-256 checksum How to use checksums |
9fa4df4204393a13d5fda0fa2bc8aaf62cea95da70c6ad0594fcf1e1363fed91
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.10.13
|
