recce
recce is a environment diff tool for dbt
Features
- Support both Web UI & CLI
- Multiple diff tools, including lineage diff, schema diff, and query diff. And more in the future.
- Use the dbt-core adapter framework to connect to your data warehouse. No additional configuration required.
Use cases
- During development, we can verify new results by contrasting them with those from production prior to pushing the changes.
- While reviewing PR, you can grasp the extent of the changes and their impact before merge.
- For troubleshooting, you can execute ad-hoc diff queries to pinpoint the root causes.
Usage
Prerequisites
You have to have at least two environments in your dbt project. For example, one is for developing and another is for production. You can prepare two targets with separate schemas in you dbt profile. Here is profiles.yml example
jaffle_shop:
target: dev
outputs:
dev:
type: duckdb
path: jaffle_shop.duckdb
schema: dev
prod:
type: duckdb
path: jaffle_shop.duckdb
schema: main
Getting Started
5 minutes walkthrough by jaffle shop example
-
Installation
pip install recce -
Go to your dbt project
cd your-dbt-project/ -
Prepare base artifacts: DBT generates artifacts when every invocation. You can find these files in the
target/folder.artifacts dbt command manifest.json dbt run,dbt build, ..catalog.json (optional) dbt docs generateCopy the artifacts for base environment to
target-base/ -
Run the recce server.
recce serverRecce would diff environments between
target/andtarget-base/
Query Diff
You can run query diff in both Web UI and CLI
-
Web UI: Go to Query tab
select * from {{ ref("mymodel") }}
-
CLI:
recce diff --sql 'select * from {{ ref("mymodel") }}'
Primay key
In the query diff, primary key columns serve as the fundamental identifiers for distinguishing each record uniquely across both sides.
-
Web UI: In the query result, click the key icons in the column headers to toggle if it is in the primary key list.
-
CLI: Use the option
--primary-keysto specify the primary keys. Use comma to separate the columns if it is a compound key.recce diff --primary-keys event_id --sql 'select * from {{ ref("events") }} order by 1'
Q&A
Q: How recce connect to my data warehouse? Does recce support my data warehouse?
recce use the dbt adapter to connect to your warehouse. So it should work for your data warehouse.
Q: What credential does recce connect to the two environments?
Recce uses the same target in the profile to connect your warehouse. If you use the default target dev, it use the credentials to connect to both environments. So please make sure that the credential able to access both environments.
1 maintainer
Metadata
1 maintainer
Release files for recce-nightly 0.1.0.20231129
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| recce-nightly-0.1.0.20231129.tar.gz | 520.6 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| recce_nightly-0.1.0.20231129-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 1.0 MB
Release files / recce-nightly-0.1.0.20231129.tar.gz
| Download URL | recce-nightly-0.1.0.20231129.tar.gz |
|---|---|
| Size | 520.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
6261a8c4491d6ba03e72cdab38300d7c94de11662e55d38be7b6756a45d2d188
|
|
BLAKE2b-256 checksum How to use checksums |
5e3bc3c6726877289dc21d3eb0554dd77bdbddd25cfa1da721aaed02b52a19e9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.10.13
|
Release files / recce_nightly-0.1.0.20231129-py3-none-any.whl
| Download URL | recce_nightly-0.1.0.20231129-py3-none-any.whl |
|---|---|
| Size | 526.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2c4940092847e93a6ee42d18f16d78ba35593d0e72f593664299a20d4e232862
|
|
BLAKE2b-256 checksum How to use checksums |
b68f2128b76da3c6385b4ac098fbee68be191f86d0e868ac2dabca616c469d41
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.10.13
|
