recce

recce is a environment diff tool for dbt

Features

Support both Web UI & CLI
Multiple diff tools, including lineage diff, schema diff, and query diff. And more in the future.
Use the dbt-core adapter framework to connect to your data warehouse. No additional configuration required.

Use cases

During development, we can verify new results by contrasting them with those from production prior to pushing the changes.
While reviewing PR, you can grasp the extent of the changes and their impact before merge.
For troubleshooting, you can execute ad-hoc diff queries to pinpoint the root causes.

Usage

Prerequisites

You have to have at least two environments in your dbt project. For example, one is for developing and another is for production. You can prepare two targets with separate schemas in you dbt profile. Here is profiles.yml example

jaffle_shop:
  target: dev
  outputs:
    dev:
      type: duckdb
      path: jaffle_shop.duckdb
      schema: dev
    prod:
      type: duckdb
      path: jaffle_shop.duckdb
      schema: main

Getting Started

Installation
```
pip install recce
```
Recce use dbt artifacts to interact with your dbt project. You need to prepare the artifacts for the base environment.
```
# transform the data to data warehouse
dbt run --target prod

# generate the catalog.json
dbt docs generate --target prod
```
The artifacts are generated within the target/ directory. Copy these artifacts into the target-base/ directory as the base state to diff.
```
mkdir -p target-base/
cp -R target/ target-base/
```

Develop your awesome features

# transform the data to data warehouse
dbt run

# generate the catalog.json
dbt docs generate

Run the recce server
```
recce server
```
and open the url link
Check the lineage diff to see the modified node. Click one node to see the schema difference.
Switch to query tab, Write and run a query diff. It would query on the both side and diff the query results.
```
select * from {{ ref("mymodel") }}
```
where ref is a Jinja function to reference a model name.

Query Diff

You can run query diff in both Web UI and CLI

Web UI: Go to Query tab
```
select * from {{ ref("mymodel") }}
```

CLI:

recce diff --sql 'select * from {{ ref("mymodel") }}'

Primay key

In the query diff, primary key columns serve as the fundamental identifiers for distinguishing each record uniquely across both sides.

Web UI: In the query result, click the key icons in the column headers to toggle if it is in the primary key list.
CLI: Use the option --primary-keys to specify the primary keys. Use comma to separate the columns if it is a compound key.
```
recce diff --primary-keys event_id --sql 'select * from {{ ref("events") }} order by 1'
```

Q&A

Q: How `recce` connect to my data warehouse? Does recce support my data warehouse?

recce use the dbt adapter to connect to your warehouse. So it should work for your data warehouse.

Q: What credential does `recce` connect to the two environments?

Recce uses the same target in the profile to connect your warehouse. If you use the default target dev, it use the credentials to connect to both environments. So please make sure that the credential able to access both environments.

Metadata

Release files for recce-nightly 0.1.0.20231128

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for recce-nightly 0.1.0.20231128
File	Size	Uploaded
recce-nightly-0.1.0.20231128.tar.gz	589.3 kB	Nov 28, 2023	Details

Built distribution (wheel)

Table of built distributions (wheels) for recce-nightly 0.1.0.20231128
File	Interpreter	ABI	Platform	Reset
recce_nightly-0.1.0.20231128-py3-none-any.whl 597.4 kB Nov 28, 2023	Python 3	none	any	Details

Total release size: 1.2 MB

Release files / recce-nightly-0.1.0.20231128.tar.gz

Download URL	recce-nightly-0.1.0.20231128.tar.gz
Size	589.3 kB
Tags	Source
SHA-256 checksum How to use checksums	`95cbec356e23113e232d67a705845c3d244c8429fc27024aa73157eebbdd37af`
BLAKE2b-256 checksum How to use checksums	`db5d154041e707264aa75846d61096270b750c9d081ffe0466b166ff13307264`
Upload date	Nov 28, 2023
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.2 CPython/3.10.13`

Release files / recce_nightly-0.1.0.20231128-py3-none-any.whl

Download URL	recce_nightly-0.1.0.20231128-py3-none-any.whl
Size	597.4 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`de03d2ab8ec2f89bd1afd3a61ba5ebd6af014183f30e95b8dbbfa56d67785a3b`
BLAKE2b-256 checksum How to use checksums	`b2dc08b109f470833dc119c243749573114c6ecbb47c33f30080cb67b3d2e007`
Upload date	Nov 28, 2023
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.2 CPython/3.10.13`