recce

recce is a environment diff tool for dbt

Features

Support the same dbt adapter framework as dbt.
Support both Web UI & CLI
Lineage diff

Use cases

When developing, we can check the new result by comparing against the production one.
When reviewing PR, you can understand the change impacts.
When trouble shooting, you can run adhoc dif query to find the root causes.

Usage

Prerequisites

You have to have at least two environments in your dbt project. For example, one is for developing and another is for production. You can prepare two targets with separate schemas in you dbt profile. Here is profiles.yml example

jaffle_shop:
  target: dev
  outputs:
    dev:
      type: duckdb
      path: jaffle_shop.duckdb
      schema: dev
    prod:
      type: duckdb
      path: jaffle_shop.duckdb
      schema: main

Getting Started

Installation

git clone git@github.com:InfuseAI/recce.git
cd recce
pip install -e .

Put the manifest.json of production (or any environment you would like to diff) in the target-base/ folder. manifest.json is one of the generated artifacts for each dbt command execution. You can find it in target/ folder by default.
Develop your awesome features
```
dbt run
```
Run the recce command
```
recce server
```
Review the linage diff.
Switch to query tab, Write and run a query
```
select * from {{ ref('mymodel') }}
```
where ref is a Jinja macro to reference a model name.

Under the hood, recce uses the manifest.json under target/ and target-base/ to geenrate query and execute.

Run Query Diff

You can either run in Web UI

recce server

or run in CLI

recce diff --sql 'select * from {{ ref('mymodel') }}'

Specify the primary key columns

In the query diff, we use primary key columns as the basis for identifying the same record on both sides.

There are two ways to specify the primary key

Define in the SQL: Add the config macro in your sql.

{{
   config( primary_key=['DATE_WEEK', 'COUNTRY'])
}}

select ...

Select in the query result: In the Web UI, you can click the key icons in the column headers to toggle if a column is a primary key.

Q&A

Q: How `recce` connect to my data warehouse? Does recce support my data warehouse?

recce use the dbt adapter to connect to your warehouse. So it should work for your data warehouse.

Q: What credential does `recce` connect to the two environments?

Recce uses the same target in the profile to connect your warehouse. If you use the default target dev, it use the credentials to connect to both environments. So please make sure that the credential able to access both environments.

Metadata

Release files for recce-nightly 0.1.0.20231127

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for recce-nightly 0.1.0.20231127
File	Size	Uploaded
recce-nightly-0.1.0.20231127.tar.gz	584.6 kB	Nov 27, 2023	Details

Built distribution (wheel)

Table of built distributions (wheels) for recce-nightly 0.1.0.20231127
File	Interpreter	ABI	Platform	Reset
recce_nightly-0.1.0.20231127-py3-none-any.whl 593.2 kB Nov 27, 2023	Python 3	none	any	Details

Total release size: 1.2 MB

Release files / recce-nightly-0.1.0.20231127.tar.gz

Download URL	recce-nightly-0.1.0.20231127.tar.gz
Size	584.6 kB
Tags	Source
SHA-256 checksum How to use checksums	`e98f27d3a5d5a2a72f6af49984774995fade44d3fa167c95918eb3380f697291`
BLAKE2b-256 checksum How to use checksums	`adad2a0dc988065ecc2e4b02ff67b324675d03f4db6388fa5a0b55ed94607980`
Upload date	Nov 27, 2023
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.2 CPython/3.10.13`

Release files / recce_nightly-0.1.0.20231127-py3-none-any.whl

Download URL	recce_nightly-0.1.0.20231127-py3-none-any.whl
Size	593.2 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`a724293e337cceeff9a0633e7e64ea53a6aeccb4b8271a0a608dd31d1942d855`
BLAKE2b-256 checksum How to use checksums	`70f2af0d1b3bb9a4194de8476f3ef0ac19ae7f790430a0d5670597c677c83574`
Upload date	Nov 27, 2023
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.2 CPython/3.10.13`