Skip to main content

Environment diff tool for dbt

Project description

Recce

install pipy Python downloads license

InfuseAI Discord Invite

recce is an environment diff tool for DBT projects. It helps you to compare the results of two environments, such as development and production, and identify the differences.

Features

  1. Support both Web UI & CLI
  2. Multiple diff tools, including lineage diff, schema diff, and query diff. And more in the future.
  3. Use the dbt-core adapter framework to connect to your data warehouse. No additional configuration is required.

Use cases

  1. During development, we can verify new results by contrasting them with those from production prior to pushing the changes.
  2. While reviewing PR, you can grasp the extent of the changes and their impact before merging.
  3. For troubleshooting, you can execute ad-hoc diff queries to pinpoint the root causes.

Usage

Prerequisites

You have to have at least two environments in your dbt project. For example, one is for development and another is for production. You can prepare two targets with separate schemas in your DBT profile. Here is profiles.yml example

jaffle_shop:
  target: dev
  outputs:
    dev:
      type: duckdb
      path: jaffle_shop.duckdb
      schema: dev
    prod:
      type: duckdb
      path: jaffle_shop.duckdb
      schema: main

Getting Started

5 minutes walkthrough by jaffle shop example

  1. Installation

    pip install recce
    
  2. Go to your DBT project

    cd your-dbt-project/
    
  3. Prepare base artifacts: DBT generates artifacts when every invocation. You can find these files in the target/ folder.

    artifacts DBT command
    manifest.json dbt run, dbt build, ..
    catalog.json (optional) dbt docs generate

    Copy the artifacts for the base environment to target-base/ folder.

  4. Run the recce server.

    recce server
    

    Recce would diff environments between target/ and target-base/

Query Diff

You can run query diff in both Web UI and CLI

  • Web UI: Go to Query tab

    select * from {{ ref("mymodel") }}
    
  • CLI:

    recce diff --sql 'select * from {{ ref("mymodel") }}'
    

Primay key

In the query diff, primary key columns serve as the fundamental identifiers for distinguishing each record uniquely across both sides.

  • Web UI: In the query result, click the key icons in the column headers to toggle if it is in the primary key list.

  • CLI: Use the option --primary-keys to specify the primary keys. Use a comma to separate the columns if it is a compound key.

    recce diff --primary-keys event_id --sql 'select * from {{ ref("events") }} order by 1'
    

Q&A

Q: How recce connect to my data warehouse? Does recce support my data warehouse?

recce use the dbt adapter to connect to your warehouse. So it should work for your data warehouse.

Q: What credential does recce connect to the two environments?

Recce uses the same target in the profile to connect your warehouse. If you use the default target dev, it uses the credentials to connect to both environments. So please make sure that the credential can access both environments.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

recce-nightly-0.1.0.20240108.tar.gz (585.3 kB view details)

Uploaded Source

Built Distribution

recce_nightly-0.1.0.20240108-py3-none-any.whl (595.6 kB view details)

Uploaded Python 3

File details

Details for the file recce-nightly-0.1.0.20240108.tar.gz.

File metadata

File hashes

Hashes for recce-nightly-0.1.0.20240108.tar.gz
Algorithm Hash digest
SHA256 c31dec9f04ceea84fcbcb8d91c7235fef5b75d7a5642f9073a0d40755696613a
MD5 8642194a8e9158b31c4097d4a43a0458
BLAKE2b-256 cd9b1c1d9d55c033b239b29c2368d308ed9405cbbd53ba98aabe26b64a4a0b42

See more details on using hashes here.

File details

Details for the file recce_nightly-0.1.0.20240108-py3-none-any.whl.

File metadata

File hashes

Hashes for recce_nightly-0.1.0.20240108-py3-none-any.whl
Algorithm Hash digest
SHA256 2209cf412f9048e21f990ef3f9a0a057cd40f6c43f7d6ef525e3e610691b8d75
MD5 d46903e41c7909e784531f5ab43d3d6a
BLAKE2b-256 d467e8a5157f4967b2dd4620f5cb211c1e15b091ae504c1b7d2cb2a89aa8ee07

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page