dbt-refmerge

Finds duplicated import CTEs in your dbt models — two CTEs reading different columns from the same {{ ref(...) }} — proves the merged version returns identical rows on your warehouse, and rewrites the file. Anything it can't prove, it leaves alone.

Use it

pip install dbt-refmerge
cd your-dbt-project
dbt-refmerge scan

That's it. The dialect is detected automatically from your profiles.yml/manifest.json (override with --adapter postgres if needed). You'll get one line per opportunity:

models/marts/orders.sql:3: CTEs order_items, order_items_summary import model.shop.stg_order_items; run dbt-refmerge check to prove a merge

Each line is a lead: two CTEs over the same upstream model that may merge into one. Then, on a branch:

git checkout -b refmerge-cleanup
export DBT_REFMERGE_SCRATCH_SCHEMA=refmerge_scratch     # where verification views go
dbt-refmerge check                                      # proves each merge on your warehouse
dbt-refmerge fix models/marts/orders.sql --dry-run      # preview the rewrite
dbt-refmerge fix models/marts/orders.sql                # re-proves, then applies it
git diff                                                # review, test, open a PR

scan and check never touch your files. fix refuses to write unless the proof passes on current data (applied=false plus a reason means it's working, not broken).

To prove a merge, check builds the original and the merged model as two views in the scratch schema (dbt creates it if needed; it must not be a schema your models build into), compares their column types and their rows as multisets in a single query, then drops both views. All models in a check share one batch of dbt calls, so a large project costs about as many dbt invocations as a small one, and a model that fails doesn't block the others. If a run is interrupted, dbt-refmerge cleanup --run-id <id> drops whatever it left (the id is in check --json). Verification supports PostgreSQL; scan parses 16 dialects (snowflake, bigquery, duckdb, databricks, redshift, trino, spark, sqlite, tsql, oracle, exasol, clickhouse, and more).

Keep new duplicates out

scan needs no dbt run and no warehouse, so it works as a commit hook or a pull request check (both need 0.2.0 or later):

# .pre-commit-config.yaml
- repo: https://github.com/John-Cusack/dbt-refmerge
  rev: v0.2.0
  hooks:
    - id: dbt-refmerge-scan

# a GitHub Actions step: annotates pull requests at each duplicate
- uses: John-Cusack/dbt-refmerge@v0.2.0
  with:
    adapter: snowflake

See integrations for options and other CI systems.

Documentation

• Command-line reference, with exit codes
• Configuration: .dbt-refmerge.toml, environment variables and adapters
• Verification: what check creates, the privileges it needs and its limits
• Reason codes: why a model was refused and what to do about it
• JSON output
• Integrations: pre-commit, GitHub Actions and other CI

Develop it

Keep the inner test loop focused and in memory:

pip install -e ".[dev,integration]"
python3 -m pytest -q tests/unit/test_odd_scenarios.py  # odd-scenario safety lane
python3 -m pytest -q                                  # unit lane (default)
python3 -m ruff check .
python3 -m mypy --strict src

The default run is the in-memory unit lane and takes a few seconds on a typical development machine. Two slower lanes are opt-in:

python3 -m pytest -q -m fake_dbt      # subprocess tests against tests/fakes/fake_dbt.py

docker run --rm -d --name refmerge-pg -p 5432:5432 \
  -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=refmerge postgres:16
export REFMERGE_TEST_PG_DSN=postgresql://postgres:postgres@localhost:5432/refmerge
python3 -m pytest -q -m warehouse     # needs Postgres and dbt-postgres
python3 -m pytest -q -m "" --cov      # every lane; enforces the coverage floor

See CONTRIBUTING.md for conventions and the release process, and SECURITY.md to report a vulnerability. IMPROVEMENT_PLAN.md tracks what comes next.

Support

If this saves you an afternoon, buy me a coffee.

MIT license. Requires Python ≥ 3.11 and your own dbt installation.