dbt-hodor

Documentation | Interactive guide with a run simulator | Design spec

dbt-hodor is a free, self-hosted alternative to dbt Labs' dbt State. It runs inside dbt. For every selected node it decides whether to reuse the object that is already in the warehouse or to build it again. A node is reused when its logic has not changed and its upstream data has not changed since the last build.

It needs no dbt platform account and sends nothing to external services. State lives in a Delta table on Databricks, in a Postgres database such as Databricks Lakebase, or in a local DuckDB file.

Status: early development. hodor has end-to-end tests on dbt-duckdb and live tests on a Databricks SQL warehouse and on Databricks Lakebase; the test docs list what they cover. It has not run in a large production project yet. Before enabling reuse in production, run in shadow mode for at least a week of scheduled runs and review its decisions. See the rollout guidance. Changes between releases, including the breaking ones, are in Upgrading.

Why "hodor"

The name is a backronym: Hold Old Data, Or Rebuild. That is the only decision hodor makes, once per node.

hodor holds the door in front of your warehouse. A node gets through to a rebuild only when its code changed or new data arrived upstream. Everything else keeps the data it already has.

How it works

For each node in dbt run, dbt build, dbt test, dbt seed or dbt snapshot, hodor:

1. Hashes the logic. For a seed this is the CSV file. Otherwise the compiled SQL is normalized with sqlglot (Databricks dialect), so case, quoting, comments and table aliases do not count as changes. Values that differ on every run, such as invocation_id, are replaced first. The node config is hashed as well.
2. Checks the upstream data. Each upstream table has a data version. On Databricks it comes from the Delta commit history; maintenance commits such as OPTIMIZE or VACUUM do not count as data changes. On DuckDB it is a content fingerprint.
3. Decides. It compares both against the record of the node's last build and returns reuse or build, with a reason code.
4. Records. It appends one record per node to the state table.

A run with no code changes skips sqlglot entirely: when the compiled SQL is byte-identical to the last build, hodor reuses the stored analysis. For 10,000 compiled models, hashing then takes about 0.1 seconds instead of about 4 minutes.

Safety: fail open

Every failure leads to a build, never to a reuse. An error while deciding, a store that cannot be read, a table whose data version is unknown, an input hodor cannot trace (for example read_files(...)): in each case the node builds as plain dbt would. hodor never fails a dbt run.

Reason codes

The rules are checked in this order; the first match decides. dbt-hodor explain -s <model> shows the reason for one node, with the hashes and upstream data versions it compared. The full rules are in the design spec.

Modes

Installing the package never skips anything until someone sets on.

Configuration

Node configs

hodor reads the same state: block as dbt State, so a project can switch between the two. Set it in YAML:

# dbt_project.yml
models:
  my_project:
    +state:
      lag_tolerance: 4h
      require_fresh_data_from: any

pre_clone, allow_clones and defer_to_target are accepted and ignored, with a warning.

Settings

Set them on the profiles.yml target, or for one run with --vars '{hodor_mode: off}'. Precedence: environment variable, then --vars, then the profile key.

Name parts may contain letters, digits, _ and -. hodor always quotes them.

Where the state lives

delta has no default schema on purpose: hodor writes nothing to a catalog until someone names the schema. Without it, hodor turns itself off with a warning.

# profiles.yml
my_project:
  target: prod
  outputs:
    prod:
      type: databricks
      catalog: my_catalog
      # ... connection settings ...
      hodor_mode: shadow
      hodor_schema: ops
      # state table: my_catalog.ops.hodor_state
    local:
      type: duckdb
      path: dev.duckdb
      hodor_store: duckdb
      # state table: main.hodor_state in .hodor/state.duckdb

The state table decides what gets skipped in prod. Grant MODIFY on it only to the job service principals.

Scope of v1

Supported: models materialized as table, incremental (append, merge, delete+insert, insert_overwrite, replace_where; an unset strategy counts as merge) and view, seeds, snapshots, and data tests (generic and singular), on dbt-databricks 1.12.7 and dbt-duckdb, with dbt-core 1.12.x.

Model versions: each version is its own node with its own table and records. Moving latest_version rebuilds the models that ref() it.

Slim CI: --defer and --favor-state are supported, and each push of a pull request reuses what the last push built. See Slim CI. The node configs pre_clone, allow_clones and defer_to_target are not.

Always built as plain dbt: Python models, microbatch, materialized views, streaming tables, metric views, custom materializations, custom incremental strategies and unit tests.

Known limit: a change to dbt's or the adapter's own macros, for example a dbt-databricks upgrade, does not show in any hash. Run once with HODOR_MODE=shadow after such an upgrade: shadow builds every node and records the builds. off records nothing, so the old records would still count. Project and package macros that a materialization or a hook runs are in the config hash.

Development

Requires uv. Local development and tests run on DuckDB, with no Databricks access.

uv sync
uv run pytest
uv run ruff check
uv run ruff format

The live Databricks test is opt-in: see docs/development.md.

dbt-hodor is available on PyPI:

pip install dbt-hodor

To build a wheel from the source checkout, run uv build.

Contributing

Issues and pull requests are welcome. CONTRIBUTING.md has the setup, the checks to run, and what a decision change needs. A wrong reuse, where hodor skipped a node it should have rebuilt, is a security-grade bug: report it privately through the Security tab.

License

Apache-2.0. NOTICE carries the attribution for the derived files and the trademark notice; LICENSE is the full licence text.