dot-for-dbt 0.2.0

dot - the Data Orchestration Tool (for dbt)

The Data Orchestration Tool for dbt (dot-for-dbt)

dot is a lightweight companion CLI for dbt that lets you run any dbt command for an optional named context and an exact git commit/ref using the shorthand <context?>@<ref>. Adding @<ref> builds that historical version into a schema automatically suffixed with the commit’s short hash (e.g. analytics_a1b2c3d4) so your current schemas stay untouched. This enables reproducible historical builds, safe experimentation, side‑by‑side diffing, and confident migration or release validation.

Installation

Requires Python >= 3.12 (resolved automatically by uv).

Using uv (recommended persistent install):

uv tool install dot-for-dbt
dot --help

Upgrade:

uv tool upgrade dot-for-dbt

Ephemeral run (no global install):

uvx dot-for-dbt --help

Pin version:

uv tool install dot-for-dbt==0.1.1
uvx dot-for-dbt==0.1.1 --help

Via pip (alternative):

pip install dot-for-dbt

Uninstall:

uv tool uninstall dot-for-dbt

Quick Example

# Build current project using default context
dot build

# Build historical commit (isolated schema)
dot build @abc1234

# Build specific context at a ref
dot run dev@feature/my-branch

CLI Usage

Basic usage:

dot <dbt_command> <context> [--dry-run] [--no-gitignore-check]

• <dbt_command> is any supported dbt command (e.g., build, run, test).
• <context> (Optional) is the environment/context which you want to target as defined in your vars.yml. If you do not specify a context, the default context from vars.yml will be used.

To build or run against a specific git commit in an isolated schema, append @<gitref or commit> to the context:

dot <dbt_command> <context>@<gitref or commit>

You can also build into the default context at a certain commit:

dot <dbt_command> @<gitref or commit>

This will check out the specified commit in a git worktree, generate a dedicated profiles.yml, and build into yourschema_<short git hash>. This enables reproducible, isolated builds for any point in your repository history.

.gitignore Requirement

The .dot directory contains build artifacts and must be ignored by git. By default, the CLI enforces this by checking for a .dot/ entry in your .gitignore file before running any commands. If missing, you will be prompted to add it automatically. The CLI will refuse to run if .dot/ is not ignored.

To bypass this enforcement (not recommended for normal use), use the --no-gitignore-check flag:

dot build --no-gitignore-check

To ensure correct setup, add the following line to your .gitignore:

.dot/

vars.yml Behavior

• vars.yml is optional. If it does not exist in your working directory, dot will proceed with default settings and no context-based variables.
• If vars.yml exists but is malformed (invalid YAML), dot will print an error and exit.
• If you specify a context that does not exist in vars.yml, dot will print an error and exit.
• If no context is specified and no default is set in vars.yml, dot will proceed with default settings.

Isolated Builds

Isolated builds let you execute a dbt command against the exact contents of any git commit (or ref) in a clean, temporary worktree while writing all database objects into a schema that is namespaced by the commit hash. This provides:

• Reproducibility (build exactly what existed at that commit)
• Confidence to roll forward/back by inspecting isolated artifacts

Future features are planned to make more extensive use of isolated builds

Quick Start

To build using the default context specified in vars.yml at a particular historical git reference, simply omit the context and use @<ref>:

dot build @abc1234

Build an explicit context against a ref:

dot run dev@feature/my-branch
dot test prod@v1.2.0

Build using a short or symbolic ref (branch, tag, HEAD~N, etc.):

dot run dev@HEAD~1
dot build prod@main

Syntax Summary

<context?>@<gitref>

• context (optional) — name defined under context in vars.yml
• gitref (optional) — branch, tag, full/short hash, reflog expression, etc.
• If @<gitref> is supplied with no leading context, the default context is used.
• If no @ suffix is provided, this is a normal (non‑isolated) build against the current state of your project.

What Happens Internally

1. Resolve the supplied <gitref> to a full commit hash.

2. Construct: .dot/isolated_builds/<commit_hash>/

3. Create (or reuse if it already exists) a clean git worktree at:

   .dot/isolated_builds/<commit_hash>/worktree/
   

4. Locate the dbt project inside that worktree matching your original working project path.

5. Detect the active profiles.yml location by invoking dbt debug --config-dir.

6. Read the selected profile + target (your context name).

7. Write an isolated profiles.yml to:

   .dot/isolated_builds/<commit_hash>/<context>/profiles.yml
   

   with the target schema updated to _<commit_hash>.

8. Set dbt CLI args so that:

   • --project-dir points at the isolated worktree project
   • --profiles-dir points at .dot/isolated_builds/<commit_hash>/<context>
   • --target-path is .dot/isolated_builds/<commit_hash>/<context>/target
   • --log-path is .dot/isolated_builds/<commit_hash>/<context>/logs

9. Execute your dbt command.

Schema Naming

The target schema becomes:

<original_schema>_<short_hash>

Where <short_hash> is the first 8 characters of the full commit hash. For example, if your original target schema is analytics and the commit is 6b777b8c94771a74..., the isolated schema is:

analytics_6b777b8c

Directory Layout

Example layout for an isolated build:

.dot/
  isolated_builds/
    <full_commit_hash>/
      worktree/             # Clean checkout at that commit
      dev/                  # One folder per context used with this commit
        profiles.yml        # Auto-generated, schema rewritten with _<short_hash>
        target/             # dbt artifacts (manifest, run results, etc.)
        logs/               # dbt logs for this isolated run
      prod/
        profiles.yml
        target/
        logs/

If you build multiple contexts (dev, prod) for the same commit, each gets its own context subdirectory.

Examples

Diff models between current development and a feature branch:

dot build dev
dot build dev@feature/new-metric
# Compare artifacts or query both schemas: analytics vs analytics_<shot_hash>

Test a migration before merging:

dot run prod@migration/rename-columns
dot test prod@migration/rename-columns

Roll forward validation (red/green):

dot build prod@current_prod_tag
dot build prod@next_release_candidate
# Validate row counts, constraints, performance before switching consumers

Historical investigation:

dot run dev@2024-12-01-tag

profiles.yml Detection & Rewriting

dot invokes dbt debug --config-dir (wrapped through its own command builder) to locate the effective profiles.yml. It then:

• Loads the user’s configured profile
• Extracts the target matching the active context
• Updates only the schema field (preserving credentials, threads, etc.)
• Writes a minimal isolated profiles.yml containing just that profile + target

Passing Additional dbt Args

Anything after -- is passed through untouched:

dot run dev@main -- --select my_model+

Cleanup

Currently there is no automatic cleanup. To reclaim space:

• Drop old schemas manually from your warehouse
• Remove stale directories under .dot/isolated_builds/

Automatic management of old build artifacts and schemas is planned for a future release. Please let me know if this would be important to you!

Troubleshooting

Symptom	Cause	Action
Error: Profile not found	Active context or profile missing	Verify profiles.yml and context name
Commit not found	Bad ref	Run git show <ref> to validate
Schema clutter	Many builds kept	Periodically prune .dot/isolated_builds and drop old schemas
Wrong default context	context.default unset or unexpected	Set default under context in vars.yml

Reference

For architectural rationale see: ADR 0001: Isolated Builds.

Architectural Decision Records

Architectural decisions are documented in the adr/ directory.

• ADR 0001: Isolated Builds

Changelog

See CHANGELOG.md for released versions.

License

This project is licensed under the MIT License. See the LICENSE file for full details.

SPDX-License-Identifier: MIT