dot-for-dbt 0.4.0

dot - the Data Orchestration Tool (for dbt)

The Data Orchestration Tool for dbt (dot-for-dbt)

dot is a lightweight companion CLI for dbt that lets you run any dbt command for an optional named environment and an exact git commit/ref using the shorthand <environment?>@<ref>. Adding @<ref> builds that historical version into a schema automatically suffixed with the commit’s short hash (e.g. analytics_a1b2c3d4) so your current schemas stay untouched. This enables reproducible historical builds, safe experimentation, side‑by‑side diffing, and confident migration or release validation.

Installation

Requires Python >= 3.12 (resolved automatically by uv).

Using uv (recommended persistent install):

uv tool install dot-for-dbt
dot --help

Upgrade:

uv tool upgrade dot-for-dbt

Ephemeral run (no global install):

uvx dot-for-dbt --help

Pin version:

uv tool install dot-for-dbt==0.1.1
uvx dot-for-dbt==0.1.1 --help

Via pip (alternative):

pip install dot-for-dbt

Uninstall:

uv tool uninstall dot-for-dbt

Quick Example

# Build current project using default environment
dot build

# Build historical commit (isolated schema)
dot build @abc1234

# Build specific environment at a ref
dot run dev@feature/my-branch

CLI Usage

Basic usage:

dot <dbt_command> <environment> [--dry-run] [--no-gitignore-check]

• <dbt_command> is any supported dbt command (e.g., build, run, test).
• <environment> (Optional) is the environment which you want to target as defined in your dot_environments.yml under the top-level environment: key. If you do not specify an environment, the default environment from dot_environments.yml will be used.

To build or run against a specific git commit in an isolated schema, append @<gitref or commit> to the environment:

dot <dbt_command> <environment>@<gitref or commit>

You can also build into the default environment at a certain commit:

dot <dbt_command> @<gitref or commit>

This will check out the specified commit in a git worktree, generate a dedicated profiles.yml, and build into yourschema_<short git hash>. This enables reproducible, isolated builds for any point in your repository history.

.gitignore Requirement

The .dot directory contains build artifacts and must be ignored by git. By default, the CLI enforces this by checking for a .dot/ entry in your .gitignore file before running any commands. If missing, you will be prompted to add it automatically. The CLI will refuse to run if .dot/ is not ignored.

To bypass this enforcement (not recommended for normal use), use the --no-gitignore-check flag:

dot build --no-gitignore-check

To ensure correct setup, add the following line to your .gitignore:

.dot/

Configuration Files

dot uses two project-level configuration files plus an optional user override:

1. dot_vars.yml (optional)
   Defines variable specifications (metadata and validation). If absent, no variable validation (required/strict) occurs.
2. dot_environments.yml (optional)
   Defines execution environments, default environment selection, dbt CLI argument values, and per‑environment variable value assignments.
3. dot_environments.user.yml (optional, uncommitted)
   Adds or overrides environment definitions / variable assignments locally for a developer.

There is a strict separation:

• Variable specifications (description, allowed values, required, strict) live only in dot_vars.yml.
• Variable values are assigned only inside the environment: structure of the environments files (dot_environments.yml and optionally dot_environments.user.yml) under environment.all.vars or environment.<name>.vars.

dot_vars.yml Example

vars:
  feature_flag:
    description: Enables new metric logic
    values: [true, false]
    strict: true
    required: true
  sample_rate:
    description: Percentage of events to process
    values: [1, 5, 10, 25, 50, 100]
    strict: true
    required: false

dot_environments.yml Example

environment:
  default: dev
  all:
    indirect-selection: buildable
    vars:
      feature_flag: false
  dev:
    target: dev
    vars:
      sample_rate: 10
  prod:
    target: prod
    vars:
      feature_flag: true
      sample_rate: 100

dot_environments.user.yml Example (Local Override)

environment:
  dev:
    vars:
      feature_flag: true   # locally override baseline
    threads: 12

Rules & Behavior

• All files are optional; absence yields default behavior (no environments, no validation).
• If environment.default is set it must name a defined environment.
• Variable validation (required / strict) is applied only to variables declared in dot_vars.yml.
• Undeclared variables assigned in environments are passed through without warnings or errors.
• User override merges shallowly with the project environments file; nested vars mappings are deep merged.

See:

• ADR 0002 for environment configuration & overrides
• ADR 0003 for variable specifications

Isolated Builds

Isolated builds let you execute a dbt command against the exact contents of any git commit (or ref) in a clean, temporary worktree while writing all database objects into a schema that is namespaced by the commit hash. This provides:

• Reproducibility (build exactly what existed at that commit)
• Confidence to roll forward/back by inspecting isolated artifacts

Future features are planned to make more extensive use of isolated builds.

Quick Start

To build using the default environment specified in dot_environments.yml at a particular historical git reference, simply omit the environment and use @<ref>:

dot build @abc1234

Build an explicit environment against a ref:

dot run dev@feature/my-branch
dot test prod@v1.2.0

Build using a short or symbolic ref (branch, tag, HEAD~N, etc.):

dot run dev@HEAD~1
dot build prod@main

Syntax Summary

<environment?>@<gitref>

• environment (optional) — name defined under environment in dot_environments.yml
• gitref (optional) — branch, tag, full/short hash, reflog expression, etc.
• If @<gitref> is supplied with no leading environment, the default environment is used.
• If no @ suffix is provided, this is a normal (non‑isolated) build against the current state of your project.

What Happens Internally

1. Resolve <gitref> to the full 40‑char commit hash and the abbreviated short hash via:
   • git rev-parse <gitref>
   • git rev-parse --short <gitref>
2. Construct: .dot/build/<short_hash>/
3. Create (or reuse) a clean git worktree at:

   .dot/build/<short_hash>/worktree/
   

4. Locate the dbt project inside that worktree matching the original project path.
5. Detect the active profiles.yml location (dbt debug --config-dir).
6. Read the selected profile + target (environment name).
7. Write an isolated profiles.yml to:

   .dot/build/<short_hash>/env/<environment>/profiles.yml
   

   with the target schema updated to <schema>_<short_hash>.
8. Set dbt CLI args so that:
   • --project-dir points at the isolated worktree project
   • --profiles-dir points at .dot/build/<short_hash>/env/<environment>
   • --target-path is .dot/build/<short_hash>/env/<environment>/target
   • --log-path is .dot/build/<short_hash>/env/<environment>/logs
9. Write the full hash to:

   .dot/build/<short_hash>/commit
   

10. Execute the dbt command.

Schema Naming

The target schema becomes:

<original_schema>_<short_hash>

Where <short_hash> is the abbreviated commit hash reported by git rev-parse --short <ref> (length chosen automatically by git to avoid ambiguity). For example, if your original target schema is analytics and the short hash is 6b777b8c, the isolated schema is:

analytics_6b777b8c

Directory Layout

Example layout for an isolated build:

.dot/
  build/
    <short_hash>/           
      worktree/             
      commit                
      env/                  
        dev/
          profiles.yml      
          target/           
          logs/             
        prod/
          profiles.yml
          target/
          logs/

If you build multiple environments (dev, prod) for the same commit, each gets its own environment subdirectory under env/.

Examples

Diff models between current development and a feature branch:

dot build dev
dot build dev@feature/new-metric
# Compare artifacts or query both schemas: analytics vs analytics_<short_hash>

Test a migration before merging:

dot run prod@migration/rename-columns
dot test prod@migration/rename-columns

Roll forward validation (red/green):

dot build prod@current_prod_tag
dot build prod@next_release_candidate
# Validate row counts, constraints, performance before switching consumers

Historical investigation:

dot run dev@2024-12-01-tag

profiles.yml Detection & Rewriting

dot invokes dbt debug --config-dir to locate the effective profiles.yml. It then:

• Loads the user’s configured profile
• Extracts the target matching the active environment
• Updates only the schema field (preserving credentials, threads, etc.)
• Writes a minimal isolated profiles.yml containing just that profile + target

Passing Additional dbt Args

Anything after -- is passed through untouched:

dot run dev@main -- --select my_model+

Cleanup

Currently there is no automatic cleanup. To reclaim space:

• Drop old schemas manually from your warehouse
• Remove stale directories under .dot/build/

Troubleshooting

Symptom	Cause	Action
Error: Profile not found	Active environment or profile missing	Verify profiles.yml and environment name
Commit not found	Bad ref	Run git show <ref> to validate
Schema clutter	Many builds kept	Prune .dot/build & drop old schemas
Wrong default environment	Missing or unexpected environment.default	Set default under environment

Reference

For architectural rationale see: ADR 0001: Isolated Builds.

Architectural Decision Records

Architectural decisions are documented in the adr/ directory.

• ADR 0001: Isolated Builds
• ADR 0002: Environment Configuration
• ADR 0003: Variable Configuration (dot_vars.yml)

Changelog

See CHANGELOG.md for released versions.

License

This project is licensed under the MIT License. See the LICENSE file for full details.

SPDX-License-Identifier: MIT