dbt-data-contracts (dbt_data_contracts)

An enterprise-grade, lightweight data contract control plane for multi-repository dbt ecosystems, mesh architectures, and data platforms.

---

Overview

dbt model contracts validate what a model produces internally within its own project. However, in modern multi-repository data ecosystems and data mesh topologies, data producers face critical challenges:

• Invisible Downstream Breakage: How do producers know if altering a model breaks downstream consumers across separate git repositories?
• No Heavy Cloning in CI: Producer CI pipelines should not clone, parse, or execute hundreds of downstream repositories just to verify compatibility.
• Explicit Consumer Commitments: Downstream teams need a structured, versioned way to declare exact column, type, quality, and SLA requirements.
• Transitive Impact Simulation: A single model change can cascade across multiple hops of models, dashboards, and APIs.

dbt-data-contracts provides a complete, vendor-neutral control plane to declare, ingest, publish, verify, discover, simulate, and enforce data contracts across distributed data teams.

---

Core Capabilities

┌─────────────────────────────────────────────────────────────────────────────────────────┐
│                                   dbt Ecosystem & Mesh                                  │
│   dbt manifest.json / static YAML / OpenAPI Specs / Consumer Declarations / Warehouses  │
└────────────────────────────────────────────┬────────────────────────────────────────────┘
                                             │ Ingestion & Auto-Discovery
                                             ▼
┌─────────────────────────────────────────────────────────────────────────────────────────┐
│                         Canonical Data Contract Domain Layer                            │
│           PublishedContract • ColumnContract • SemanticModels • FreshnessSLA            │
└───────────────────────┬─────────────────────────────────────────┬───────────────────────┘
                        │                                         │
                        ▼                                         ▼
┌──────────────────────────────────────┐       ┌──────────────────────────────────────────┐
│       Universal Contract Registry     │       │        Compatibility & Blast Engine      │
│  SQLite • PostgreSQL • GitOps • HTTP  │◄─────►│    SemVer Rules • Transitive DAG Graphs   │
│  Immutability • Promotion • Lifecycle │       │    Quality • SLAs • Warehouse Drift      │
└───────────────────────┬──────────────┘       └──────────────────┬───────────────────────┘
                        │                                         │
                        ▼                                         ▼
┌─────────────────────────────────────────────────────────────────────────────────────────┐
│                              Control Plane & CI/CD Gates                                │
│    dbt-data-contracts CLI • Pre-Commit Hooks • PR Comment Reporters • Web Dashboard     │
│    OpenAPI/ODCS/Avro/Protobuf Exporters • Mock Generators • Slack/Teams Webhooks        │
└─────────────────────────────────────────────────────────────────────────────────────────┘

• Canonical Data Contracts: Pure domain representations independent of dbt, databases, or cloud vendors.
• SemVer Compatibility Engine: Deterministic classification of changes (PATCH, MINOR, MAJOR, UNKNOWN) based on column changes, multi-dialect typing, nullability, and quality assertions.
• Explicit Consumer Expectations: Downstream teams declare version ranges (e.g., ^1.0.0) and required columns in data-contract-consumers.yml.
• Automated SQL AST Lineage & Consumer Inference: SQLGlot-based column-level lineage engine that inspects Jinja ref() expressions and SQL ASTs to auto-generate consumer declarations (dbt-data-contracts consumer infer).
• Multi-Hop Transitive Blast-Radius & Dependency DAG Simulator: Construct complete dependency graphs, discover cascading impact paths across N tiers of consumers, and render ASCII trees or Mermaid diagrams (dbt-data-contracts impact).
• Pre-Commit Git Hook & Local Developer Loop: Fast local validation of staged dbt files against contracts, governance policies, and waivers before pushing (dbt-data-contracts hook check).
• Automated PR Sticky Comments & CI Gates: Native sticky PR thread status management on GitHub Actions and Azure DevOps Pipelines.
• Organization-Scale Auto-Discovery with Activity Filtering: Discover dbt projects, manifests, and consumer contracts across Azure DevOps, GitHub, GitLab, and Bitbucket with commit age filtering (--max-commit-age-days 7) and multi-branch scanning (--branch).
• OpenAPI 3.0 / 3.1 & GraphQL SDL Adapters: Auto-discover OpenAPI specifications in repositories, import API schemas as contracts (import openapi), and export published contracts to OpenAPI 3.1 and GraphQL SDL.
• Open Data Standards Interoperability: Bi-directional translation for Open Data Contract Standard (ODCS v3.0), Apache Avro (.avsc), Protocol Buffers v3 (.proto), JSON Schema (2020-12), OpenLineage assertion facets, and DataHub / OpenMetadata emitters.
• Synthetic Test Data & Mock Dataset Generator: Deterministically generate mock test datasets conforming strictly to contract types, constraints, nullability, allowed values, and patterns in CSV, JSON, or SQL INSERT format (dbt-data-contracts codegen mock).
• Automated Backward-Compatibility View & Shim Generator: Generate dbt SQL shim models mapping new breaking outputs back to legacy schemas, and generate consumer migration YAMLs (dbt-data-contracts codegen shim / codegen migrate-consumer).
• Contract Sunset & Deprecation Lifecycle: Track contract statuses (ACTIVE, DEPRECATED, TOMBSTONED), calculate breaking consumer impact when models disappear, and enforce deprecation/sunset deadlines (dbt-data-contracts check sunset / registry deprecate/tombstone/reconcile).
• Multi-Environment Registry Promotion: Promote contracts and consumer expectations between environments (e.g. dev -> staging -> prod) with conflict detection (dbt-data-contracts registry promote).
• Live Warehouse Schema Introspection & Drift Detection: Introspect live physical warehouse tables across Snowflake, BigQuery, Databricks, Postgres, Redshift, and DuckDB to detect live schema drift (dbt-data-contracts check drift).
• Control Plane REST API & Web Dashboard: FastAPI server with interactive web UI at / for catalog exploration, version histories, and consumer lineage DAG visualization (dbt-data-contracts server start).
• Real-Time Webhook Notifications: Event-driven alerts with Slack Block Kit, Microsoft Teams Adaptive Cards, and HMAC-SHA256 signature verification.

---

Installation

# Core package with CLI
pip install dbt-data-contracts

# Optional server extensions (FastAPI & Uvicorn for REST API / Web Dashboard)
pip install "dbt-data-contracts[server]"

Both dbt-data-contracts and dbt-contracts are available as CLI aliases.

---

Quickstart & Core Workflow

1. Ingest contracts from a dbt manifest

Inspect public contracts discovered in a dbt project manifest:

dbt-data-contracts ingest --manifest target/manifest.json

2. Publish a contract version to the registry

Publish contracts for public dbt models (e.g. version 1.0.0):

dbt-data-contracts publish \
  --manifest target/manifest.json \
  --version 1.0.0 \
  --registry .contracts.db

3. Register a downstream consumer expectation

Downstream teams define data-contract-consumers.yml (or generate it automatically with consumer infer):

consumer: finance_analytics
owner: team-finance@example.com

dependencies:
  - product: core_sales
    model: fct_orders
    version: "^1.0.0"
    expectations:
      columns:
        order_id:
          data_type: bigint
          required: true
        amount:
          data_type: numeric
          required: true
      quality:
        amount:
          min_value: 0
      sla:
        max_staleness: "24h"

Register the expectation in the registry:

dbt-data-contracts consumer register \
  --file data-contract-consumers.yml \
  --registry .contracts.db

4. Check proposed changes in Producer CI

In producer pull request pipelines, test proposed contracts against published contracts and registered consumers:

dbt-data-contracts check \
  --manifest target/manifest.json \
  --proposed-version 1.1.0 \
  --registry .contracts.db \
  --post-pr-comment

If a breaking change is detected (e.g., removing amount while finance_analytics requires it), the gate fails with exit code 1 and posts an explanatory breakdown:

BREAKING CHANGE DETECTED: core_sales.fct_orders

Proposed: 1.1.0 (Current: 1.0.0)
Severity: MAJOR

Breaking changes:
  - Column 'amount' was removed

Affected consumers:
  - finance_analytics (expects 'amount: numeric', pinned to ^1.0.0)

SemVer Validation:
  FAIL: Breaking changes require a MAJOR version bump (expected >= 2.0.0).

If the proposed version is updated to 2.0.0, the check passes with exit code 0, confirming that 1.0.0 remains intact for existing consumers while 2.0.0 introduces the new interface version.

---

CLI Command Reference

Command Group	Command	Description
Ingest & Publish	ingest	Inspect public models from a dbt manifest.json.
	publish	Publish public dbt models as versioned immutable contracts.
Compatibility & CI	check	Check proposed contract changes against registry and consumers (with --auto-review).
	check drift	Inspect live physical warehouse tables and detect schema drift.
	check sunset	Evaluate contract deprecation and sunset deadlines against consumers.
	diff	Diff contract schemas between manifests or against the registry.
Discovery	discover	Discover local nested dbt projects, manifests, and consumer files.
	discover organization	Scan remote repos (Azure DevOps, GitHub, GitLab, Bitbucket) with recent commit filtering (--max-commit-age-days) and branch scanning (--branch).
Dependency & Impact	impact / blast-radius	Evaluate multi-hop transitive blast radius and export DAG diagrams (ASCII/Mermaid).
Consumers	consumer register	Register a data-contract-consumers.yml file into the registry.
	consumer list	List registered consumer expectations.
	consumer infer	Auto-generate consumer expectations from SQL AST column lineage and ref() calls.
Code Generation	codegen shim	Generate a backward-compatible dbt SQL shim model for major upgrades.
	codegen migrate-consumer	Generate consumer migration YAML for upgrading contract version pins.
	codegen mock	Generate synthetic mock datasets (CSV, JSON, SQL INSERT) conforming to contract specs.
	codegen dbt-test	Generate dbt schema test YAML, singular test SQL, or compliance macros from a contract.
	codegen changelog	Generate Markdown release notes and changelog between contract versions.
	codegen pipeline	Generate complete CI/CD pipeline code (producer gating or consumer checks) for Azure DevOps, GitHub Actions, GitLab CI, and Bitbucket.
Git Hooks	hook check	Pre-commit git hook validation for staged dbt files and schema changes.
Standards & Schemas	export	Export contracts to openapi, graphql, odcs, avro, proto, jsonschema.
	import	Import contract definitions from openapi (3.0/3.1) or odcs.
Registry Management	registry list	List published contracts and version histories.
	registry deprecate	Mark a contract version as deprecated with a sunset date.
	registry tombstone	Mark a contract version as tombstoned.
	registry reconcile	Detect disappeared/deleted models and compute breaking consumer impacts.
	registry promote	Replicate and promote contracts across registry environments (dev -> prod).
Control Plane Server	server start	Launch the FastAPI control plane REST API and interactive web dashboard.

---

Developer Workflow & Pre-Commit Hook

Integrate contract validation into your local development workflow using .pre-commit-config.yaml:

repos:
  - repo: https://github.com/dbt-data-contracts/dbt-data-contracts
    rev: v0.3.1
    hooks:
      - id: dbt-data-contracts-check
        args: ["--registry", ".contracts.db"]

Or run locally directly:

dbt-data-contracts hook check --registry .contracts.db

---

Multi-Hop Transitive Blast Radius

Simulate cascading downstream impacts across complex data mesh graphs:

dbt-data-contracts impact \
  --product core_sales \
  --model fct_orders \
  --registry .contracts.db \
  --depth 5 \
  --format ascii

Output:

TRANSITIVE BLAST RADIUS REPORT: core_sales.fct_orders
Total Direct Consumers:     1
Total Transitive Consumers: 2
Total Impacted Models:      2

IMPACT GRAPH:
core_sales.fct_orders [PRODUCER]
  ├── finance_analytics (direct) [fct_monthly_revenue]
  │     └── executive_dashboard (transitive) [mart_arr_summary]
  └── marketing_attribution (direct) [fct_customer_ltv]

Or export as Mermaid diagrams for documentation:

dbt-data-contracts impact --product core_sales --model fct_orders --format mermaid

---

Standards & API Interoperability

OpenAPI 3.0/3.1 Import & Export

Export dbt data product contracts to OpenAPI 3.1 JSON/YAML:

dbt-data-contracts export openapi \
  --product core_sales \
  --model fct_orders \
  --registry .contracts.db \
  --output openapi.json

Import existing OpenAPI definitions as data contracts:

dbt-data-contracts import openapi \
  --file specs/orders-api.yaml \
  --product core_sales \
  --version 1.0.0 \
  --registry .contracts.db

GraphQL SDL Export

dbt-data-contracts export graphql \
  --product core_sales \
  --model fct_orders \
  --registry .contracts.db \
  --output schema.graphql

Open Data Contract Standard (ODCS v3.0) & Streaming

# Export to ODCS YAML
dbt-data-contracts export odcs --product core_sales --model fct_orders --registry .contracts.db

# Export to Apache Avro schema (.avsc)
dbt-data-contracts export avro --product core_sales --model fct_orders --registry .contracts.db

# Export to Protocol Buffers v3 (.proto)
dbt-data-contracts export proto --product core_sales --model fct_orders --registry .contracts.db

---

Universal Registry Backends

dbt-data-contracts supports SQLite, PostgreSQL, GitOps repositories, and HTTP REST endpoints:

# Local SQLite file
dbt-data-contracts check --registry .contracts.db

# Central PostgreSQL database
export CONTRACTS_DATABASE_URL="postgresql://user:password@pg.company.internal:5432/contracts"
dbt-data-contracts check --manifest target/manifest.json

# GitOps-backed repository catalog
dbt-data-contracts check --registry "git+https://github.com/company/data-contracts-catalog.git"

# Remote Control Plane REST API
dbt-data-contracts check --registry "http://contracts.internal.net:8000" --token "$CONTRACTS_TOKEN"

---

Running the Control Plane Server

Launch the REST API server and interactive web dashboard:

pip install "dbt-data-contracts[server]"
dbt-data-contracts server start --host 0.0.0.0 --port 8000 --registry .contracts.db

Open http://localhost:8000 in your browser to explore the catalog, inspect contract schemas and version histories, visualize consumer lineage DAGs, and verify compatibility interactively. Use http://localhost:8000/graphql for the live GraphiQL playground.

---

Static Documentation Portal & Markdown Docs

Generate standalone zero-dependency static documentation sites (e.g. for GitHub Pages, S3, or Azure Blob):

# Build single-page static HTML documentation portal with client-side search
dbt-data-contracts docs build --registry .contracts.db --output ./site

# Export structured Markdown folder tree for MkDocs / Docusaurus
dbt-data-contracts docs export-markdown --registry .contracts.db --output docs/contracts

---

Automated Consumer Version Upgrades & PRs

Synchronize consumer expectation files when upstream data products publish new versions:

dbt-data-contracts consumer upgrade \
  --file data-contract-consumers.yml \
  --product core_sales \
  --model fct_orders \
  --target-version "^2.0.0" \
  --registry .contracts.db

---

CI/CD Pipeline Generator for Producer & Consumer Checks

Generate turnkey CI/CD pipeline definitions across Azure DevOps, GitHub Actions, GitLab CI, and Bitbucket Pipelines:

# Generate Producer Gate (checks PRs against downstream consumers)
dbt-data-contracts codegen pipeline \
  --platform azure-devops \
  --role producer \
  --manifest target/manifest.json \
  --registry "$CONTRACTS_DATABASE_URL" \
  --post-pr-comment \
  --auto-review \
  --output azure-pipelines-contracts.yml

# Generate Consumer Verification Pipeline (checks upstream dependencies & sunsets)
dbt-data-contracts codegen pipeline \
  --platform github-actions \
  --role consumer \
  --consumer-file data-contract-consumers.yml \
  --registry .contracts.db \
  --output .github/workflows/consumer-contracts.yml

---

Runtime Model Execution Guard

Prevent orchestrators (Airflow, Dagster, Prefect, GitHub Actions) from executing dbt run/dbt build against tombstoned or expired sunset models:

dbt-data-contracts exec -- dbt build --select +fct_orders

---

Warehouse Data Diff & Value Distribution Validator

Run database pushdown SQL queries to validate actual table data against quality constraints (nullability, min/max bounds, allowed enums, uniqueness):

dbt-data-contracts check data-diff \
  --product core_sales \
  --model fct_orders \
  --connection-url "snowflake://user:pass@account/db/schema" \
  --fail-on-violation

---

Governance Policies & Breaking-Change Waivers

Enforce organizational governance policies using .dbt-contracts-policy.yml:

require_owner: true
require_model_description: true
require_column_descriptions: true
min_column_description_coverage: 0.8
allowed_data_types:
  - bigint
  - numeric
  - text
  - timestamp
  - boolean

Temporarily authorize breaking changes during migrations using .waivers.yml:

waivers:
  - product: core_sales
    model: fct_orders
    rule: ColumnRemovedRule
    reason: "Decommissioning legacy amount column in Q3 refactor (Jira: DATA-1234)"
    expires_at: "2026-12-31T23:59:59Z"
    approved_by: "data-gov-board"

---

Development & Testing

# Clone the repository
git clone https://github.com/your-org/dbt_contracts.git
cd dbt_contracts

# Set up virtual environment
python -m venv .venv
source .venv/bin/activate  # Or .venv\Scripts\Activate.ps1 on Windows

# Install in editable mode with dev dependencies
pip install -e ".[dev,server]"

# Run tests and linting
pytest
ruff check .
ruff format --check .
mypy src tests

---

License

MIT License. See LICENSE for details.