dvt-core 0.2.7

Cross-engine data transformation built on dbt — federate SQL models across any database, warehouse, or bucket

DVT — Data Virtualization Tool

Any source. Any target. One SQL project.

DVT is a cross-engine data transformation CLI built on dbt. Write SQL models that JOIN tables living on different database engines — MySQL with Snowflake, Oracle with PostgreSQL, anything with anything — and DVT handles extraction, federation, and loading automatically.

{{ config(materialized='f_table') }}   -- "f" = federated

select
    c.customer_name,
    o.order_date,
    o.total_amount
from {{ source('crm', 'customers') }} c      -- lives on MySQL
join {{ source('erp', 'orders') }} o          -- lives on Snowflake
  on c.customer_id = o.customer_id
where o.order_date >= '2024-01-01'

pip install dvt-core
dvt sync       # installs every adapter + driver your profiles.yml needs
dvt run        # dbt runs what dbt can; DVT federates the rest
dvt version    # (worth running once — you'll see)

Website & full docs: getdvt.net · Quickstart · PyPI

---

Architecture

DVT is a wrapper around stock dbt-core, not a fork. dbt compiles the project and runs every standard model; DVT picks up the models dbt can't express — cross-engine federation models and locally-executed Python models — and runs them through its own pipeline:

                ┌──────────────────────────────────────────────────────────┐
                │                      dvt run / build                     │
                └──────────────────────────────────────────────────────────┘
                                          │
                         dbt compile  ──  manifest.json
                                          │
              ┌───────────────────────────┴───────────────────────────┐
              │                                                       │
      standard models                                        federated models
   (table / view / incremental)                          (f_table / f_incremental
              │                                          / Python models)
              ▼                                                       │
      ┌───────────────┐                 ┌─────────────────────────────┴───────┐
      │   stock dbt   │                 │ 1. DECOMPOSE  — SQLGlot parses the  │
      │  runs them on │                 │    compiled SQL, maps every table   │
      │  the default  │                 │    to a profiles.yml connection     │
      │    target     │                 │ 2. TRANSPILE + PUSH DOWN — minimal  │
      └───────────────┘                 │    per-source queries in each       │
                                        │    engine's own dialect: only the   │
                                        │    columns used, every single-source│
                                        │    filter pushed to the source      │
                                        │ 3. EXTRACT  — Sling streams each    │
                                        │    result to Parquet, in parallel   │
                                        │ 4. COMPUTE  — Parquet loads into a  │
                                        │    local DuckDB cache; the model's  │
                                        │    JOIN/GROUP BY logic runs there   │
                                        │ 5. LOAD     — Sling delivers the    │
                                        │    result to the target: any        │
                                        │    database, warehouse, or cloud    │
                                        │    bucket (as Parquet/CSV objects)  │
                                        └─────────────────────────────────────┘

The Sling direct path (0.2.5)

When every source in an f_table model lives on one connection, steps 3–5 collapse into a single hop: the whole query — joins and aggregations included — is transpiled to that engine's dialect and submitted to Sling as a custom-SQL stream loaded straight to the target. No Parquet staging, no DuckDB. The check is automatic and per-run; any failure falls back to the standard pipeline. Add a source on another engine tomorrow and the same model silently switches to the federation pipeline — the model never changes.

Execution paths

Your model	Who runs it	How
table / view / incremental	stock dbt	natively on the default target — whole-query pushdown by definition
f_table / f_incremental, sources on one connection	DVT	Sling direct: full query pushed to the source engine, result streamed to the target
f_table / f_incremental, sources span connections	DVT	decompose → per-source pushdown extraction → DuckDB compute → load
.py models	DVT	locally executed; result loads to any target

Standard models that set a non-default target= are refused with guidance (dbt silently ignores the kwarg — your model would land on the wrong engine).

Design principles

• Wrapper, not fork. DVT shells out to the dbt CLI and reads manifest.json — it never imports dbt internals. Your dbt version can move; DVT keeps working. Every dbt feature (Jinja, macros, packages, tests, snapshots, docs) works because it is dbt running it.
• Dual-dialect by design. Federation models are written in DuckDB dialect and are transpiled per source — they never depend on any engine's syntax. Standard models compile in the default target's dialect via the official dbt adapter. Result: switching your default target (e.g. PostgreSQL → Databricks) is a profile change, not a rewrite.
• Pushdown first. The fastest row to move is the one you never extract. Filters and column pruning push down to source engines; when every table in a model lives on one engine, the entire query pushes down and nothing moves at all.
• Your hardware does the joining. Cross-engine compute happens in DuckDB on the machine running DVT — source databases only ever serve simple filtered SELECTs, and no warehouse bills you for the join.

What DVT adds on top of dbt

Capability	How
Cross-engine models	materialized='f_table' / 'f_incremental' (long aliases: federated_*, federation_*)
Incremental federation	watermark-based delta extraction; persistent local cache between runs
Per-model targets	config(target='other_output') on federated models — results land on any profile output (standard models are refused a foreign target)
Bucket targets	materialize models to S3 / GCS / Azure as Parquet or CSV
Python models, locally	dbt-style .py models run on your machine (APIs, pandas, ML) — results land on any target
Bulk seeds	dvt seed loads csv / tsv / parquet / json / jsonl / avro / arrow / xlsx via Sling to any engine, with enforced snake_case columns
One-command setup	dvt sync installs every adapter and driver your profiles.yml requires
Connection diagnostics	dvt debug --all checks every connection on both layers (dbt adapter + Sling)
Cross-engine catalog	dvt docs builds one catalog + lineage UI across all engines in your project

Everything else — profiles.yml in ~/.dbt/, sources.yml, ref/source/config, tests, snapshots — is standard dbt. Existing dbt projects work unchanged; sources gain an optional meta.connection to bind them to other engines.

Supported engines

Every engine supported by both dbt and Sling is a DVT adapter — as federation source, federation target, seed target, and default target:

PostgreSQL · MySQL · MariaDB · SQL Server · Oracle · SQLite · DuckDB · Snowflake · BigQuery · Redshift · Databricks · ClickHouse · Trino · Athena · Microsoft Fabric — plus S3, GCS, and Azure Blob as bucket storages.

Anything else (MongoDB, Elasticsearch, REST APIs, …) is reachable through locally-executed Python models. Details per adapter: getdvt.net/docs/adapters.

Learn more

• The federation engine, step by step
• Predicate pushdown deep dive
• Switching the default target (lakehouse portability)
• Pricing & plans

License

DVT is distributed as compiled wheels on PyPI. Free tier covers local files, transactional databases, and Python models; Pro adds cloud buckets (S3/GCS/Azure); Max adds cloud warehouses. See getdvt.net/pricing.