dvt-core 0.2.4

Cross-engine data transformation built on dbt — federate SQL models across any database, warehouse, or bucket

DVT — Data Virtualization Tool

Any source. Any target. One SQL project.

DVT is a cross-engine data transformation CLI built on dbt. Write SQL models that JOIN tables living on different database engines — MySQL with Snowflake, Oracle with PostgreSQL, anything with anything — and DVT handles extraction, federation, and loading automatically.

{{ config(materialized='f_table') }}   -- "f" = federated

select
    c.customer_name,
    o.order_date,
    o.total_amount
from {{ source('crm', 'customers') }} c      -- lives on MySQL
join {{ source('erp', 'orders') }} o          -- lives on Snowflake
  on c.customer_id = o.customer_id
where o.order_date >= '2024-01-01'

pip install dvt-core
dvt sync       # installs every adapter + driver your profiles.yml needs
dvt run        # dbt runs what dbt can; DVT federates the rest

Website & full docs: getdvt.net · Quickstart · PyPI

---

Architecture

DVT is a wrapper around stock dbt-core, not a fork. dbt compiles the project and runs every standard model; DVT picks up the models dbt can't express — cross-engine federation models and locally-executed Python models — and runs them through its own pipeline:

                ┌──────────────────────────────────────────────────────────┐
                │                      dvt run / build                     │
                └──────────────────────────────────────────────────────────┘
                                          │
                         dbt compile  ──  manifest.json
                                          │
              ┌───────────────────────────┴───────────────────────────┐
              │                                                       │
      standard models                                        federated models
   (table / view / incremental)                          (f_table / f_incremental
              │                                          / Python models)
              ▼                                                       │
      ┌───────────────┐                 ┌─────────────────────────────┴───────┐
      │   stock dbt   │                 │ 1. DECOMPOSE  — SQLGlot parses the  │
      │  runs them on │                 │    compiled SQL, maps every table   │
      │  the default  │                 │    to a profiles.yml connection     │
      │    target     │                 │ 2. TRANSPILE + PUSH DOWN — minimal  │
      └───────────────┘                 │    per-source queries in each       │
                                        │    engine's own dialect: only the   │
                                        │    columns used, every single-source│
                                        │    filter pushed to the source      │
                                        │ 3. EXTRACT  — Sling streams each    │
                                        │    result to Parquet, in parallel   │
                                        │ 4. COMPUTE  — Parquet loads into a  │
                                        │    local DuckDB cache; the model's  │
                                        │    JOIN/GROUP BY logic runs there   │
                                        │ 5. LOAD     — Sling delivers the    │
                                        │    result to the target: any        │
                                        │    database, warehouse, or cloud    │
                                        │    bucket (as Parquet/CSV objects)  │
                                        └─────────────────────────────────────┘

Design principles

• Wrapper, not fork. DVT shells out to the dbt CLI and reads manifest.json — it never imports dbt internals. Your dbt version can move; DVT keeps working. Every dbt feature (Jinja, macros, packages, tests, snapshots, docs) works because it is dbt running it.
• Dual-dialect by design. Federation models are written in DuckDB dialect and are transpiled per source — they never depend on any engine's syntax. Standard models compile in the default target's dialect via the official dbt adapter. Result: switching your default target (e.g. PostgreSQL → Databricks) is a profile change, not a rewrite.
• Pushdown first. The fastest row to move is the one you never extract. Filters and column pruning push down to source engines; when every table in a model lives on one engine, the entire query pushes down and nothing moves at all.
• Your hardware does the joining. Cross-engine compute happens in DuckDB on the machine running DVT — source databases only ever serve simple filtered SELECTs, and no warehouse bills you for the join.

What DVT adds on top of dbt

Capability	How
Cross-engine models	materialized='f_table' / 'f_incremental' (long aliases: federated_*, federation_*)
Incremental federation	watermark-based delta extraction; persistent local cache between runs
Per-model targets	config(target='other_output') on federated models — results land on any profile output (standard models are refused a foreign target)
Bucket targets	materialize models to S3 / GCS / Azure as Parquet or CSV
Python models, locally	dbt-style .py models run on your machine (APIs, pandas, ML) — results land on any target
Bulk seeds	dvt seed loads csv / tsv / parquet / json / jsonl / avro / arrow / xlsx via Sling to any engine, with enforced snake_case columns
One-command setup	dvt sync installs every adapter and driver your profiles.yml requires
Connection diagnostics	dvt debug --all checks every connection on both layers (dbt adapter + Sling)
Cross-engine catalog	dvt docs builds one catalog + lineage UI across all engines in your project

Everything else — profiles.yml in ~/.dbt/, sources.yml, ref/source/config, tests, snapshots — is standard dbt. Existing dbt projects work unchanged; sources gain an optional meta.connection to bind them to other engines.

Supported engines

Every engine supported by both dbt and Sling is a DVT adapter — as federation source, federation target, seed target, and default target:

PostgreSQL · MySQL · MariaDB · SQL Server · Oracle · SQLite · DuckDB · Snowflake · BigQuery · Redshift · Databricks · ClickHouse · Trino · Athena · Microsoft Fabric — plus S3, GCS, and Azure Blob as bucket storages.

Anything else (MongoDB, Elasticsearch, REST APIs, …) is reachable through locally-executed Python models. Details per adapter: getdvt.net/docs/adapters.

Learn more

• The federation engine, step by step
• Predicate pushdown deep dive
• Switching the default target (lakehouse portability)
• Pricing & plans

License

DVT is distributed as compiled wheels on PyPI. Free tier covers local files and transactional databases; Pro adds cloud buckets and Python models; Max adds cloud warehouses. See getdvt.net/pricing.