delta-engine 0.3.0

Framework for defining, validating, and evolving tables in a consistent, safe, and controlled way.

delta-engine

Declarative schema management for Delta Lake tables on Databricks. You declare the state a table should have; the engine reads the state it actually has, computes the difference, checks that the difference is safe to apply, and runs exactly the DDL needed to close the gap.

There is no migration script to write and no DDL to hand-order. The declaration is the source of truth, and every sync reconciles the catalog to it.

Backends: delta-engine targets Delta Lake on Databricks with Unity Catalog today. The planning core is deliberately backend-free — backends plug in as adapters that read catalog state and execute plans — so other backends, such as open-source Unity Catalog, can be added without changing the model.

Install

pip install delta-engine

The base package is pure Python with no runtime dependencies: declaring and planning schemas needs no PySpark. Running a sync needs a Databricks environment, which provides Spark and Delta. See the installation guide for the [databricks] extra used for local development.

Quickstart

from delta_engine.databricks import build_engine
from delta_engine.schema import Column, DeltaTable, Integer, String

customers = DeltaTable(
    catalog="dev",
    schema="silver",
    name="customers",
    columns=[
        Column("id", Integer(), nullable=False),
        Column("name", String()),
    ],
)

engine = build_engine(spark)  # `spark` is provided by your Databricks notebook
engine.sync(customers)         # creates the table, or no-ops if it already matches

What a sync does

Every sync runs the same phase chain for each table: read the current catalog state, diff it against your declaration, validate that the drift is safe to fix in place, plan deterministic DDL, order tables so foreign-key dependencies are created first, execute, and return a per-table SyncReport. Unsafe changes — dropping data, changing a column's type, repartitioning — fail validation with a named rule before any SQL runs.

Documentation

Start with how a sync works for the model, or jump to what you need:

Getting started

• Installation
• Getting started tutorial — define a table and run your first sync

Concepts

• How a sync works — the phases between calling sync and getting a report
• The safety model — what the engine blocks, and why

How-to guides

• Configure a table — properties, tags, comments, keys, and partitioning
• Deploy metadata only — roll out governance metadata with no schema change
• Preview changes with a dry run
• Handle sync failures — inspect SyncReport and act on each status

Reference

• Capabilities and limitations — what the engine can and cannot manage
• Data types — supported types and Spark SQL equivalents
• Safe-change rules — changes the engine blocks at validation
• API reference

Architecture

• Architecture — layers, ports and adapters, design decisions
• Implement a custom adapter — the CatalogStateReader and PlanExecutor ports
• Add a new action type — extend Action, ActionPhase, and the compiler