delta-engine 0.4.0

Framework for defining, validating, and evolving tables in a consistent, safe, and controlled way.

delta-engine

Declarative, safety-first schema and metadata management for Delta Lake tables on Databricks.

Define the state your tables should have in Python. Delta Engine reads their current Unity Catalog state, calculates the difference, validates whether each change is safe, and executes only the DDL needed to reconcile them.

There are no migration scripts to maintain and no DDL statements to hand-order. Your declarations remain the source of truth across the lifetime of the table.

What Delta Engine gives you

• Desired-state reconciliation: declare the complete table state rather than a sequence of migrations.
• Safe in-place evolution: unsafe changes, such as type narrowing, repartitioning or destructive schema changes, are rejected before any SQL runs.
• Reviewable plans: preview semantic changes and compiled DDL with a dry run before applying them.
• Drift detection: compare version-controlled declarations against the live Unity Catalog state in local workflows or CI.
• Scoped ownership: manage governance metadata without taking ownership of a table’s schema or data lifecycle.
• Dependency-aware execution: synchronise groups of tables in the correct order when primary-key and foreign-key relationships exist.

Delta Engine does not need to own how a table’s data is produced. It can manage the catalog state around tables populated by PySpark jobs, declarative pipelines, dbt models or other systems.

Install

pip install delta-engine

The base package is pure Python with no runtime dependencies: declaring and planning schemas needs no PySpark. Running a sync needs either a Spark session or a Databricks SQL warehouse connection. Inside a Databricks notebook, Spark is already provided; elsewhere, install the [spark] extra for local development against a Spark session, or the [sql] extra to sync through a SQL warehouse instead — schema sync from CI without a Spark session or cluster. See the installation guide for both extras and their requirements.

Install delta-engine[cli] to run delta-engine plan MODULE:ATTRIBUTE through any standard Databricks unified-auth configuration. The CLI always shows the semantic diff, report, and planned SQL and never applies the generated plan. See the CLI reference.

Quickstart

from delta_engine.databricks import build_spark_engine
from delta_engine.schema import Column, DeltaTable, Integer, String

customers = DeltaTable(
    catalog="dev",
    schema="silver",
    name="customers",
    columns=[
        Column("id", Integer(), nullable=False),
        Column("name", String()),
    ],
)

engine = build_spark_engine(spark)  # `spark` is provided by your Databricks notebook
engine.sync(customers)         # creates the table, or no-ops if it already matches

Every sync follows the same process:

read live state
    → compare with the declaration
    → validate the differences
    → build a deterministic plan
    → resolve table dependencies
    → execute

Validation happens before execution. When a table contains an unsafe change, Delta Engine does not execute a partially valid plan for that table.

Documentation

Start with how a sync works for the model, or jump to what you need:

Getting started

• Installation
• Getting started tutorial — define a table and run your first sync

Concepts

• How a sync works — the phases between calling sync and getting a report
• The safety model — what the engine blocks, and why

How-to guides

• Configure a table — properties, tags, comments, keys, and partitioning
• Deploy metadata only — roll out governance metadata with no schema change
• Preview changes with a dry run
• Gate schema changes in CI — report planned changes and fail unreadable or unsafe plans
• Handle sync failures — inspect SyncReport and act on each status

Reference

• CLI — the read-only plan command, connection contract, output, and exit codes
• Capabilities and limitations — what the engine can and cannot manage
• Data types — supported types and Spark SQL equivalents
• Safe-change rules — changes the engine blocks at validation
• Run report schema — the to_dict() payload, field by field
• API reference

Architecture

• Architecture — layers, ports and adapters, design decisions
• Implement a custom adapter — the CatalogStateReader and PlanExecutor ports
• Add a new action type — extend Action, ActionPhase, and the compiler