Data Hub Gandalf

Python library for Databricks data pipelines — SCD Types 0/1/2/3/4/6, upserts, surrogate keys, checksums, and schema validations. You shall not pass... bad data.

Installation

Published on PyPI as data-hub-gandalf (imported as gandalf):

pip install data-hub-gandalf

In a Databricks notebook or cluster:

%pip install data-hub-gandalf

For an offline / air-gapped install, build the wheel with Poetry and install it from a Unity Catalog volume:

poetry build            # -> dist/data_hub_gandalf-<version>-py3-none-any.whl

%pip install /Volumes/<catalog>/<schema>/<volume>/data_hub_gandalf-<version>-py3-none-any.whl

Quickstart

from gandalf import merge_delta_table

merge_delta_table(
    spark=spark,
    df=df,
    path="main.gold.dim_client",
    ids=["id_client"],
    merge_type="scd",       # SCD Type 2, full snapshot — see Capabilities
    sk_type="dim",
    sk_name="sk_client",
)

Capabilities

A scan-map of the merge strategies and helpers gandalf provides. Legend: ✅ supported · ⚙️ automatic · ⚠️ supported with a caveat · — not applicable.

SCD history strategies manage four control columns — scd_start_dt, scd_end_dt, scd_is_current, and scd_checksum — automatically. Point gandalf at a table that uses different names by passing scd_columns=SCDColumns(is_current="current_flag", checksum="check_sum").

See COMPARISON.md for how gandalf compares to other SCD libraries (mack, koheesio, hydro, fabricks, dbt, Databricks DLT…) — SCD-type coverage, test rigor (real-engine E2E / Spark / Databricks), and where gandalf is, and isn't, distinctive.

What it does

merge_delta_table

Persists a Spark DataFrame into a Delta table. Internally it:

1. Validates the target Delta table exists
2. Filters incoming data incrementally via delta_col_filter (optional)
3. Detects duplicates in the source
4. Validates column schema against the target
5. Generates checksum for change detection
6. Generates surrogate key (if sk_type is set)
7. Auto-casts column types to match target schema
8. Executes the chosen merge strategy
9. Validates row count after merge

from gandalf import merge_delta_table

merge_delta_table(
    spark=spark,
    df=df,
    path="main.gold.dim_client",
    ids=["id_client"],
    merge_type="scd",       # see strategies below
    sk_type="dim",
    sk_name="sk_client",
)

Merge strategies — canonical names (legacy aliases accepted):

Advanced strategies (scd0/scd3/scd4/scd6) are configured through SCDConfig:

from gandalf import SCDColumns, SCDConfig, merge_delta_table

# SCD Type 3 — previous-value columns
merge_delta_table(
    spark, df, "catalog.schema.dim_client", ["id_client"],
    scd_config=SCDConfig(scd_type="scd3", ids=["id_client"],
                         previous_columns={"address": "previous_address"}),
)

# SCD Type 4 — current table + history table
merge_delta_table(
    spark, df, "catalog.schema.current_client", ["id_client"],
    scd_config=SCDConfig(scd_type="scd4", ids=["id_client"],
                         history_path="catalog.schema.hist_client"),
)

# SCD Type 6 — Type 2 rows + previous-value columns
merge_delta_table(
    spark, df, "catalog.schema.dim_client", ["id_client"],
    scd_config=SCDConfig(scd_type="scd6", ids=["id_client"],
                         previous_columns={"address": "previous_address"}),
)

SCDConfig also carries the control-column names (columns=SCDColumns(...)), return_metrics, and tracked_columns. Pass either scd_columns or scd_config — not both.

Utilities

from gandalf import generate_check_sum, generate_dim_sk, generate_fact_sk, hash_columns

hash_columns(*cols)                    # SHA-256 Column expr for withColumn/select/when
generate_check_sum(df, control_cols)   # adds scd_checksum over all business columns
generate_dim_sk(df, ids, sk_name)      # SHA-256 surrogate key (unique per SCD version)
generate_fact_sk(df, ids, sk_name)     # xxhash64 surrogate key (deterministic)

Data protection guidance

• Prefer checksum comparisons over blind updates so reruns remain idempotent.
• Validate source-vs-target schemas before Delta MERGE.
• Use deterministic surrogate keys for dimensions and facts.
• Keep partition overwrites constrained by explicit predicates.
• Avoid embedding notebook-only globals inside core library logic; pass dependencies as function arguments.