ODGS Databricks Bridge

Transform your Databricks Unity Catalog into active ODGS runtime enforcement schemas.

Unity Catalog describes your data. ODGS enforces it.

The ODGS Databricks Bridge is an institutional connector that translates Databricks Unity Catalog table and column metadata into cryptographically addressable ODGS enforcement schemas. It converts passive catalog definitions — column constraints, non-nullable fields, type assertions — into mechanically executable governance rules enforced at pipeline runtime.

Architecturally aligned with federated data sovereignty principles, and on a path toward standardization.

See it in action

Real bridge, real engine, real ES256-signed rule pack, real Ed25519 audit seal — click to watch on GitHub. Only the Unity Catalog REST API is mocked; see examples/marketplace-demo/ for the full honest-accounting demo rig.

Architecture

flowchart LR
    subgraph UC["Databricks Unity Catalog"]
        CAT["Catalogs & Schemas"]
        TBL["Tables & Columns"]
        CON["Constraints\n(NOT NULL, Types)"]
    end

    subgraph Bridge["odgs-databricks-bridge"]
        T["DatabricksBridge\n.sync()"]
        TR["Transformer\n→ ODGS Schema"]
    end

    subgraph ODGS["ODGS Protocol (odgs>=6.1.0)"]
        I["Universal Interceptor\nHARD_STOP / SOFT_STOP / WARNING / LOG_ONLY"]
        WB["Write-Back\n→ Unity Catalog Comments"]
    end

    subgraph MP["Metric Provenance (Commercial)"]
        SC["S-Cert Registry\n(JWS Audit Seal)"]
    end

    CAT & TBL & CON --> T --> TR --> I
    I -->|"sovereign_audit.log"| WB --> UC
    I -->|"Certified S-Cert"| SC

What Gets Generated

Note on TYPE_CHECK rules: severity is always INFO regardless of --severity — the generated logic_expression uses type(), which the ODGS engine's sandboxed evaluator doesn't allow, so these rules can't actually block a pipeline today. Binding them for real enforcement needs a custom evaluator extension.

Install

pip install odgs-databricks-bridge

Quick Start

Python API

from odgs_databricks import DatabricksBridge

bridge = DatabricksBridge(
    workspace_url="https://adb-1234567890.azuredatabricks.net",
    token="dapi...",
    organization="acme_corp",
)

# Sync all tables from a catalog → ODGS metric definitions
bridge.sync(
    catalog="production",
    output_dir="./schemas/custom/",
    output_type="metrics",
)

# Sync column constraints → ODGS enforcement rules
bridge.sync(
    catalog="production",
    schema_filter="finance",
    output_dir="./schemas/custom/",
    output_type="rules",
    severity="HARD_STOP",
)

CLI

# Using standard Databricks SDK environment variables
export DATABRICKS_HOST=https://adb-1234567890.azuredatabricks.net
export DATABRICKS_TOKEN=dapi...

odgs-databricks sync \
    --org acme_corp \
    --catalog production \
    --schema finance \
    --type rules \
    --severity HARD_STOP \
    --output ./schemas/custom/

# Push compliance results back to Unity Catalog table comments
odgs-databricks write-back \
    --log-path ./sovereign_audit.log \
    --url https://adb-1234567890.azuredatabricks.net \
    --token dapi...

Output Schema

{
  "$schema": "https://metricprovenance.com/schemas/odgs/v5",
  "metadata": {
    "source": "databricks",
    "organization": "acme_corp",
    "tables_processed": 12,
    "items_generated": 47
  },
  "items": [
    {
      "rule_urn": "urn:odgs:custom:acme_corp:rule:revenue_amount_not_null",
      "name": "revenue.amount NOT NULL",
      "severity": "HARD_STOP",
      "constraint_type": "NOT_NULL",
      "target_table": "production.finance.revenue",
      "plain_english_description": "The revenue amount field must never be null in financial transaction records",
      "content_hash": "a1b2c3..."
    }
  ]
}

Bi-Directional Write-Backs

The bridge supports Bi-Directional Sync: it parses your sovereign_audit.log offline and pushes compliance results back into Unity Catalog table comments — creating a deterministic feedback loop for Data Stewards without compromising the air-gapped nature of the core ODGS protocol.

Authentication

Regulatory Alignment

This bridge is designed for organisations governed by:

For cryptographic attestation (Ed25519 JWS audit seals and signed .odgs-bundle Regulation Packs for EU AI Act, DORA, and CSRD), see the Metric Provenance Enterprise Platform.

Get started: metricprovenance.com/executive-brief (Executive Brief) · Certified packs: metricprovenance.com/#suite

Requirements

• Python ≥ 3.9
• odgs ≥ 6.1.0 (core protocol)
• Databricks workspace with Unity Catalog enabled

Related

• ODGS Protocol — The core enforcement engine
• ODGS FLINT Bridge — TNO FLINT legal ontology connector
• ODGS Collibra Bridge — Collibra integration
• ODGS Snowflake Bridge — Snowflake integration

License

Apache 2.0 — Metric Provenance | The Hague, NL 🇳🇱