databricks-labs-sdp-meta 0.1.0

Project Overview

SDP-META is a metadata-driven framework for Lakeflow Spark Declarative Pipelines. Define your Bronze and Silver pipelines in a JSON or YAML onboarding file — a single generic Declarative Pipeline reads the resulting DataflowSpec at runtime and builds the full processing graph automatically. No pipeline code to write.

📺 New to SDP-META? Watch the SDP-Meta Deep-Dive: Building Data Pipelines at Scale on Databricks — a video walkthrough of the framework with a Demo.

Who it's for: platform and data engineering teams standardizing repeatable Bronze/Silver pipelines across many datasets — onboarding new feeds through metadata instead of new pipeline code, with consistent data quality, quarantine, CDC, clustering, and sink patterns available through Bundles, CLI, UI, MCP, and agent workflows.

When it's not the best fit: one or two simple pipelines, Gold-layer business modeling, tables that each need unique application logic, a managed connector and downstream logic that already satisfy the complete Bronze/Silver requirement, or a need for a formal support SLA (SDP-META is a Databricks Labs project). See the Introduction for the full positioning.

Components

Metadata Interface

• Onboarding file (JSON or YAML) — sources, targets, CDC config, DQE rules. Examples: demo/conf/json/onboarding.template · demo/conf/yml/onboarding.template.yml
• Data Quality Expectations — per-table JSON or YAML rule files. Examples: demo/conf/json/dqe/customers/ · demo/conf/yml/dqe/customers/
• Silver transformation file — SQL select_exp and where_clause definitions. Examples: demo/conf/json/silver_transformations.json · demo/conf/yml/silver_transformations.yml

Generic Lakeflow Spark Declarative Pipeline

• Reads DataflowSpec at runtime and dynamically wires sources, transformations, expectations, CDC flows, and sinks
• Supports Autoloader, Delta, Kafka, Eventhub, and Snapshot sources
• Applies create_auto_cdc_flow, append_flow, and create_sink based on metadata

High-Level Process Flow

Feature Matrix

Pipeline Capabilities

Deployment & Tooling

Getting Started

SDP-META has three ways to use it — pick the one that matches your role and how far you intend to take it:

1. Declarative Automation Bundle (recommended) — dev/prod targets, git-tracked state, CI/CD-ready. bundle-init --quickstart skips every prompt and gets you a working bundle in one command.
2. Interactive onboard + deploy CLI — kick the tires against a single workspace.
3. SDP-META Databricks App — browser-based UI for non-developers, demos, and click-driven workflows.

See GETTING_STARTED.md for prereqs, install commands, full per-path walkthroughs, local development setup, and troubleshooting. Long form: docs site.

Upgrading from DLT-META

The v0.1.0 release renames DLT-META to SDP-META. Existing onboarding JSON/YAML files and pipeline behavior are unchanged, but new code should move to the new package, CLI, and import path:

The dlt-meta compatibility package remains available during v0.1.x, but it emits deprecation warnings and receives no new features. Legacy src.* imports are planned for removal in v0.2.0.

See the DLT-META → SDP-META migration guide for the step-by-step plan, compatibility details, and deprecation timeline.

Resources

• Documentation
• Video: SDP-Meta Deep-Dive
• FAQ
• Migration Guide
• Release Notes
• GitHub Issues

Project Support

Please note that all projects released under Databricks Labs are provided for your exploration only, and are not formally supported by Databricks with Service Level Agreements (SLAs). They are provided AS-IS and we do not make any guarantees of any kind. Please do not submit a support ticket relating to any issues arising from the use of these projects.

Any issues discovered through the use of this project should be filed as issues on the Github Repo.
They will be reviewed as time permits, but there are no formal SLAs for support.