A source column got renamed. Tonight your Silver notebook fails, tomorrow three Power BI dashboards silently show blanks, and nobody knows why until the CFO asks.

This project fixes that. An autonomous agent watches every layer of a Fabric medallion architecture, walks a column-level lineage graph to find what a change breaks three layers downstream, asks Claude to judge the business impact and draft fixes, opens a Git PR with the mechanical repairs, and alerts your team on Teams, Outlook, and Slack — before the dashboards break.

Runs against a real Fabric tenant (live mode) or a local DuckDB replica of AdventureWorksLT (simulate mode) with zero capacity cost — same code path.

📦 Install

pip install fabric-schema-drift-detective
fabric-drift --help          # console script (also: python -m fabric_drift_detective)

Optional extras per integration:

pip install "fabric-schema-drift-detective[live]"       # Fabric SQL endpoint (pyodbc)
pip install "fabric-schema-drift-detective[hana]"       # SAP HANA direct-connect
pip install "fabric-schema-drift-detective[snowflake]"  # Snowflake direct-connect
pip install "fabric-schema-drift-detective[databricks]" # Databricks / Unity Catalog
pip install "fabric-schema-drift-detective[sqlserver]"  # Azure SQL / SQL Server
pip install "fabric-schema-drift-detective[postgres]"   # PostgreSQL (RDS/Aurora)
pip install "fabric-schema-drift-detective[redshift]"   # AWS Redshift
pip install "fabric-schema-drift-detective[mysql]"      # MySQL / Aurora MySQL
pip install "fabric-schema-drift-detective[cosmos]"     # Azure Cosmos DB

⚡ Quickstart — 60 seconds, no Fabric account

The demo needs the repo (sample data isn't shipped in the wheel):

git clone https://github.com/naveenjujaray/microsoft-fabric-schema-drift-detection.git
cd microsoft-fabric-schema-drift-detection
pip install -e .        # deps + the `fabric-drift` CLI
bash scripts/run_demo.sh

Run the tests and quality gates any time:

pip install -e .[dev]     # pytest, ruff, mypy, bandit + type stubs
pytest -q                 # 330 tests

Loads AdventureWorksLT → builds the medallion → snapshots baselines → injects six kinds of drift → prints the drift report, the PR it would open, and every notification payload. No API keys required (Claude falls back to a deterministic mock; set ANTHROPIC_API_KEY for real reasoning).

🏛️ Architecture

Bronze (Lakehouse)   raw AdventureWorksLT tables, ingested as-is via Dataflow Gen2
      │  drift watch: source columns appearing/disappearing/retyped
      ▼
Silver (Lakehouse)   cleaned, deduped, conformed keys, standardized types
      │  drift watch: transformation contract changes, key/type drift
      ▼
Gold  (Warehouse)    star schema — Dim_Customer, Dim_Product, Dim_Date,
      │              Fact_Sales — consumed by the semantic model
      ▼
Semantic Model       relationships, DAX measures, RLS
      │  drift watch: a Gold column drop/rename that breaks a measure or relationship
      ▼
Power BI reports (PBIP in Git)   final consumers that break silently today

Detection pipeline:

 SchemaBackend (ABC)                 baselines (.baselines/*.json)
 ├─ FabricBackend (fab CLI + REST)          │
 └─ LocalBackend  (DuckDB + JSON)           ▼
        │ current schemas ────────►  schema_diff  ──►  lineage graph  ──►  Claude
        │                            (17 drift types)  (cross-layer +   (severity,
        ▼                                               cross-workspace  fixes, PR text)
   five LayerSchemas                                    breaks)   │
                                                                  ▼
                                    git_handler (branch+PR)   notifications
                                                              (console/Slack/Teams/Outlook)

🔗 The differentiator: cross-layer lineage

silver.customers.email renamed to email_address? The lineage graph walks:

silver:customers.email
  └─► gold:Dim_Customer.Email
        └─► semantic_model:Customer.Email
              └─► reports:Customer Detail.Customer.Email

…and emits a cross_layer_break for each — so the alert says "the Customer Detail report will break", not just "a column changed". DAX measures are parsed for Table[Column] references, so SUM(Sales[Freight]) breaks when silver.sales_orders.freight is dropped, three layers up.

🌐 …and cross-WORKSPACE lineage

Real Fabric estates span workspaces: ingestion in one, the enterprise warehouse in another, reporting in a third — connected by OneLake shortcuts, mirrored databases and semantic-model bindings. Declare that topology in a workspace manifest and the same drift becomes:

Contoso-Ingestion (Workspace A)      silver:customers.email   ← the rename
  └─ OneLake shortcut ─► Contoso-Enterprise-DW (Workspace B)
                           gold:Dim_Customer.Email            cross_workspace_break
                           semantic_model:Customer.Email      cross_workspace_break
                             └─► Contoso-Reporting (Workspace C)
                                   reports:Customer Detail    cross_workspace_break

Impact analysis then reports the workspace name, artifact, workspace path and per-workspace blast radius — and states plainly: "This schema change impacts assets across multiple Microsoft Fabric workspaces." Tenant-boundary crossings are flagged too. Full guide: docs/CROSS_WORKSPACE.md.

🔌 Supported sources

Two integration modes — pick per source:

• Already mirrored/shortcut into Fabric? Use mode: live — the FabricBackend sees it in the lakehouse. No source backend needed.
• Direct-connect upstream (mode: source) — read schema straight from the source system and catch drift before it lands in Fabric: the Bronze-boundary contract.

A new backend is ~100 lines: connection factory + catalog query + type map on top of the shared SQL-catalog base, with cross-source type normalization so NVARCHAR vs STRING never reads as false drift. Per-source setup: docs/BACKENDS.md · claim one: CONTRIBUTING.md.

Lineage is data, not code: declare your own medallion's column mappings in a lineage manifest (lineage.manifest in config.yaml); the AdventureWorksLT demo mappings are just the fallback. Scope the watch with watch.layers / watch.mode: boundaries (contract-enforced layers: only boundary breaks surface).

📖 Example story — the rename that would have reached the CFO

9:02 AM. An upstream engineer renames a column in the CRM export: email → email_address. Nobody tells the BI team. The nightly Dataflow succeeds. No error anywhere.

Here is what happens with this project running as a scheduled Fabric pipeline:

1 — Detect. The 2 AM run snapshots every layer and diffs against the stored baselines. schema_diff sees email gone and email_address arrived in silver.customers — same type, same position, near-identical name — and classifies it as a column_rename (not a scarier drop+add):

[CRITICAL] silver:customers.email column_rename ('email' -> 'email_address')

2 — Walk the lineage. The column-level graph follows the rename downstream and synthesizes a cross_layer_break at every stop:

silver:customers.email
  └─► gold:Dim_Customer.Email
        └─► semantic_model:Customer.Email
              └─► reports:Customer Detail.Customer.Email   ← the dashboard the CFO opens

So the alert reads "the Customer Detail report will break" — not just "a column changed somewhere."

3 — Reason (Claude). Claude judges business impact, confirms the rename is mechanically fixable, and drafts the exact TMDL find/replace plus PR prose. (No API key? A deterministic mock does the same shape offline.)

4 — Fix. git_handler cuts a branch drift-fix/<timestamp>, rewrites sourceColumn: email → email_address in the PBIP TMDL, and opens a PR — never touching main.

5 — Alert. One DriftAlert fans out to Teams / Outlook / Slack: severity counts, the three worst drifts, the blast radius, and the PR link.

6 — Gate. The pipeline fails the run (exit 1) on critical drift, so drift is alertable like any other pipeline failure — and CI blocks the merge until it's resolved.

9:05 AM. Instead of blank dashboards next week, the BI team has a PR waiting, a Teams card explaining what and why, and a red pipeline that already stopped the bad state from shipping.

Want to interrogate it yourself? Ask an agent:

fabric-drift --agent lineage_qa --task "what breaks if silver.customers.email is renamed?"
fabric-drift --agent triage        # rank everything currently drifting, P1→P3

Run the whole story locally in 60 seconds with bash scripts/run_demo.sh — it injects this rename (plus a money-truncating precision_scale_change, a dropped column, a type change, and more) and prints every step above.

🔬 Drift types

Seventeen typed drifts, grouped by the level they hit. Severity is judged by whether a change breaks consumers or just risks them; auto_fixable means the repair is mechanical (a downstream find/replace), not a business decision.

Column-level

Table-level

Semantic-model-level

Cross-layer / cross-workspace

Whitespace-only DAX reformatting (Power BI Desktop rewrites TMDL on every save) is normalized away, so measure_change fires on real logic edits — not noise.

Real-world drift the model is designed for

The catalog maps directly to how enterprise Fabric estates actually break: upstream source-system schema evolution (SAP/Salesforce/Dynamics adding, retyping, renaming fields), Dataflow Gen2 / notebook refactors silently changing the Silver contract, money-precision narrowing in financial marts, and semantic-model edits that move numbers without breaking a query. Drifts that need metadata the backends don't yet capture — Delta partition-column changes, collation/case-sensitivity — are the tracked roadmap; the engine's typed, lineage-aware design extends to them without touching consumers. (default_change/flag_change fire wherever a backend captures defaults/flags — the simulate backend does; SQL backends opt in by widening their catalog query.)

🚀 Usage

Installed via pip, the CLI is fabric-drift (equivalently python -m fabric_drift_detective):

fabric-drift --mode simulate --baseline        # capture baseline snapshots
fabric-drift --mode simulate --once            # one detection cycle
fabric-drift --mode simulate --once --dry-run  # render all payloads, send nothing
fabric-drift --mode live --once --open-pr      # real Fabric + real PR
fabric-drift --mode snowflake --once           # direct-connect upstream source
fabric-drift --provision                       # show fab provisioning steps

Exit codes: 0 clean · 1 critical drift (usable as a CI gate) · 2 config error · 3 missing/corrupt baselines.

Baselines are never recreated implicitly. If baseline files vanish or corrupt, the run fails with exit code 3 and tells you to re-capture with --baseline — silent recreation would hide exactly the changes a drift detector exists to catch.

Everything is configured in config.yaml (IDs, model, channels) + .env (secrets — see .env.example). No hardcoded IDs anywhere.

🤖 Agents — ten tool-use specialists

Beyond the scheduled pipeline, ten Claude agents run tool-use loops over the same backends, differ and lineage graph — for interactive investigation, verified repair and operations:

fabric-drift --list-agents
fabric-drift --agent lineage_qa --task "what breaks if silver.sales_orders.freight is dropped?"
fabric-drift --agent fix_verify --allow-writes    # propose → apply → re-diff → retry

Production guard rails: write tools hard-gated behind --allow-writes (✏️ agents plan-only otherwise) · sandboxed file/SQL access (SELECT-only, row-capped, .env/.git denied) · per-run turn + token budgets · JSONL transcript of every run in .agent_runs/ · no key? clean offline result, never a crash. Full guide: docs/AGENTS.md.

🏭 Run it inside Fabric — notebook, pipeline, CI/CD

The fabric/ folder ships Fabric-native artifacts (Git-integration format):

FABRIC_WORKSPACE_ID=<guid> bash scripts/deploy_fabric.sh   # one-command deploy

Auth adapts automatically — four methods behind one credential (src/azure_auth.py):

Full guide: docs/FABRIC_NATIVE.md.

📣 Notifications

One DriftAlert object, three formatters — content logic never duplicated:

Each channel independently toggleable; a failing channel never blocks the others; --dry-run prints every payload. Teams/Outlook Graph calls reuse the same credential as Fabric — one app registration, one auth stack (permissions: ChannelMessage.Send, Mail.Send — see docs/FABRIC_SETUP.md).

📚 Docs

✅ Tests & quality gates

pytest                    # 330 tests: differ (17 drift types), deterministic
                          # rename matching, cross-workspace lineage, baseline
                          # fail-loud policy, REST retry/backoff, git handler
                          # security guards, agents, notifications, lineage
                          # manifest, watch scope, type normalization, and the
                          # backend contract suite (Local/HANA/Snowflake/
                          # Databricks/SQL Server/Postgres/Redshift/MySQL/
                          # Cosmos DB)
ruff check .              # lint (incl. bugbear + security rules)
mypy                      # strict-leaning type check, 0 errors
bandit -c pyproject.toml -r src main.py   # security scan

CI (.github/workflows/ci.yml) runs all four on every push/PR across Python 3.10 and 3.12, plus pip-audit for dependency vulnerabilities, with a 75% coverage floor.

📄 License

MIT License

Copyright (c) 2026 Naveen Jujaray

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Made with ❤️ by Naveen Jujaray