bricksentry

Change-safety for Databricks pipelines and the AI agents that edit them.

Before a change reaches production, bricksentry answers three questions, in a pull-request comment for humans and in structured JSON for agents:

1. What does it do to the data? A key-based data diff: added, removed and changed rows, schema changes, per-column changes, numeric totals, duplicate keys, and where the change is concentrated ("100% of affected rows are region = APAC").
2. Who gets hurt? The downstream blast radius from Unity Catalog lineage: tables, jobs, notebooks, Lakeflow pipelines, dashboards and queries, with owners and tags.
3. Is it safe, and how do I undo it? A deterministic policy verdict (pass · warn · needs_human · block) and the exact RESTORE TABLE … VERSION AS OF n rollback.

pip install "bricksentry[databricks,mcp]"

Why it exists

AI now writes much of our pipeline code, and checking what that code does to the data hasn't kept up. Code review can't see a silent 3% revenue shift or a join that fans out. The tools that do catch this mostly cover dbt, or need a paid platform. Most Databricks work is PySpark notebooks, Jobs and Lakeflow pipelines, and bricksentry is built for that work:

• Unity Catalog runtime lineage, not SQL parsing, so PySpark notebooks and jobs are included.
• SQL pushdown: all diffing runs on your SQL warehouse, so no Spark session is needed and calls stay cheap enough for an agent loop.
• One engine, three faces: CLI and GitHub Action · Python API · MCP server.

For AI agents: the MCP server

bricksentry mcp --config bricksentry.yml                 # stdio
claude mcp add bricksentry -- bricksentry mcp --config /abs/path/bricksentry.yml

Designed for agents:

• The verdict never comes from an LLM. Rules in policy.yml decide, so the same change always gets the same answer.
• Read-only by default. Only preview_write writes, and only to a sandbox clone. It accepts a single INSERT / MERGE / UPDATE / DELETE / CREATE OR REPLACE whose target must be the clone, and it refuses DROP, TRUNCATE, ALTER, GRANT and multiple statements.
• Errors come back as {"error": ...}, so agents can recover instead of crashing.
• Bounded cost: deterministic hash sampling (sample_pct), capped samples, and lineage limited by depth and lookback window.

The same tools are plain Python functions in bricksentry.agent_tools, so you can use them from LangGraph, the OpenAI Agents SDK or the Databricks Agent Framework without MCP.

See the agent loop (no Databricks needed)

pip install "bricksentry[duckdb]"
python examples/demo/agent_loop_demo.py

In the demo, an agent previews a risky DELETE (needs_human, production untouched), ships a pipeline change with a join fan-out (blocked: 40 duplicate keys), fixes it, and gets routed to the certified table's owners. It ends with the pull-request comment a reviewer would see.

For data engineers: CLI and CI

bricksentry init                                        # starter bricksentry.yml + policy.yml
bricksentry impact main.sales.orders                    # blast radius
bricksentry diff main.sales.orders main.sales_pr_42.orders
bricksentry preview main.sales.orders --sql "MERGE INTO {{target}} t USING updates u ON t.order_id = u.order_id WHEN MATCHED THEN UPDATE SET *"
bricksentry assess --from-git origin/main --pr 42 --format markdown --fail-on needs_human
bricksentry rollback main.sales.orders

Exit codes: 0 below --fail-on, 1 at or above it, 2 for configuration errors.

In GitHub Actions. Build your PR's version of the pipeline into a per-PR schema (for example with a Databricks Asset Bundle ci target), then add:

- run: pip install "bricksentry[databricks]"
- run: >
    bricksentry assess
    --from-git origin/${{ github.base_ref }}
    --pr ${{ github.event.pull_request.number }}
    --comment --output bricksentry-report.md
    --fail-on needs_human
  env:
    GITHUB_TOKEN: ${{ github.token }}
    DATABRICKS_HOST: ${{ secrets.DATABRICKS_HOST }}
    DATABRICKS_CLIENT_ID: ${{ secrets.DATABRICKS_CLIENT_ID }}
    DATABRICKS_CLIENT_SECRET: ${{ secrets.DATABRICKS_CLIENT_SECRET }}
    DATABRICKS_WAREHOUSE_ID: ${{ secrets.DATABRICKS_WAREHOUSE_ID }}

It detects changed tables from the files in the PR and posts or updates one PR comment. Check out with fetch-depth: 0 and give the job pull-requests: write permission. The source repository also ships a composite action.yml that wraps these steps.

Python API

from bricksentry import Sentry

s = Sentry()                                   # reads ./bricksentry.yml
a = s.assess(["main.sales.orders"], pr="42")
print(a.verdict, a.notify)
print(a.tables[0].summary)

s.preview_write("DELETE FROM main.sales.orders WHERE status = 'test'", "main.sales.orders")

Configuration

backend: {type: databricks, warehouse_id: abc123}      # auth: standard Databricks SDK chain
sandbox_schema: main.bricksentry_sandbox
candidate_pattern: "{catalog}.{schema}_pr_{pr}.{table}"
sample_pct: 10
lineage: {type: databricks, depth: 3, lookback_days: 90}
tables:
  main.sales.orders:
    keys: [order_id]
    segment_by: [region]
    paths: ["src/pipelines/orders/**", "resources/orders_job.yml"]
policy: policy.yml

Policy conditions

table (glob) · tag · downstream_tag · downstream_kind · downstream_count_gt · removed_pct_gt · added_pct_gt · changed_pct_gt · row_count_change_pct_gt · row_count_drop_pct_gt · column_removed · column_added · column_type_changed · duplicate_keys · sum_change_pct_gt / null_increase_gt (optionally with column) · any_change · diff_error

All conditions in a when must hold. The most severe matching rule sets the verdict. Unknown conditions are rejected at load time, so a typo can't silently disable a rule.

Permissions needed on Databricks

• CAN USE on a SQL warehouse.
• SELECT on the tables you assess and their candidates.
• SELECT on system.access.table_lineage, plus system.information_schema for owners and tags. Without these, bricksentry still diffs; lineage is just empty.
• CREATE TABLE on sandbox_schema, for preview_write.

Known limits (v0.1)

• Lineage is recorded at runtime. Brand-new tables have no downstream history yet. Unity Catalog doesn't capture RDD operations, and UDFs hide column-level mapping.
• Keys are required. Tables without a stable key can't be diffed row by row yet (an aggregate-profile mode is planned).
• Changed-table detection uses path globs from config. It doesn't parse Asset Bundles yet.
• The Databricks backend is tested against a mocked SDK. Run it against a dev workspace before relying on it in CI.

Roadmap

Asset Bundle parsing for automatic table detection · column-level blast radius from system.access.column_lineage · an aggregate diff for keyless tables · Lakeflow expectations in the verdict · bricksentry watch for scheduled drift checks · a Databricks App UI.

License

Apache-2.0