databricks-schema

A CLI tool and Python library that uses the Databricks SDK to extract and diff Unity Catalog schemas in YAML format. It can also generate Databricks Spark SQL to apply schema changes across catalogs.

Getting Started

Install from PyPI (requires Python 3.11+):

pip install databricks-schema

Configure your Databricks credentials (see Authentication for all supported methods).

Then extract, diff, and generate SQL for your Unity Catalog schemas:

# 1. Find the catalog you want to snapshot
databricks-schema list-catalogs

# 2. Extract its schemas to YAML files (one file per schema)
databricks-schema extract prod_catalog --output-dir ./schemas/

# 3. Diff those files against a catalog (same or different)
databricks-schema diff test_catalog ./schemas/

# 4. Generate SQL to bring that catalog in line with the YAML files
databricks-schema generate-sql test_catalog ./schemas/ --output-dir ./migrations/

Databricks Authentication

The tool uses the Databricks SDK for auth. Configure it via environment variables:

export DATABRICKS_HOST=https://<databricks-url>
export DATABRICKS_TOKEN=<your-personal-access-token>

Or use a Databricks CLI profile (~/.databrickscfg) — the SDK will pick it up automatically.

You can also pass credentials directly as flags using --host / --token below.

Development

Requires Python 3.11+ and uv.

git clone https://github.com/frizzleqq/databricks-schema
cd databricks-schema
uv sync --all-groups  # includes pytest and ruff

# Run tests
uv run pytest

# Lint
uv run ruff check databricks_schema/ tests/

# Format
uv run ruff format databricks_schema/ tests/

Agent Skill

This repo ships an Agent Skill at skills/databricks-schema-cli/SKILL.md that teaches an AI coding agent (e.g. Claude Code) how to use the databricks-schema CLI to explore, snapshot, diff, and generate migration SQL for a live Unity Catalog. Copy the skill into your own project and it can use the CLI directly instead of writing ad-hoc Databricks SDK calls or queries.

The skill assumes the databricks-schema command is on PATH. Install it as a standalone tool with uv:

uv tool install databricks-schema

To use the skill in your own project without cloning this repo, download it directly into your project's .claude/skills/:

mkdir -p .claude/skills/databricks-schema-cli
curl -o .claude/skills/databricks-schema-cli/SKILL.md \
  https://raw.githubusercontent.com/frizzleqq/databricks-schema/main/skills/databricks-schema-cli/SKILL.md

For agentic harnesses other than Claude Code that look under .agents/skills/ instead:

mkdir -p .agents/skills/databricks-schema-cli
curl -o .agents/skills/databricks-schema-cli/SKILL.md \
  https://raw.githubusercontent.com/frizzleqq/databricks-schema/main/skills/databricks-schema-cli/SKILL.md

Output Format

Each schema is written to {output-dir}/{schema-name}.yaml if --output-dir is specified. Fields with no value (null comments, empty tag dicts, empty FK lists) are omitted. Use --format json to write .json files with the same structure.

name: main
comment: Main production schema
tags:
  env: prod
tables:
  - name: users
    table_type: MANAGED
    comment: User accounts
    tags:
      domain: identity
    columns:
      - name: id
        data_type: bigint
        nullable: false
        comment: Primary key
      - name: email
        data_type: string
      - name: org_id
        data_type: bigint
    primary_key:
      name: pk_users
      columns:
        - id
    foreign_keys:
      - name: fk_org
        columns:
          - org_id
        ref_schema: orgs
        ref_table: organizations
        ref_columns:
          - id

Without --output-dir, extract prints a single Catalog document to stdout instead — the same schema shape nested under a schemas list, with the catalog name as a top-level field:

name: prod_catalog
schemas:
  - name: main
    comment: Main production schema
    tags:
      env: prod
    tables: [...]

CLI Usage

databricks-schema [OPTIONS] COMMAND [ARGS]...

extract, diff, generate-sql, and diff-files print progress messages (e.g. Extracting catalog 'X'...) to stderr as they run. Pass --quiet / -q to suppress them; results and errors are unaffected.

list-catalogs

List all accessible catalogs:

databricks-schema list-catalogs

list-schemas

List schemas in a catalog:

databricks-schema list-schemas <catalog>

Both print one name per line by default. Pass --format json / -f json (or --format yaml) to get the names as a JSON array or YAML list instead.

extract

Extract all schemas from a catalog to YAML files:

databricks-schema extract <catalog> --output-dir ./schemas/

Print to stdout instead of writing files (no --output-dir) — the catalog and its schemas are printed as a single Catalog document (name + schemas: [...]):

databricks-schema extract <catalog>

Extract specific schemas only:

databricks-schema extract <catalog> --schema main --schema raw --output-dir ./schemas/

Or extract a single schema or table by naming it directly, dotted onto the catalog (output keeps the same Catalog/Schema shape, just filtered down). This cannot be combined with --schema:

databricks-schema extract <catalog>.<schema>
databricks-schema extract <catalog>.<schema>.<table>

Include Unity Catalog tags in the output (extra API calls per entity; excluded by default):

databricks-schema extract <catalog> --output-dir ./schemas/ --include-tags

Include additional metadata (owner, storage_location) in the output:

databricks-schema extract <catalog> --output-dir ./schemas/ --include-metadata

Control the number of parallel workers (default: 4), useful for large schemas with many tags:

databricks-schema extract <catalog> --output-dir ./schemas/ --workers 8

diff

Compare the live catalog against previously extracted schema files (format auto-detected from the directory — YAML or JSON, not mixed):

databricks-schema diff <catalog> ./schemas/

Exits with code 0 if no differences are found, 1 if there are — making it suitable for CI pipelines. Output example:

~ Schema: main [MODIFIED]
  ~ Table: users [MODIFIED]
    ~ Column: score [MODIFIED]
        data_type: 'int' -> 'double'
    + Column: phone [ADDED]
  + Table: events [ADDED]
- Schema: legacy [REMOVED]

Markers: + added in catalog, - removed from catalog, ~ modified.

Or compare two live catalogs directly, e.g. dev against prod. The second argument is treated as a directory if one exists at that path, otherwise as a catalog name:

databricks-schema diff dev_catalog prod_catalog

Compare specific schemas only:

databricks-schema diff <catalog> ./schemas/ --schema main --schema raw

Include Unity Catalog tags in the comparison (excluded by default):

databricks-schema diff <catalog> ./schemas/ --include-tags

Include owner in the comparison (excluded by default):

databricks-schema diff <catalog> ./schemas/ --include-metadata

Get the same comparison as structured JSON or YAML instead of the +/-/~ tree (exit codes unchanged) — useful for scripts or agents:

databricks-schema diff <catalog> ./schemas/ --format json
databricks-schema diff <catalog> ./schemas/ --format yaml

generate-sql

Generate Databricks Spark SQL statements to bring the live catalog in line with local schema files (format auto-detected, YAML or JSON, not mixed). Statements are printed to stdout by default:

databricks-schema generate-sql <catalog> ./schemas/

Write one .sql file per schema to a directory instead:

databricks-schema generate-sql <catalog> ./schemas/ --output-dir ./migrations/

Destructive statements (DROP SCHEMA, DROP TABLE, DROP COLUMN) are emitted as SQL comments by default. Pass --allow-drop to emit them as executable statements:

databricks-schema generate-sql <catalog> ./schemas/ --allow-drop

Filter to specific schemas:

databricks-schema generate-sql <catalog> ./schemas/ --schema main --schema raw

Include Unity Catalog tags in the comparison (excluded by default):

databricks-schema generate-sql <catalog> ./schemas/ --include-tags

Include owner in the comparison (excluded by default):

databricks-schema generate-sql <catalog> ./schemas/ --include-metadata

validate

Validate local schema files for structural integrity (no Databricks connection needed):

databricks-schema validate ./schemas/

Pass --format json or --format yaml for an {"issues": [...]} document instead of OK/ERROR: text.

diff-files

Compare two local directories of schema files (no Databricks connection needed). Same output, exit codes, --format json/yaml, and --schema / --include-metadata flags as diff:

databricks-schema diff-files ./schemas-prod/ ./schemas-test/

json-schema

Print the JSON Schema for one of the CLI's structured output shapes — no Databricks connection needed. Useful before parsing --format json/yaml output (or extract's output) programmatically. Add --format yaml to render the schema itself as YAML:

databricks-schema json-schema catalog    # extract's stdout Catalog document
databricks-schema json-schema schema     # one per-schema extract file
databricks-schema json-schema diff       # diff / diff-files --format json/yaml
databricks-schema json-schema validate   # validate --format json/yaml

Python Library Usage

See also the examples/ directory for standalone scripts.

from pathlib import Path
from databricks_schema import CatalogExtractor, catalog_to_yaml, schema_from_yaml
from databricks_schema import diff_catalog_with_dir, diff_catalogs, diff_schemas, schema_diff_to_sql

# Extract using configured auth (max_workers controls parallel table extraction)
extractor = CatalogExtractor(max_workers=4)
catalog = extractor.extract_catalog("my_catalog", schema_filter=["main", "raw"])

# Include Unity Catalog tags (extra API calls per entity; excluded by default)
catalog = extractor.extract_catalog("my_catalog", include_tags=True)

# Include additional metadata (owner, storage_location)
catalog = extractor.extract_catalog("my_catalog", include_metadata=True)

# Serialise to YAML
yaml_text = catalog_to_yaml(catalog)

# Deserialise from YAML
schema = schema_from_yaml(open("schemas/main.yaml").read())
print(schema.tables[0].columns)

# Compare live catalog against local YAML files
result = diff_catalog_with_dir(catalog, Path("./schemas/"))
if result.has_changes:
    for schema_diff in result.schemas:
        print(schema_diff.name, schema_diff.status)

# Compare two live catalogs directly
prod_catalog = extractor.extract_catalog("prod_catalog")
result = diff_catalogs(live=catalog, stored=prod_catalog)

# Compare two Schema objects directly
stored = schema_from_yaml(open("schemas/main.yaml").read())
diff = diff_schemas(live=catalog.schemas[0], stored=stored)

# Generate SQL to bring live in line with stored
sql = schema_diff_to_sql("my_catalog", diff, stored_schema=stored, allow_drop=False)
print(sql)