dbx-cost-guardian 0.1.0

Shift-left FinOps for Databricks: cluster linting, cost estimation, and CI policy gates

dbx-cost-guardian

Shift-left FinOps for Databricks: catch the expensive mistake in the PR, not the invoice.

---

dbx-cost-guardian is a Python package that embeds Databricks cost awareness directly into your development workflow. It statically analyses cluster configs, job definitions, and PySpark source code to catch cost anti-patterns — no live Databricks connection required.

Features

• Cluster Config Linting — 6 rules (DBX001–DBX006) covering compute type, autoscaling, spot instances, runtime version, SQL workloads, and driver sizing
• PySpark Code Analysis — AST-based scanning for .toPandas(), .collect(), Python UDFs, and Pandas import anti-patterns
• Cost Estimation — DBU-based cost projection using configurable price tables (no API key needed)
• Policy Gates — Configurable pass/fail thresholds for CI/CD integration
• CLI + Python API — Use from the command line or embed in notebooks and test suites
• JSON Reports — Machine-readable output for dashboards and automation

Installation

pip install dbx-cost-guardian

For development:

pip install dbx-cost-guardian[dev]

Quick Start

CLI

# Scan cluster configs and source files
dbx-cost-guardian scan cluster.json notebook.py

# Scan entire directories
dbx-cost-guardian scan configs/ src/ notebooks/

# With custom config and JSON output
dbx-cost-guardian scan --config guardian.yml --json-out report.json configs/

# List available lint rules
dbx-cost-guardian scan --list-rules

# Strict mode (fail on warnings too)
dbx-cost-guardian scan --strict configs/

Python API

from dbx_cost_guardian import scan

result = scan(
    cluster_jsons=["configs/production.json"],
    source_files=["src/etl_pipeline.py"],
)

# Check policy outcome
if not result.policy_result.passed:
    print("Cost policy violated!")
    for msg in result.policy_result.violated_thresholds:
        print(f"  • {msg}")

# Inspect violations
for v in result.all_violations():
    print(f"[{v.severity.value}] {v.rule_id}: {v.message}")

# Check cost estimates
for e in result.cost_estimates:
    print(f"{e.source_file}: ${e.estimated_run_usd:.2f}/run, ${e.estimated_month_usd:.2f}/month")

Configuration

Create a guardian.yml at your repo root (see examples/guardian.yml for all options):

# Override DBU pricing for your contract
dbu_prices:
  jobs: 0.10
  all_purpose: 0.40

# Assume 2-hour runtime for cost estimation
assumed_runtime_hours: 2.0

# Skip rules that don't apply
skip_rules:
  - DBX001

# Policy thresholds
policy:
  max_estimated_cost_usd_per_run: 100.0
  max_estimated_cost_usd_per_month: 10000.0
  fail_on_error: true
  max_violations:
    error: 0
    warning: 20

CI/CD Integration

GitHub Actions

- name: Run cost scan
  run: |
    pip install dbx-cost-guardian
    dbx-cost-guardian scan \
      --config guardian.yml \
      --json-out cost-report.json \
      configs/ src/ notebooks/

See examples/github-actions.yml for a complete workflow.

Pre-commit Hook

repos:
  - repo: local
    hooks:
      - id: dbx-cost-guardian
        name: Databricks Cost Guardian
        entry: dbx-cost-guardian scan
        language: system
        types_or: [json, python]
        pass_filenames: true

Rules Reference

Rule ID	Severity	Description
DBX001	ERROR	All-Purpose Compute used for jobs (4× cost)
DBX002	WARNING	Missing or unbounded autoscaling
DBX003	WARNING	No spot/preemptible instances
DBX004	WARNING	Outdated Databricks runtime
DBX005	WARNING	SQL workload on Spark cluster
DBX006	WARNING	Oversized driver node
AST001	ERROR	.toPandas() — collects full DataFrame to driver
AST002	WARNING	.collect() — brings all rows to driver
AST003	INFO	.toDF() — Pandas to Spark conversion
AST010	WARNING	Pandas import in Spark context
AST020	WARNING	Plain Python UDF (row-by-row serialisation)

Full details in docs/rules.md.

Exit Codes

Code	Meaning
0	All policy gates passed
1	Policy violation detected

Non-Goals

• Real-time cluster monitoring (use Databricks built-in or Datadog)
• Actual spend ingestion (no API calls in MVP)
• Auto-fixing configs (lint-and-report only)

Roadmap

Version	Feature
v0.2	Databricks Asset Bundle (DAB) YAML support
v0.2	Delta Live Tables cost rules
v0.3	Historical runtime from Databricks Jobs API
v0.3	GitHub PR comment reporter
v0.4	VSCode extension
v1.0	Plugin API for custom rules

License

MIT