sofe-dbt

ESLint for your data pipelines. FinOps policies as code for dbt projects — with the real Athena bill attached to every model.

sofe-dbt reads your dbt artifacts (manifest.json, run_results.json, catalog.json) plus real Athena query statistics, and evaluates FinOps policies against your models. It's built on the SOFE engine — SOFE is ESLint for your cloud; sofe-dbt is ESLint for your data pipelines.

$ sofe-dbt check ./
✓ partition-required ...... 42/42 models
⚠ zombie-model ........... stg_legacy_orders (0 consumers)
✗ full-refresh-in-prod ... fct_events (scans 1.2TB daily)
✗ freshness-test-missing . src_salesforce

# 2 errors → PR blocked. Estimated savings: -$380/mo if fixed

Why

Data engineers using dbt know what their models do, whether they pass tests, and whether freshness is met — but they have zero visibility into cost efficiency: how much each model costs, whether materialization is optimal, which models are zombies, and which large tables scan full data on every query.

sofe-dbt closes that gap. Every dollar shown is derived from real Athena DataScannedInBytes — never a heuristic, never invented.

Install

pip install sofe-dbt

Quick start

# 1. Generate dbt artifacts (writes real Athena scan stats into run_results.json)
dbt build

# 2. Evaluate FinOps policies with real cost per model
sofe-dbt check .

# 3. Gate a PR in CI (exit 1 on high-severity findings)
sofe-dbt check . --fail-on high

Live Athena cost enrichment (optional, needs AWS creds):

sofe-dbt check . --profile my-aws-profile

The 10 policies

Real cost, not estimates

Cost per model comes from real Athena bytes scanned:

• Offline (CI): dbt-athena writes data_scanned_in_bytes into run_results.json at dbt build time.
• Live: Athena GetQueryExecution → QueryExecutionStatistics.DataScannedInBytes.

cost_per_run = max(bytes_scanned, 10 MB) / 1e12 × $5.00/TB   (us-east-1 on-demand)

If real bytes aren't available, sofe-dbt reports cost: unknown — it does not guess.

Reproducible demo

See GUIDE.md for an end-to-end walkthrough: deploy the AWS stack (CDK: S3 + Glue + Athena), load golden data, run dbt build against real Athena, and see sofe-dbt check surface real cost per model. The whole demo costs cents and tears down with cdk destroy.

Setup (development)

uv venv --python 3.12 .venv
source .venv/bin/activate
uv pip install -e ".[dev]"
pytest

License

Apache-2.0