dbt-meta 0.2.3

AI-first CLI for dbt metadata extraction - Fast, modern Python tool

dbt-meta

⚡ AI-first CLI for dbt metadata extraction

dbt-meta is a lightning-fast command-line tool that extracts metadata from dbt's artifacts for DEs and AI agents, eliminating the need to parse .sql files or query your data warehouse for schema information. This is especially useful for fast and accurate agent operation, for example, Claude Code.

✨ Features

• 🎯 Works out-of-box — Simple Mode: just run dbt compile and start using
• ⚙️ TOML configuration — Modern config files with XDG compliance (optional)
• ⚡ Lightning fast — Optimized Python with LRU caching and orjson parser
• 🔄 Production Mode — Full defer workflow support for multi-project setups and development environment
• 📊 AI-friendly JSON — Machine-readable structured output (-j flag) on every command
• 🔍 Rich metadata — Schema, columns, dependencies, config, compiled SQL, column docs
• 🌳 Dependency navigation — Trace upstream/downstream models (flat or tree view)
• 🔎 Advanced filtering — Filter models by tags, config, path, package with OR/AND logic
• 🔀 Git-aware — Find modified models and dependencies needing --full-refresh
• 📋 Smart search — Find models by name or description
• ✅ SQL validation — Validate SQL syntax using BigQuery dry run
• 📏 Scan estimation — Estimate query scan size before running (MB / GB)
• 📊 Optimization analysis — Find hotspots, analyze single models, branch-level alignment
• 🔗 Power BI integration — Extract BigQuery tables (+ DAX measures & column schemas) from dashboards
• 🔁 Artifact sync — meta refresh pulls prod artifacts or parses local project
• 🎨 Beautiful UI — Rich terminal formatting with categorized help panels
• ⚡ Combined flags — Use -dj, -adj, -mf for faster typing (order-independent)

🤖 Built for AI Workflows

dbt-meta was specifically designed to eliminate AI agent hallucinations when working with dbt projects.

The Problem

AI agents (like Claude Code, GitHub Copilot, ChatGPT) often hallucinate when working with dbt:

• ❌ Wrong table names - Confusing alias vs filename (customers vs dim_customers)
• ❌ Wrong schema names - Confusing prod and dev schemas
• ❌ Unknown dependencies - Missing refs/sources in lineage
• ❌ Incorrect column types - Using wrong data types in WHERE clauses
• ❌ Non-existent fields - Querying columns that don't exist

The Solution

Following Anthropic's recommendation to use CLI tools over MCP for AI agents, dbt-meta provides:

• ✅ Fast - Optimized Python with caching, no repeated manifest parsing
• ✅ Machine-readable JSON - Every command has -j flag for structured output, no text parsing needed
• ✅ Schema validation - Prevents hallucinations by providing accurate metadata
• ✅ Type-safe - Mypy strict mode, comprehensive test coverage (91%+)
• ✅ 3-level fallback - Production manifest → Dev manifest → your database (always finds metadata)
• ✅ Git-aware - Auto-detects model state (modified, new, deleted) with helpful warnings

Integration

dbt-meta integrates seamlessly with:

• Claude Code (Anthropic) - Add to allowed commands in .claude/settings.local.json
• GitHub Copilot - Use in terminal and inline suggestions
• ChatGPT / Custom GPTs - Execute commands and parse JSON output
• Other AI agents - Standard CLI interface with JSON output

Why CLI over MCP?

• Have deterministic, structured output
• Are faster and more reliable
• Work in any environment
• Don't require additional infrastructure

Performance

dbt-meta uses several optimization techniques:

• LRU Caching: ManifestParser cached with @lru_cache(maxsize=1)
• orjson: Fast JSON parsing (2-3x faster than standard json)
• Lazy loading: Manifest parsed only when needed
• Catalog fallback: Use catalog.json instead of BigQuery queries

Measured performance (~900 models manifest):

Command	Time	Notes
meta schema	~250ms	Manifest only
meta info	~335ms	Manifest only
meta parents --all	~300ms	Traversed 295 ancestors
meta columns (catalog)	~50ms	With fresh catalog.json
meta columns (BigQuery via bq CLI)	~2-3s	Fallback when catalog stale

Tip: Keep catalog.json fresh (prod state) for fastest columns performance. But this only works for unmodified columns. For models built using defer in dev schema, column metadata is only in DWH.

📦 Installation

PyPI Installation (Recommended)

# Install from PyPI (when published)
pip install dbt-meta

# Verify installation
dbt-meta --version
# or use shorter alias
meta --version

Development Installation

# Clone repository
git clone https://github.com/Filianin/dbt-meta.git
cd dbt-meta

# Install in development mode
pip install -e .

# Or install with dev dependencies
pip install -e ".[dev]"

# Verify installation
meta --version

Requirements

• Python 3.9+ (3.12+ recommended for best performance)
• dbt project with manifest.json
• Optional: jq for advanced JSON processing

🚀 Quick Start

Simple Mode (no configuration)

# Step 1: Compile your dbt project
dbt compile

# Step 2: Use dbt-meta immediately!
meta schema customers           # → your_project.analytics.customers
meta columns -j orders          # → JSON array of columns
meta deps customers             # → Dependencies list
meta list tag:daily             # → Filter models by tag
meta search "customer"          # → Find models by description

# Get comprehensive help with examples
meta --help
meta

Production Mode (with defer workflow)

To organize a dev environment, you need to have the current version of the prod manifest.json, which will be regularly updated to the latest state. For example, you can regularly compile your manifest and upload it to some cloud storage. From there, you can download this file to your machine in any way you like. If you generate documentation and upload it as a static website, then catalog.json is generated as part of this process, which is recommended to be uploaded along with the manifest. This file contains data about columns and data types that are missing from the manifest.

# One-time setup: Create config file
meta settings init

# Edit ~/.config/dbt-meta/config.toml:
prod_manifest_path = "~/dbt-state/manifest.json"
dev_schema = "personal_myname"

# Now works from any directory!
cd /tmp && meta schema customers  # → Uses production manifest

# For dev models (after defer run):
defer run --select customers
meta schema --dev customers      # → personal_myname.customers
meta columns -dj customers       # → Dev columns with JSON output

Combined Flags (faster typing)

meta schema -dj customers                    # → Dev + JSON
meta parents -ajd model                      # → All ancestors + JSON + Dev
meta columns -j --manifest ~/path.json m     # → JSON + Custom manifest

📚 Commands Reference

All commands accept -h/--help for detailed per-command help.

Core Metadata

Command	Description	Key flags	Example
info <model>	Model summary (name, schema, table, materialization, tags)	-j, -d	meta info -j customers
schema <model>	Full table name (database.schema.table)	-j, -d	meta schema customers
path <model>	Relative file path to .sql file	-j, -d	meta path customers
columns <model>	Column names and types	-j, -d	meta columns -dj customers
config <model>	Full dbt config (partition_by, cluster_by, incremental, etc.)	-j, -d	meta config -j customers
sql <model>	Compiled SQL (default) or raw with --jinja	-j, -d, --jinja	meta sql --jinja customers
docs <model>	Column names, types, and descriptions	-j, -d	meta docs customers
deps <model>	Dependencies by type (refs, sources, macros)	-j, -d	meta deps -j customers

Lineage

Command	Description	Key flags	Example
parents <model>	Upstream dependencies (direct or all ancestors)	-j, -d, -a	meta parents -aj customers
children <model>	Downstream dependencies (direct or all descendants)	-j, -d, -a	meta children -a customers

• Without -a: direct parents/children only (classic format).
• With -a -j and ≤20 nodes: nested JSON with children key; otherwise flat array.

Discovery & Filtering

Command	Description	Key flags	Example
list [selectors…]	Filter with selectors (tag:, config., path:, package:)	-j, -d, -m, -f, -a, --and, --group	meta list tag:daily --and tag:core
models [pattern]	Simple substring search over model names	-j	meta models staging
search <query>	Search models by name or description	-j	meta search "customer" -j

list selectors:

• tag:name — filter by tag (OR logic by default)
• config.key:value — filter by config value (e.g. config.materialized:incremental)
• path:dir/ — filter by file path prefix
• package:name — filter by package

list flags:

• --and — require ALL selectors to match (default: OR)
• --group — group output by tag combinations (headers)
• -m, --modified — git-aware: only changed/new models
• -f, --full-refresh — models needing --full-refresh (modified + downstream + intermediate models on paths between them)
• -a, --all — only with -f: tree view showing lineage from modified models down

SQL Validation & Cost

Uses BigQuery dry run (no rows processed, no charge for dry run itself). Both commands fall back through manifest → target/compiled/ → auto dbt compile (with --dev) — see details below.

Command	Description	Key flags	Example
validate <model>	Validate compiled SQL syntax	-j, -d	meta validate customers
scan <model>	Estimate query scan size (🟢 <1 GB, 🟡 1–10 GB, 🔴 ≥10 GB)	-j, -d	meta scan --dev -j customers

Optimization (requires dbt_bigquery_monitoring)

Command	Description	Key flags	Example
hotspots	Rank all tables by optimization score (cost, partitioning, clustering, cache)	-j, -n/--limit, --min-gb	meta hotspots -n 20 --min-gb 10
analyze <model>	Deep analysis of a single model — storage, usage, recommendations	-j	meta analyze -j customers
branch <model>	Branch-level analysis — upstream/downstream partitioning alignment	-j	meta branch customers

hotspots flags:

• -n, --limit N — number of hotspots to display (default: 10)
• --min-gb GB — minimum table size in GB (default: 1.0)

Integration — Power BI

Requires Azure AD Service Principal with Power BI Admin API access.

Command	Description	Key flags	Example
powerbi [workspace_id]	Power BI → BigQuery → dbt model mapping	-j, --by-table, --measures, --columns, --full	meta powerbi --by-table -j

powerbi flags:

• --measures — include DAX measure expressions
• --columns — include column schemas with data types
• --full — include all metadata (measures + columns)
• --by-table — aggregated view grouped by BigQuery table

Artifacts & Settings

Command	Description	Key flags	Example
refresh	Sync production artifacts from remote storage	-d/--dev (parse local via dbt parse)	meta refresh
settings init	Create config file from template	-f/--force (overwrite existing)	meta settings init
settings show	Display merged configuration (TOML + env)	-j	meta settings show -j
settings validate	Validate active config file	—	meta settings validate
settings path	Show active config file path	—	meta settings path

Global Flags

Flag	Description	Applies to
-h, --help	Show help	All commands
-v, --version	Show version and exit	Main app
--manifest PATH	Explicit path to manifest.json (takes precedence over --dev)	All metadata commands
-d, --dev	Use dev manifest/schema (./target/manifest.json, personal_USERNAME)	Most metadata commands
-j, --json	Output as JSON (AI-friendly structured data)	Most commands

Combined short flags work in any order: -dj, -adj, -mf, -fa, etc.

Note: --manifest has no short form. If --manifest and --dev are both provided, --dev is ignored with a warning.

💡 Common Use Cases

Querying BigQuery with Correct Table Names

# Get production table name (eliminates AI hallucinations)
TABLE=$(meta schema customers)
bq query "SELECT * FROM $TABLE LIMIT 10"
# → SELECT * FROM your_project.analytics.dim_customers LIMIT 10

# Or with JSON output
TABLE=$(meta schema -j customers | jq -r '.full_name')
bq query "SELECT * FROM $TABLE LIMIT 10"

Finding All Columns for a Model

# Get column list for WHERE clauses
meta columns -j orders | jq -r '.[] | .name'
# → order_id, customer_id, order_date, status, amount

# Get column types for schema validation
meta columns -j orders | jq -r '.[] | "\(.name): \(.data_type)"'
# → order_id: INTEGER, customer_id: INTEGER, order_date: DATE, ...

Analyzing Dependencies

# Get all upstream models (for CI/CD impact analysis)
meta parents -aj customers | jq -r '.[] | .path'
# → staging/customers.sql, staging/orders.sql, staging/payments.sql

# Find downstream impact of model changes
meta children -a customers
# → Shows all models that depend on customers

Working with Dev Models

# Build dev model
defer run --select customers

# Query dev table (not production)
TABLE=$(meta schema --dev customers)
bq query "SELECT * FROM $TABLE LIMIT 10"
# → SELECT * FROM personal_USERNAME.customers LIMIT 10

Validating SQL and Estimating Scan Size

# Validate SQL syntax before running (uses BigQuery dry run)
meta validate customers
# → ✅ Valid

# Check for syntax errors
meta validate broken_model
# → ❌ Error: Unrecognized name: unknown_column at [3:5]

# Estimate query scan size
meta scan customers
# → Scan size: 3.2 GB

# JSON output for scripting
meta scan -j customers | jq -r '.formatted'
# → 3.2 GB

# Use in CI/CD to prevent expensive queries
BYTES=$(meta scan -j customers | jq -r '.bytes')
if [ "$BYTES" -gt 10000000000 ]; then  # 10 GB limit
  echo "Query too expensive: $BYTES bytes"
  exit 1
fi

# With local changes: --dev auto-compiles if needed
meta validate --dev my_model
# → ℹ️  No compiled SQL for 'my_model'. Running `dbt compile --select my_model --target dev`...
# → ✅ Valid

Compiled SQL lookup (validate / scan): 3-level fallback strategy:

1. model['compiled_code'] from manifest (always works after dbt compile/dbt run)
2. target/compiled/{package}/{original_file_path} on disk (works if you ran dbt compile separately)
3. With --dev: auto-runs dbt compile --select <model> --target dev and re-reads from disk

The third level only fires when --dev is set, since auto-compilation is only safe in your local project. If dbt isn't on PATH, compilation fails, or the project root can't be found, you get a clear error with a suggested manual command.

Refreshing Artifacts

# Sync production manifest.json + catalog.json from remote storage
meta refresh
# → Downloads to ~/dbt-state/ (always --force)

# Parse local project to ./target/manifest.json (use after editing models)
meta refresh --dev
# → Runs: dbt parse --target dev

Optimization Analysis

# Find top optimization opportunities
meta hotspots --limit 10
# → Shows tables with highest optimization potential (scoring_details included)

# Deep analysis of specific model
meta analyze customers
# → Storage metrics, query costs, partition info, recommendations

# Analyze branch impact before merging
meta branch
# → Shows optimization impact of current branch changes

# JSON output for AI agents (includes recommendations)
meta hotspots -j | jq '.hotspots[0].scoring_details'
# → [{"criterion": "no_partition", "points": 60, "recommendation": "Add partition_by config"}]

Power BI Integration

# Extract table mappings from default workspace (from config)
meta powerbi
# → Shows datasets, reports, BigQuery tables, and dbt model mappings

# Table usage view: aggregated by BigQuery table with report/dataset counts
meta powerbi --by-table
# → Table | Reports | Datasets | dbt Model

# Include DAX measure expressions
meta powerbi --measures

# Include column schemas (name, data type, visibility)
meta powerbi --columns

# All metadata: tables + measures + columns
meta powerbi --full

# Specific workspace
meta powerbi 677db568-5923-4c5b-9b45-f14ec16a2b62

# JSON output for automation (always returns full metadata when -j is set)
meta powerbi -j | jq '.datasets[] | {name: .name, tables: .tables | length}'

# Table usage as JSON (includes report/dataset lists per table)
meta powerbi --by-table -j | jq '.tables[] | select(.in_manifest == false) | .bigquery_table'
# → BigQuery tables used in Power BI but not tracked in dbt

# Configuration (in ~/.config/dbt-meta/config.toml)
[powerbi]
enabled = true
tenant_id = "your-tenant-id"
client_id = "your-client-id"
client_secret = "your-secret"
workspaces = ["workspace-id-1", "workspace-id-2"]

Search and Discovery

# Simple substring search (old command)
meta models staging

# Advanced filtering with selectors
meta list tag:daily                           # Models with 'daily' tag
meta list config.materialized:incremental     # Incremental models
meta list path:models/core/                   # Models in specific folder

# Multiple selectors with OR logic (default)
meta list tag:daily tag:core
# → Returns models with 'daily' OR 'core' tag

# Multiple selectors with AND logic
meta list tag:daily tag:core --and
# → Returns models with both 'daily' AND 'core' tags

# Git-aware filtering
meta list -m                                  # Show modified models
meta list -f                                  # Models needing --full-refresh

# Group by tag combinations
meta list tag:daily tag:core --group
# → Groups results by tag combinations

# JSON output for scripting
meta list tag:daily -j | jq -r '.[].model'

# Search models by description (different from list)
meta search "customer dimension" -j | jq -r '.[] | .name'

# Get file path for editing
meta path customers
# → models/marts/customers.sql

⚙️ Configuration

Priority: CLI flags > TOML config > Environment variables > Defaults

Simple Mode (Zero Configuration)

No configuration needed! Just run dbt compile and start using:

cd ~/my-dbt-project
dbt compile
meta schema customers  # ✓ Works immediately with ./target/manifest.json

TOML Configuration (Recommended)

# Create config file with template
meta settings init

Edit ~/.config/dbt-meta/config.toml:

# Manifest paths
prod_manifest_path = "~/dbt-state/manifest.json"
dev_manifest_path = "./target/manifest.json"

# Dev environment
dev_schema = "personal_myname"

# Fallback behavior
fallback_dev_enabled = true      # Try dev manifest if model not in prod
fallback_bigquery_enabled = true # Query BigQuery if model not in manifests

# Production naming (optional)
prod_table_name_strategy = "alias_or_name"  # alias_or_name | name | alias
prod_schema_source = "config_or_model"      # config_or_model | model | config

Config file locations (priority order):

1. ./.dbt-meta.toml - Project-local config
2. ~/.config/dbt-meta/config.toml - User config (XDG standard)
3. ~/.dbt-meta.toml - Fallback location

Settings commands:

meta settings show      # View current configuration
meta settings validate  # Check config file for errors
meta settings path      # Show active config file path

Environment Variables (Alternative)

All TOML settings can be set via environment variables. CLI flags > TOML > env vars > defaults.

Manifest & catalog:

TOML key	Environment variable	Default
prod_manifest_path	DBT_PROD_MANIFEST_PATH	~/dbt-state/manifest.json
dev_manifest_path	DBT_DEV_MANIFEST_PATH	./target/manifest.json
prod_catalog_path	DBT_PROD_CATALOG_PATH	~/dbt-state/catalog.json
dev_catalog_path	DBT_DEV_CATALOG_PATH	./target/catalog.json
dev_schema	DBT_DEV_SCHEMA	personal_{USER}

Fallback behavior:

TOML key	Environment variable	Default
fallback_dev_enabled	DBT_FALLBACK_TARGET	true
fallback_bigquery_enabled	DBT_FALLBACK_BIGQUERY	true
fallback_catalog_enabled	DBT_FALLBACK_CATALOG	true

Production naming strategy:

TOML key	Environment variable	Options
prod_table_name_strategy	DBT_PROD_TABLE_NAME	alias_or_name (default), name, alias
prod_schema_source	DBT_PROD_SCHEMA_SOURCE	config_or_model (default), model, config

Power BI integration (optional):

TOML key	Environment variable
powerbi.enabled	POWERBI_ENABLED
powerbi.tenant_id	POWERBI_TENANT_ID
powerbi.client_id	POWERBI_CLIENT_ID
powerbi.client_secret	POWERBI_CLIENT_SECRET
powerbi.workspaces	POWERBI_WORKSPACES (comma-separated)

🧪 Development

Running Tests

# Install with dev dependencies
pip install -e ".[dev]"

# Run all tests
pytest

# Run with coverage
pytest --cov=dbt_meta --cov-report=html

# Run specific test categories
pytest -m unit              # Unit tests only
pytest -m integration       # Integration tests only
pytest -m performance       # Performance benchmarks

# Run tests in parallel
pytest -n auto

Code Quality

# Type checking
mypy src/dbt_meta

# Linting
ruff check src/dbt_meta

# Formatting
ruff format src/dbt_meta

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

Development Guidelines

• Write tests for new features (maintain 90%+ coverage)
• Follow type hints (mypy strict mode)
• Use ruff for formatting and linting
• Add docstrings for public APIs
• Update README with new features

📄 License

Copyright © 2025 Pavel Filianin

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

---

Built with ❤️ for the dbt community