dbt-autodoc 1.0.10

Automated documentation generator for dbt projects using Google Gemini AI

DBT Autodoc Documentation

dbt-autodoc is the ultimate tool for Automated Documentation and Logging for your dbt projects. It combines the power of Google Gemini AI with a robust Database Logging system to ensure your documentation is always up-to-date, accurate, and auditable.

🌟 Why dbt-autodoc?

• 🤖 Automatic AI Documentation: Generate comprehensive descriptions for your tables and columns automatically.
• 💾 Database Logging & History: Every description is stored in a database (duckdb or postgres). This acts as a "Source of Truth" and provides a full history of changes.
• 🔄 Full Synchronization: Seamlessly integrates with dbt-osmosis to keep your YAML files in sync with your SQL models.
• 🔒 Protect Manual Work: Respects human-written documentation. If you write it, we lock it.
• 👥 Team Ready: Use Postgres to share documentation cache across your entire team.

🛠️ Setup

1. Install:

   pip install dbt-autodoc
   

2. Configuration: Run dbt-autodoc to generate dbt-autodoc.yml. Important: Edit company_context in this file to give the AI knowledge about your business logic.

3. Environment Variables:

   GEMINI_API_KEY=your_api_key_here
   POSTGRES_URL=postgresql://user:pass@host:port/db (optional)
   

🚀 Quick Start

The easiest way to document your entire project in one go:

1. Automatic Mode (Recommended)

Generates table descriptions, syncs columns, and generates column descriptions using AI.

dbt-autodoc --generate-docs-ai

2. Manual Mode (No AI)

Syncs your project structure and restores descriptions from the database cache without calling AI.

dbt-autodoc --generate-docs

📋 Detailed Workflow

If you prefer granular control, you can run steps individually:

1. Generate Table Descriptions:

   dbt-autodoc --generate-docs-config-ai
   

2. Generate Column Descriptions:

   dbt-autodoc --generate-docs-yml-ai
   

3. Sync Structure Only (No AI):

   dbt-autodoc --generate-docs-yml
   

📖 Arguments Reference

Argument	Description
--generate-docs-ai	🔥 Full Auto. Runs the complete workflow: SQL generation, Osmosis sync, and YAML generation using AI.
--generate-docs	🔄 Full Sync. Runs the complete workflow using only the database cache (no AI).
--model-path	Restrict processing to a specific directory (e.g. models/staging).
--generate-docs-config-ai	Generate table descriptions in .sql files using AI.
--generate-docs-yml-ai	Generate column descriptions in .yml files using AI.
--generate-docs-config	Sync .sql files from cache (no AI).
--generate-docs-yml	Sync .yml files from cache (no AI).
--cleanup-db	Reset Database. Wipes the description cache and history.
--concurrency	Max threads for AI/DB requests (default: 10).

🧠 Smart Caching System

dbt-autodoc isn't just a wrapper for GPT. It maintains a sophisticated state:

1. doc_cache Table: Stores the current active description.
2. doc_cache_log Table: logs every single change to a description, by whom, and when.

This means if you accidentally delete a description from your code, dbt-autodoc --generate-docs will restore it instantly from the database history!

📄 License

MIT License - see LICENSE for details.

🙏 Attribution

Brought to you by JustDataPlease.