deepdoc 2.3.1

Auto-generate beautiful docs from any codebase

DeepDoc

Repository Layout

Directory	What it is	Where to start
deepdoc/	The Python package — CLI, pipeline, planner, generator, chatbot, and site builder. This is the core product.	deepdoc/README.md
web/	Marketing and changelog site built with Astro 5 + Tailwind. Deployed to the public DeepDoc website.	web/README.md
vscode-extension/	VS Code extension — explains selected code snippets in Fast or Deep mode and inserts AI-generated comments inline.	vscode-extension/README.md
tests/	pytest test suite for the Python package.	Run python3 -m pytest -q from repo root.
scripts/	One-off release and maintenance scripts.	—

---

Auto-generate deep engineering documentation from real codebases using AI.

DeepDoc scans your repo, builds a bucket-based documentation plan, generates rich MDX pages with Mermaid diagrams, and builds a local-first Fumadocs site with Orama search.

Contents

• Quick Start
• Chatbot in 5 Minutes
• Installation
• Commands
• LLM Provider Setup
• Configuration
• Chatbot
• Supported Languages & Frameworks
• Architecture
• Generated Files
• GitHub Actions CI/CD
• Requirements
• Contributing & Release Flow

---

Features

• Bucket-based documentation architecture — system, feature, endpoint, integration, and database buckets instead of one-file-per-page noise.
• Multi-step AI planner — classifies the repo, proposes buckets, then assigns files, symbols, artifacts, and dependencies into a final reader-first plan.
• Giant-file handling — large files are decomposed into feature-aligned clusters so a single controller can feed multiple doc pages.
• MDX compile gate — every page is compile-checked before being written; broken MDX is auto-fixed via LLM retry or degraded to safe Markdown rather than shipped.
• Evidence-first chatbot answers — final code proof is hydrated from archived source snippets with exact file paths and line ranges, not just retrieval guesses.
• Incremental updates — deepdoc update regenerates only stale or structurally affected docs against the last synced commit.
• OpenAPI-aware API docs — auto-detects OpenAPI/Swagger specs and stages interactive /api/* pages in the generated site.
• Local-first Fumadocs site — generates a site/ Next.js app with Fumadocs UI, Mermaid rendering, and built-in search; deepdoc deploy exports a static site to any host.

Works with Anthropic, OpenAI, Azure OpenAI, Google Gemini, Ollama, and any other LiteLLM-compatible provider. Parses Python, JavaScript/TypeScript, Go, PHP, and Vue.

---

Installation

From PyPI (recommended)

pip install deepdoc

If you want DeepDoc's chatbot features, install the chatbot extra:

pip install "deepdoc[chatbot]"

The base install does not include chatbot dependencies.

From source (recommended during development)

git clone https://github.com/tss-pranavkumar/deepdoc.git
cd deepdoc
pip install -e .

If you want chatbot features during development:

pip install -e ".[chatbot]"

If the full install is slow due to tree-sitter compilation, install core deps first:

pip install click litellm gitpython rich pyyaml jinja2
pip install -e . --no-deps

You will also need Node 18+ on PATH. DeepDoc uses Node at generate time to compile-check each MDX page before writing it, and at serve/deploy time for the Fumadocs site. The small set of MDX validation dependencies installs into deepdoc/generator/mdx_validator/node_modules/ automatically on first deepdoc generate.

Verify installation

deepdoc --version
deepdoc --help
python -m deepdoc --help
node --version    # must report 18 or higher

If you installed the chatbot extra, you can verify those dependencies with:

pip show faiss-cpu fastapi uvicorn

---

Quick Start

Docs only

cd /path/to/your-project
deepdoc init --provider anthropic
export ANTHROPIC_API_KEY=sk-ant-...
deepdoc generate
deepdoc serve   # → http://localhost:3000

Docs + chatbot — one env var

cd /path/to/your-project
pip install "deepdoc[chatbot]"
deepdoc init --with-chatbot --provider anthropic
export ANTHROPIC_API_KEY=sk-ant-...
deepdoc generate
deepdoc serve   # → http://localhost:3000  •  chatbot at /ask

The chatbot reuses your doc-gen LLM and runs embeddings locally via fastembed. No extra keys, no extra config. Swap --provider anthropic for openai, gemini, azure, or ollama to use a different LLM.

When chatbot.enabled is false, DeepDoc generates a docs-only site: no chatbot route, no chatbot UI components, and no chatbot_backend/ scaffold.

---

Chatbot in 5 Minutes

Defaults work out of the box. The chatbot inherits your llm.* config and runs embeddings locally via fastembed. The recipes below only add config when you actually need to override the defaults (Azure endpoints, custom Ollama base URL, or mixing providers).

Single provider — one key for everything

This is the path 95% of users want. Anthropic shown; swap --provider for openai, gemini, etc.

pip install "deepdoc[chatbot]"
cd /path/to/your-project

deepdoc init --with-chatbot --provider anthropic
export ANTHROPIC_API_KEY=sk-ant-...

deepdoc generate
deepdoc serve
# → Docs at http://localhost:3000  •  Chatbot at http://localhost:3000/ask

Mix providers — different LLM for chat than for docs

Use Claude for docs (great long-form), GPT-4o-mini for chat answers (fast + cheap), local embeddings.

pip install "deepdoc[chatbot]"
cd /path/to/your-project

deepdoc init --with-chatbot --provider anthropic
export ANTHROPIC_API_KEY=sk-ant-...
export OPENAI_API_KEY=sk-...

deepdoc config set chatbot.answer.provider openai
deepdoc config set chatbot.answer.model gpt-4o-mini
deepdoc config set chatbot.answer.api_key_env OPENAI_API_KEY

deepdoc generate
deepdoc serve

Want cloud embeddings instead of local? Add:

deepdoc config set chatbot.embeddings.backend litellm
deepdoc config set chatbot.embeddings.provider openai
deepdoc config set chatbot.embeddings.model text-embedding-3-large
deepdoc config set chatbot.embeddings.api_key_env OPENAI_API_KEY

Azure OpenAI

Azure deployments need explicit endpoint + API-version values. Replace YOUR-RESOURCE and deployment names with your actual values.

pip install "deepdoc[chatbot]"
cd /path/to/your-project

export AZURE_API_KEY=...
export AZURE_API_BASE=https://YOUR-RESOURCE.openai.azure.com

deepdoc init --with-chatbot --provider azure --model azure/gpt-4o
deepdoc config set llm.base_url $AZURE_API_BASE

# Only embeddings need separate Azure config (different deployment)
deepdoc config set chatbot.embeddings.backend litellm
deepdoc config set chatbot.embeddings.provider azure
deepdoc config set chatbot.embeddings.model azure/text-embedding-3-large
deepdoc config set chatbot.embeddings.api_key_env AZURE_API_KEY
deepdoc config set chatbot.embeddings.base_url $AZURE_API_BASE
deepdoc config set chatbot.embeddings.api_version 2024-02-01

deepdoc generate
deepdoc serve

The answer LLM inherits llm.* (Azure deployment + base URL + key) automatically.

Ollama — Fully local, no API keys

Free and private. Runs everything on your machine.

pip install "deepdoc[chatbot]"
cd /path/to/your-project

# Start Ollama first: https://ollama.com
ollama pull llama3.2

deepdoc init --with-chatbot --provider ollama --model ollama/llama3.2
deepdoc config set llm.base_url http://localhost:11434

deepdoc generate
deepdoc serve
# → Docs at http://localhost:3000  •  Chatbot at http://localhost:3000/ask

Fastembed downloads the local embedding model (~300 MB) on first run. No API key needed for any step.

---

Commands

Every command supports --help, including nested config commands:

deepdoc --help
deepdoc generate --help
deepdoc config --help
deepdoc config set --help

deepdoc init

Initializes DeepDoc in the current directory by creating a .deepdoc.yaml config file.

deepdoc init
deepdoc init --provider openai --model gpt-4o
deepdoc init --provider ollama --model ollama/llama3.2
deepdoc init --provider azure --model azure/gpt-4o
deepdoc init --output-dir documentation
deepdoc init --with-chatbot
deepdoc init --provider openai --with-chatbot

Options:

Flag	Default	Description
--name	directory name	Project name
--description	empty	Short project description
--provider	anthropic	LLM provider: anthropic, openai, ollama, azure
--model	provider default	Model name
--output-dir	docs	Where generated docs are written
--with-chatbot	off	Enable chatbot scaffolding and indexing (see Chatbot section)

deepdoc generate

Full documentation generation. This is the first-run or explicit full-refresh command.

deepdoc generate
deepdoc generate --force           # Full refresh of DeepDoc-managed docs
deepdoc generate --clean --yes     # Wipe output + state and rebuild from scratch
deepdoc generate --deploy          # Generate + export the static site
deepdoc generate --batch-size 3    # Smaller batches for rate-limited APIs
deepdoc generate --include "src/**" --include "lib/**"
deepdoc generate --exclude "tests/**"

Current behavior:

• deepdoc generate
  • intended for the first run
  • refuses to run if DeepDoc docs/state already exist
  • refuses to write into a non-DeepDoc docs/ folder unless you explicitly clean it
• deepdoc generate --force
  • re-runs the full pipeline
  • regenerates all DeepDoc-managed pages even if they are not stale
  • removes stale generated pages that no longer belong in the new plan
  • preserves non-DeepDoc files
• deepdoc generate --clean --yes
  • deletes the output dir and DeepDoc state
  • rebuilds everything from scratch

What happens under the hood (5-phase pipeline):

1. Phase 1: Scan — Walk the repo, parse supported languages, detect endpoints, config/setup artifacts, runtime surfaces, integration signals, and OpenAPI specs.
2. Phase 2: Plan — Run the multi-step bucket planner. It classifies the repo, proposes bucket candidates, and assigns files/symbols/artifacts to the final doc structure.
3. Phase 3: Generate — Generate bucket pages in batches with parallel workers. High-level buckets are AI-planned; scanned endpoints enrich grouped API-reference pages instead of creating one page per route. Each page passes through an MDX compile-check (with bounded LLM-driven retry on failure) before being written to disk.
4. Phase 4: API Ref — Stage OpenAPI assets for the generated Fumadocs /api/* pages when a spec exists.
5. Phase 5: Build — Write the generated site/ Fumadocs scaffold, page tree, search route, and static assets from the generated plan.

Options:

Flag	Default	Description
--force	off	Full refresh of DeepDoc-managed docs and cleanup of stale generated pages
--clean	off	Delete output dir and DeepDoc state, then regenerate from scratch
--yes	off	Skip destructive confirmation for --clean
--include	all files	Glob patterns to include (can be repeated)
--exclude	see config	Additional glob patterns to exclude
--deploy	off	Build and export the static site after generation
--batch-size	10	Pages per batch before pausing (helps with rate limits)

deepdoc update

Incrementally update docs when source files change. This is the normal command after the first successful generate.

deepdoc update                    # Normal ongoing refresh
deepdoc update --since HEAD~3     # Changes in last 3 commits
deepdoc update --since main       # All changes since branching from main
deepdoc update --replan           # Force a full replan
deepdoc update --strict-quality   # Fail if generated docs are invalid/degraded
deepdoc update --deploy           # Update + deploy

How it works:

1. Loads the saved sync baseline, plan, and generation ledger from .deepdoc/.
2. Diffs committed changes from the last synced commit to the current HEAD.
3. Chooses a strategy automatically:
   • incremental — regenerate only the stale bucket pages
   • targeted replan — new/deleted files or endpoint structure changes; re-plans affected buckets then regenerates them. Full replan is never triggered automatically for normal code changes.
4. Compares the saved scan cache with the current scan so semantic endpoint changes can refresh impacted docs even when ownership files do not line up directly.
5. Regenerates only the affected bucket pages. Deleted files are cleaned up in-place: orphaned buckets and their MDX pages are removed, partially-emptied buckets are marked stale and regenerated.
6. Appends an entry to .deepdoc/changelog.json and regenerates docs/whats-changed.mdx so the docs site always shows a current commit-by-commit change log.
7. Incrementally refreshes the chatbot corpora from the same update run.
8. Rebuilds site config and nav afterward.

If git is unavailable, it falls back to hash-based staleness detection for recovery. When --deploy is used, DeepDoc deploys only after a fully successful update; failed page generation or chatbot sync blocks deployment.

Generation writes quality artifacts under .deepdoc/:

• .deepdoc/generation_quality.json records invalid/degraded pages, coverage metrics, local setup warnings, and consistency summary data.
• .deepdoc/consistency_warnings.json records warning-only cross-page identifier consistency findings.

Generated MDX pages include provenance frontmatter such as deepdoc_generated_commit, deepdoc_generated_at, deepdoc_generated_version, deepdoc_status, deepdoc_evidence_files, and deepdoc_prereqs (prerequisite page slugs). The generated Fumadocs site renders a subtle "Last generated from commit ..." badge and wires prev/next navigation arrows automatically. Pages with prerequisites show a "Read first:" callout at the top.

Options:

Flag	Default	Description
--since	last synced commit	Git ref to diff against
--replan	off	Force a full replan even if the change set looks incremental
--deploy	off	Deploy after a fully successful update

deepdoc status

Show how much documentation has been generated and whether any buckets are stale.

deepdoc status

This is useful after generate or update when you want a quick health check without opening the site.

deepdoc serve

Preview the generated docs locally with live reload using the generated Fumadocs app in site/.

deepdoc serve
deepdoc serve --port 8001

Requires Node.js >= 18 to be installed. Site dependencies are auto-installed into site/node_modules/ on first run.

deepdoc deploy

Build and export the generated Fumadocs site.

deepdoc deploy

This runs next build inside site/ and writes the static export to site/out/. You can deploy that directory to Vercel, Netlify, GitHub Pages, Cloudflare Pages, or any static host.

Before building, deepdoc deploy checks .deepdoc/generation_quality.json and generated MDX frontmatter. It refuses to deploy when the last generation has failed/invalid pages or when docs/ still contains pages marked deepdoc_status: "invalid" or stub: true; rerun deepdoc generate after fixing those issues.

If you want GitHub Pages specifically, this repo includes a workflow at .github/workflows/github-pages.yml that publishes the checked-in site/out/ export through the official Pages Actions flow. That means you do not need to move the export into a branch docs/ folder.

deepdoc config

View or update config values without editing YAML manually.

deepdoc config show                                    # Print all config
deepdoc config set llm.provider openai                 # Switch provider
deepdoc config set llm.model gpt-4o                    # Switch model
deepdoc config set llm.temperature 0.3                 # Adjust creativity
deepdoc config set output_dir documentation            # Change output dir
deepdoc config set llm.api_key_env AZURE_API_KEY       # Change API key env var
deepdoc config set compatibility.deprecated_version_warning.enabled false

deepdoc benchmark

Run planner benchmark cases and optionally generate a combined docs+chatbot quality scorecard.

deepdoc benchmark --catalog benchmarks/catalog.json
deepdoc benchmark --repo /path/to/repo --gold benchmarks/gold.json
deepdoc benchmark --catalog benchmarks/catalog.json --chatbot-eval benchmarks/chatbot_eval.json
deepdoc benchmark --catalog benchmarks/catalog.json --chatbot-eval benchmarks/chatbot_eval.json --scorecard-out .deepdoc/quality_scorecard.json --strict-scorecard
deepdoc benchmark --generated-root /Users/apple/autodoc/docs --scorecard-out /Users/apple/autodoc/docs/_scorecards/latest.json

Use --strict-scorecard to fail the command when completeness gates are not met.

When you do not have a hand-written benchmark catalog or chatbot eval file yet, use artifact mode (--generated-root or --artifact-repo) to compute a provisional scorecard directly from persisted .deepdoc/ outputs.

---

LLM Provider Setup

DeepDoc uses LiteLLM under the hood, which means it supports 100+ providers.

For new users, start with one of these common doc-generation setups:

Anthropic (Claude) — Default

deepdoc init --provider anthropic
export ANTHROPIC_API_KEY=sk-ant-api03-...
deepdoc generate

Models: claude-3-5-sonnet-20241022, claude-3-opus-20240229, claude-3-haiku-20240307

OpenAI (GPT)

deepdoc init --provider openai --model gpt-4o
export OPENAI_API_KEY=sk-...
deepdoc generate

Models: gpt-4.1, gpt-4.1-mini, gpt-4o, gpt-4o-mini, gpt-4-turbo

Google Gemini

If you use a simple Google AI Studio API key, use the gemini/ prefix. Without that prefix, LiteLLM defaults to Vertex AI semantics.

deepdoc init
deepdoc config set llm.provider gemini
deepdoc config set llm.model gemini/gemini-2.0-flash
deepdoc config set llm.api_key_env GEMINI_API_KEY
export GEMINI_API_KEY=...
deepdoc generate

Common choices: gemini/gemini-2.0-flash, gemini/gemini-2.5-flash, gemini/gemini-2.5-pro

Azure OpenAI

Azure requires a few more environment variables because deployments have custom names and endpoints.

# 1. Initialize with Azure
deepdoc init --provider azure --model azure/<your-deployment-name>

# 2. Set required environment variables
export AZURE_API_KEY=your-azure-api-key
export AZURE_API_BASE=https://<your-resource-name>.openai.azure.com
export AZURE_API_VERSION=2024-02-01

# 3. Update config to point to your deployment
deepdoc config set llm.model azure/<your-deployment-name>
deepdoc config set llm.base_url https://<your-resource-name>.openai.azure.com

# 4. Generate
deepdoc generate

Where to find these values in Azure Portal:

1. Go to Azure Portal → Azure OpenAI resource.
2. Click Keys and Endpoint in the sidebar → copy Key 1 (that's your AZURE_API_KEY) and the Endpoint (that's your AZURE_API_BASE).
3. Go to Model deployments → Manage Deployments → note your deployment name (e.g., gpt-4o-deployment). Use this as azure/gpt-4o-deployment in the model field.
4. API version: Use 2024-02-01 or the latest GA version shown in Azure docs.

Example .deepdoc.yaml for Azure:

project_name: my-project
output_dir: docs
llm:
  provider: azure
  model: azure/gpt-4o-deploy      # "azure/" prefix + your deployment name
  api_key_env: AZURE_API_KEY
  base_url: https://mycompany.openai.azure.com
  max_tokens: 4096
  temperature: 0.2

Azure AD / Managed Identity (token-based auth):

If you use Azure AD instead of API keys, set these instead:

export AZURE_AD_TOKEN=your-ad-token
export AZURE_API_BASE=https://<your-resource-name>.openai.azure.com
export AZURE_API_VERSION=2024-02-01

LiteLLM picks up AZURE_AD_TOKEN automatically when AZURE_API_KEY is not set.

Ollama (Local / Free)

No API key needed. Just make sure Ollama is running locally.

# 1. Install and start Ollama (https://ollama.com)
ollama pull llama3.2

# 2. Initialize
deepdoc init --provider ollama --model ollama/llama3.2

# 3. Generate (no API key needed)
deepdoc generate

Other Ollama models: ollama/codellama, ollama/mistral, ollama/mixtral

Any LiteLLM Provider

DeepDoc passes the model string directly to LiteLLM, so you can use any provider LiteLLM supports by using the correct prefix. deepdoc init only has shortcuts for the common providers above, so for everything else initialize once and then use deepdoc config set ...:

# Gemini
deepdoc config set llm.provider gemini
deepdoc config set llm.model gemini/gemini-2.0-flash
export GEMINI_API_KEY=...

# Groq
deepdoc config set llm.model groq/llama3-70b-8192
export GROQ_API_KEY=...

# Together AI
deepdoc config set llm.model together_ai/meta-llama/Llama-3-70b-chat-hf
export TOGETHER_API_KEY=...

# AWS Bedrock
deepdoc config set llm.model bedrock/anthropic.claude-3-sonnet-20240229-v1:0
# (uses AWS credentials from environment)

See LiteLLM providers for the full list.

---

Configuration

The .deepdoc.yaml file in your repo root controls everything:

project_name: my-app
description: "A web application for managing tasks"
output_dir: docs
site_dir: site

llm:
  provider: anthropic
  model: claude-3-5-sonnet-20241022
  api_key_env: ANTHROPIC_API_KEY
  base_url: null                    # Set for Ollama/custom endpoints
  max_tokens: null                  # null = no cap (recommended); set a number to limit output
  temperature: 0.2

languages:
  - python
  - javascript
  - typescript
  - go
  - php

include: []                         # Empty = include everything
exclude:
  - node_modules
  - .git
  - __pycache__
  - "*.pyc"
  - vendor
  - dist
  - build
  - .env
  - "*.lock"
  - "*.sum"

generation_mode: feature_buckets

# Generation tuning
max_pages: 0                        # 0 = no cap; set a number to limit total pages
giant_file_lines: 2000              # Files above this get LLM-based feature clustering
source_context_budget: 200000       # Raw-source char budget before DeepDoc switches overflow files to compressed evidence cards
integration_detection: auto         # "auto" | "off"

# Page type toggles
include_endpoint_pages: true        # Generate endpoint documentation
include_integration_pages: true     # Generate integration documentation

# Parallelism — tune for your LLM provider's rate limits
max_parallel_workers: 6             # Concurrent LLM calls (increase for Azure PTU)
batch_size: 10                      # Pages per batch before rate-limit pause

github_pages:
  enabled: false
  branch: gh-pages
  remote: origin

site:
  repo_url: ""                      # e.g., https://github.com/you/your-repo
  favicon: ""
  logo: ""

Configuration Reference

Key	Default	Description
project_name	directory name	Project name used in site title
description	""	Short project description
output_dir	docs	Where generated markdown pages are written
site_dir	site	Where the generated Fumadocs site lives
LLM
llm.provider	anthropic	anthropic, openai, azure, ollama, or any LiteLLM alias
llm.model	claude-3-5-sonnet-20241022	Model name (use provider prefix for non-Anthropic, e.g. azure/gpt-4.1)
llm.api_key_env	ANTHROPIC_API_KEY	Environment variable that holds the API key
llm.base_url	null	Custom endpoint URL (required for Ollama, optional for Azure)
llm.max_tokens	null	Max output tokens per LLM call. null = no cap (recommended). Set explicitly if your provider requires it (e.g. some Azure deployments). Typical values: 4096 for shorter pages, 8192–16384 for detailed docs
llm.temperature	0.2	LLM sampling temperature
Generation
generation_mode	feature_buckets	Documentation generation mode
max_pages	0	Max pages to generate. 0 = no cap
giant_file_lines	2000	Files above this line count get LLM-based feature clustering
source_context_budget	200000	Raw-source char budget per page before overflow files are represented as compressed evidence cards
integration_detection	auto	Detect third-party integrations: auto or off
include_endpoint_pages	true	Generate endpoint documentation pages
include_integration_pages	true	Generate integration documentation pages
Parallelism
max_parallel_workers	6	Concurrent LLM calls. Increase for Azure PTU or high-TPM deployments
batch_size	10	Pages per batch before rate-limit pause
File filters
languages	[python, javascript, typescript, go, php, vue]	Languages to parse
include	[]	Glob patterns to include (empty = everything)
exclude	(see config)	Glob patterns to exclude (node_modules, .git, dist, etc.)
GitHub Pages
github_pages.branch	gh-pages	Branch for GitHub Pages deploy
github_pages.remote	origin	Git remote for deploy
Site
site.repo_url	""	Repo URL shown in the generated Fumadocs navigation
site.favicon	""	Path to favicon
site.logo	""	Path to logo
Compatibility
compatibility.deprecated_version_warning.enabled	true	Warn when existing generated docs were produced by a different major version of DeepDoc (e.g. docs from v1.x with CLI v2.x). Suppressed for minor/patch version gaps.

---

Chatbot

DeepDoc can generate an AI-powered chatbot that answers questions about your codebase using evidence-first RAG. Retrieval indexes find candidate files, but final code proof always comes from archived source/config snippets with exact file paths and line ranges. Generated docs and repo-authored docs can help explain concepts, but they are returned as references, not right-pane code evidence.

Defaults you should know. The chatbot inherits provider, model, and api_key_env from your llm.* config when chatbot.answer.* is empty (the scaffolded default). Embeddings default to fastembed — fully local, no API key. You only need extra config if (a) you want a different LLM for chat than for docs, or (b) you want cloud embeddings.

If chatbot is disabled, DeepDoc keeps the generated site docs-only: it does not generate the /ask route, chatbot frontend components, or chatbot_backend/ scaffold.

Already running? If you followed the Chatbot in 5 Minutes guide above, your chatbot is already configured and working. Everything below is for advanced configuration, tuning, and production deployment.

How the Model Surfaces Work

The chatbot has three independent model surfaces that you can mix across providers:

Surface	Controls	Example
llm.*	Doc planning and page generation	Claude, GPT-4o, Gemini
chatbot.answer.*	Chatbot answer generation	GPT-4o-mini, Claude, Gemini Flash
chatbot.embeddings.*	Vector embeddings for retrieval	text-embedding-3-large, Gemini text-embedding-004

deepdoc serve auto-starts the chatbot backend alongside the Fumadocs site. The backend port is deterministically assigned from your repo path (range 8100–8799) unless you set an explicit base_url.

Evidence-First Responses

All chatbot endpoints now share one response contract:

• evidence[] is the canonical source of right-pane code/config snippets. Each item has an ID like E1, file_path, start_line, end_line, and snippet.
• references[] contains generated docs or repo-authored docs. These are read-next links, not implementation proof.
• Legacy fields such as code_citations, doc_links, code_workspace_citations, and file_inventory are derived for compatibility.
• SQLite FTS, FAISS vectors, symbol chunks, and relationship chunks are candidate retrieval artifacts only. A candidate becomes evidence only after it is hydrated from the source archive/catalog.
• Generated/internal paths such as .deepdoc*, docs/, site/, and chatbot_backend/ are excluded from source evidence.
• The answer validator rejects invented source paths, line unknown, unknown evidence IDs, and docs used as implementation proof. If a retry still fails, the backend returns a conservative answer with diagnostics.

Public endpoints remain stable:

Endpoint	Mode	Behavior
POST /query	Fast	Single-pass, index-first answer. Uses source/config evidence and optional doc references.
POST /deep-research	Deep	Multi-step synthesis. May use docs for orientation, but implementation claims must be grounded in source/config evidence.
POST /code-deep	Code Deep	Strict source-first answer with trace and file inventory compatibility fields. Missing exact evidence is reported as a gap.
POST /query-context	Diagnostics	Returns selected candidates, hydrated evidence[], references[], and diagnostics without generating an answer.
POST /code-deep/stream	Code Deep stream	Emits SSE trace events followed by the final evidence-first payload.

Provider Recipes

These are advanced recipes that override the inheritance defaults — e.g. mixing providers or using cloud embeddings. For the simple "one key for everything" path, see Chatbot in 5 Minutes above.

Claude for docs, OpenAI for embeddings

Claude doesn't offer embedding models, so cloud embeddings need a separate provider. (Or just keep the default fastembed and skip this entirely.)

export ANTHROPIC_API_KEY=...
export OPENAI_API_KEY=...

deepdoc config set llm.provider anthropic
deepdoc config set llm.model claude-3-5-sonnet-20241022
deepdoc config set llm.api_key_env ANTHROPIC_API_KEY

deepdoc config set chatbot.enabled true
deepdoc config set chatbot.embeddings.backend litellm
deepdoc config set chatbot.embeddings.provider openai
deepdoc config set chatbot.embeddings.model text-embedding-3-large
deepdoc config set chatbot.embeddings.api_key_env OPENAI_API_KEY

The chatbot answer LLM inherits llm.* (Claude) automatically.

Different model for chat than for docs

Claude for docs, GPT-4o-mini for cheaper/faster chat answers. Embeddings stay local.

export ANTHROPIC_API_KEY=...
export OPENAI_API_KEY=...

deepdoc config set llm.provider anthropic
deepdoc config set llm.model claude-3-5-sonnet-20241022
deepdoc config set llm.api_key_env ANTHROPIC_API_KEY

deepdoc config set chatbot.enabled true
deepdoc config set chatbot.answer.provider openai
deepdoc config set chatbot.answer.model gpt-4o-mini
deepdoc config set chatbot.answer.api_key_env OPENAI_API_KEY

Azure with separate embedding deployment

Azure deployments need explicit base_url + api_version, and embeddings typically live on a different deployment.

export AZURE_API_KEY=...

deepdoc config set llm.provider azure
deepdoc config set llm.model azure/gpt-4.1
deepdoc config set llm.api_key_env AZURE_API_KEY
deepdoc config set llm.base_url https://YOUR-RESOURCE.openai.azure.com/

deepdoc config set chatbot.enabled true
deepdoc config set chatbot.embeddings.backend litellm
deepdoc config set chatbot.embeddings.provider azure
deepdoc config set chatbot.embeddings.model azure/text-embedding-3-small
deepdoc config set chatbot.embeddings.api_key_env AZURE_API_KEY
deepdoc config set chatbot.embeddings.base_url https://YOUR-RESOURCE.openai.azure.com/
deepdoc config set chatbot.embeddings.api_version 2024-02-15-preview

The answer LLM inherits llm.* (Azure deployment, base URL, version, key) automatically — no chatbot.answer.* config needed for Azure.

Installation

The chatbot requires extra dependencies not included in the base install:

# From PyPI
pip install "deepdoc[chatbot]"

# From source (development)
pip install -e ".[chatbot]"

Extra dependencies: numpy, faiss-cpu, fastapi, uvicorn, httpx.

Verify the chatbot extras are installed:

pip show faiss-cpu fastapi uvicorn

Chatbot Configuration

All chatbot settings live under the chatbot key in .deepdoc.yaml. Running deepdoc init --with-chatbot populates these defaults:

These are scaffolding defaults, not a recommendation that you must use Azure. Most teams change the provider, model, and API-key env vars right after init.

chatbot:
  enabled: true
  index_dir: ".deepdoc/chatbot"

  backend:
    base_url: ""                              # Leave empty for auto-assigned local port
    allowed_origins:
      - "http://localhost:3000"
      - "http://127.0.0.1:3000"

  answer:                                     # LLM used for answering user questions
    provider: "azure"
    model: "azure/gpt-4o-mini"
    api_key_env: "DEEPDOC_CHAT_API_KEY"
    base_url: ""
    api_version: ""
    temperature: 0.1
    max_tokens: 24000
    continuation_retries: 2                   # Auto-continue if answer ends abruptly
    continuation_context_chars: 12000         # Tail chars included in continuation prompt

  embeddings:                                 # LLM used for embedding code/docs
    backend: "litellm"                       # "litellm" for cloud embeddings or "fastembed" for fully local embeddings
    fastembed_model: "nomic-ai/nomic-embed-text-v1.5"
    fastembed_batch_size: 4
    provider: "azure"
    model: "azure/text-embedding-3-large"
    api_key_env: "DEEPDOC_EMBED_API_KEY"
    base_url: ""
    api_version: ""
    batch_size: 24

  vector_store:
    kind: "faiss"

  indexing:
    include_repo_docs: true
    include_tests: false
    repo_doc_globs: []
    exclude_globs: []
    max_file_bytes: 250000
    max_repo_doc_chars: 12000

  retrieval:
    top_k_code: 15
    top_k_artifact: 8
    top_k_docs: 6
    top_k_relationship: 8
    candidate_top_k_code: 30
    candidate_top_k_artifact: 16
    candidate_top_k_docs: 12
    candidate_top_k_relationship: 12
    max_prompt_code_chunks: 12
    max_prompt_artifact_chunks: 6
    max_prompt_doc_chunks: 6
    max_prompt_relationship_chunks: 6
    max_prompt_chars: 120000
    fast_mode_use_llm_retrieval_steps: false  # Fast mode skips expansion/rerank by default
    fast_mode_iterative_retrieval: false      # Fast mode skips second-pass follow-up retrieval
    fast_mode_max_prompt_chars: 90000         # Smaller prompt budget for faster /query answers
    deep_mode_max_prompt_chars: 140000        # Larger budget for /deep-research synthesis
    code_deep_mode_max_prompt_chars: 180000   # Largest prompt budget for /code-deep
    code_deep_top_k: 16                       # Code chunks retrieved for code-aware mode
    code_deep_top_k_relationship: 12          # Relationship chunks retrieved for code-aware mode
    code_deep_top_k_docs: 4                   # Cap docs chunks in code-aware mode
    code_deep_file_inventory_limit: 18        # Max files listed in code-aware inventory
    lexical_retrieval: true
    lexical_candidate_limit: 24
    query_expansion: true
    expansion_max_queries: 3
    iterative_retrieval: true
    iterative_max_followup_queries: 2
    graph_neighbor_expansion: true
    graph_neighbor_max_files: 6
    graph_neighbor_code_chunks_per_file: 2
    graph_neighbor_artifact_chunks_per_file: 1
    graph_neighbor_relationship_chunks_per_file: 2
    graph_neighbor_max_docs: 4
    rerank: true
    rerank_candidate_limit: 32
    rerank_candidate_limit_per_kind: 8
    rerank_preview_chars: 450
    stitch_adjacent_code_chunks: true
    stitch_max_adjacent_chunks: 2
    deep_research_live_fallback: true
    live_fallback_max_files: 6
    live_fallback_max_per_file: 2
    live_fallback_context_lines: 12
    deep_research_chunk_chars: 3200
    deep_research_top_k: 10

  chunking:
    code_chunk_lines: 120
    code_chunk_overlap: 20
    artifact_chunk_lines: 140
    artifact_chunk_overlap: 20
    max_doc_summary_chunks_per_page: 4
    max_doc_summary_chars: 4000

Chatbot Configuration Reference

The defaults work for almost every project. Expand below only when you need to tune a specific knob.

Full chatbot configuration reference (60+ keys)

Key	Default	Description
General
chatbot.enabled	false	Enable chatbot indexing and backend (set automatically by deepdoc init --with-chatbot)
chatbot.index_dir	.deepdoc/chatbot	Directory for source archive/catalog, SQLite lexical index, vector indexes, relationship artifacts, and chunk data
Indexing
chatbot.indexing.include_repo_docs	true	Index selected repo-authored docs such as README/design notes in a separate corpus
chatbot.indexing.include_tests	false	Allow test/example/fixture docs into the repo-doc corpus
chatbot.indexing.repo_doc_globs	[]	Extra glob patterns for repo docs to index
chatbot.indexing.exclude_globs	[]	Additional glob patterns to exclude from repo-doc indexing
chatbot.indexing.max_file_bytes	250000	Skip oversized repo-doc files during indexing
chatbot.indexing.max_repo_doc_chars	12000	Max chars per repo-doc section chunk
Backend
chatbot.backend.base_url	""	External backend URL. Leave empty for auto-assigned local port
chatbot.backend.allowed_origins	[localhost:3000, 127.0.0.1:3000]	CORS origins the backend accepts
Answer LLM
chatbot.answer.provider	azure	Provider for the answer model
chatbot.answer.model	azure/gpt-4o-mini	Model used to generate answers
chatbot.answer.api_key_env	DEEPDOC_CHAT_API_KEY	Env var holding the answer model API key
chatbot.answer.base_url	""	Custom endpoint (for Azure, Ollama, etc.)
chatbot.answer.api_version	""	Azure API version string
chatbot.answer.temperature	0.1	Sampling temperature (lower = more deterministic)
chatbot.answer.max_tokens	24000	Max tokens per answer
chatbot.answer.continuation_retries	2	Extra completion attempts when an answer appears truncated
chatbot.answer.continuation_context_chars	12000	Number of trailing chars passed when asking the model to continue
Embeddings LLM
chatbot.embeddings.backend	litellm	Embedding backend: litellm for cloud providers or fastembed for fully local embeddings
chatbot.embeddings.fastembed_model	nomic-ai/nomic-embed-text-v1.5	Local embedding model used when chatbot.embeddings.backend=fastembed
chatbot.embeddings.fastembed_batch_size	4	Local fastembed batch size
chatbot.embeddings.provider	azure	Provider for the embedding model
chatbot.embeddings.model	azure/text-embedding-3-large	Embedding model
chatbot.embeddings.api_key_env	DEEPDOC_EMBED_API_KEY	Env var holding the embedding API key
chatbot.embeddings.base_url	""	Custom endpoint
chatbot.embeddings.api_version	""	Azure API version string
chatbot.embeddings.batch_size	24	Texts per embedding API call
Retrieval
chatbot.retrieval.top_k_code	15	Top code chunks retrieved per query
chatbot.retrieval.top_k_artifact	8	Top artifact chunks retrieved per query
chatbot.retrieval.top_k_docs	6	Top generated-doc and repo-doc chunks retrieved per query
chatbot.retrieval.top_k_relationship	8	Top relationship chunks retrieved per query
chatbot.retrieval.candidate_top_k_code	30	Candidate code chunks gathered before reranking
chatbot.retrieval.candidate_top_k_artifact	16	Candidate artifact chunks gathered before reranking
chatbot.retrieval.candidate_top_k_docs	12	Candidate doc chunks gathered before reranking
chatbot.retrieval.candidate_top_k_relationship	12	Candidate relationship chunks gathered before reranking
chatbot.retrieval.max_prompt_code_chunks	12	Max code chunks included in the final prompt
chatbot.retrieval.max_prompt_artifact_chunks	6	Max artifact chunks in the final prompt
chatbot.retrieval.max_prompt_doc_chunks	6	Max doc chunks in the final prompt
chatbot.retrieval.max_prompt_relationship_chunks	6	Max relationship chunks included in the final prompt
chatbot.retrieval.max_prompt_chars	120000	Default character budget for assembled prompts
chatbot.retrieval.fast_mode_use_llm_retrieval_steps	false	In /query fast mode, disable LLM query expansion and reranking
chatbot.retrieval.fast_mode_iterative_retrieval	false	In /query fast mode, disable iterative follow-up retrieval
chatbot.retrieval.fast_mode_max_prompt_chars	90000	Prompt budget used by /query fast mode
chatbot.retrieval.deep_mode_max_prompt_chars	140000	Prompt budget used by /deep-research
chatbot.retrieval.code_deep_mode_max_prompt_chars	180000	Prompt budget used by /code-deep
chatbot.retrieval.code_deep_top_k	16	Code chunks retrieved in code-aware mode
chatbot.retrieval.code_deep_top_k_relationship	12	Relationship chunks retrieved in code-aware mode
chatbot.retrieval.code_deep_top_k_docs	4	Docs chunk cap in code-aware mode
chatbot.retrieval.code_deep_file_inventory_limit	18	Max files listed in code-aware inventory
chatbot.retrieval.lexical_retrieval	true	Blend exact-match retrieval with embedding retrieval
chatbot.retrieval.lexical_candidate_limit	24	Max lexical candidates gathered before merge/rerank
chatbot.retrieval.query_expansion	true	Use LLM to generate alternative search queries
chatbot.retrieval.expansion_max_queries	3	Number of alternative queries to generate
chatbot.retrieval.iterative_retrieval	true	Derive focused follow-up searches from early hits
chatbot.retrieval.iterative_max_followup_queries	2	Max follow-up queries used during iterative retrieval
chatbot.retrieval.graph_neighbor_expansion	true	Pull linked files and doc neighbors into the candidate set
chatbot.retrieval.graph_neighbor_max_files	6	Max linked files considered for graph-neighbor expansion
chatbot.retrieval.graph_neighbor_code_chunks_per_file	2	Code chunks per linked file during graph expansion
chatbot.retrieval.graph_neighbor_artifact_chunks_per_file	1	Artifact chunks per linked file during graph expansion
chatbot.retrieval.graph_neighbor_relationship_chunks_per_file	2	Relationship chunks per linked file during graph expansion
chatbot.retrieval.graph_neighbor_max_docs	4	Max linked docs pulled in during graph expansion
chatbot.retrieval.rerank	true	Use LLM to rerank retrieved chunks
chatbot.retrieval.rerank_candidate_limit	32	Max candidates sent to the reranker
chatbot.retrieval.rerank_candidate_limit_per_kind	8	Per-kind candidate cap before filling the global rerank pool
chatbot.retrieval.rerank_preview_chars	450	Characters of each chunk shown to the reranker
chatbot.retrieval.stitch_adjacent_code_chunks	true	Expand exact-match code hits with adjacent windows from the same file
chatbot.retrieval.stitch_max_adjacent_chunks	2	Max adjacent code windows stitched onto a top hit
chatbot.retrieval.deep_research_live_fallback	true	Allow /deep-research to inspect bounded live repo files when indexed retrieval is weak
chatbot.retrieval.live_fallback_max_files	6	Max repo files inspected during a deep-research live fallback
chatbot.retrieval.live_fallback_max_per_file	2	Max fallback snippets returned per inspected file
chatbot.retrieval.live_fallback_context_lines	12	Lines per fallback snippet around each exact match
chatbot.retrieval.deep_research_chunk_chars	3200	Max chars per evidence chunk passed into deep-research step answers
chatbot.retrieval.deep_research_top_k	10	Retrieved chunks per deep-research sub-question
Chunking
chatbot.chunking.code_chunk_lines	120	Lines per code chunk
chatbot.chunking.code_chunk_overlap	20	Overlap lines between code chunks
chatbot.chunking.artifact_chunk_lines	140	Lines per artifact chunk
chatbot.chunking.artifact_chunk_overlap	20	Overlap lines between artifact chunks
chatbot.chunking.max_doc_summary_chunks_per_page	4	Doc summary chunks extracted per page
chatbot.chunking.max_doc_summary_chars	4000	Max chars per doc summary chunk

Chatbot Provider Examples

The chatbot's answer and embeddings sections are configured independently from the main llm section, so you can use different providers for doc generation vs. chatbot.

Azure:

chatbot:
  answer:
    provider: "azure"
    model: "azure/gpt-4o-mini"
    base_url: "https://YOUR-RESOURCE.openai.azure.com/"
    api_version: "2024-08-01-preview"
    api_key_env: "DEEPDOC_CHAT_API_KEY"
  embeddings:
    provider: "azure"
    model: "azure/text-embedding-3-large"
    base_url: "https://YOUR-RESOURCE.openai.azure.com/"
    api_version: "2024-08-01-preview"
    api_key_env: "DEEPDOC_EMBED_API_KEY"

OpenAI:

chatbot:
  answer:
    provider: "openai"
    model: "gpt-4o-mini"
    api_key_env: "DEEPDOC_CHAT_API_KEY"
  embeddings:
    provider: "openai"
    model: "text-embedding-3-large"
    api_key_env: "DEEPDOC_EMBED_API_KEY"
    batch_size: 24                            # OpenAI supports larger batches

Google Gemini:

chatbot:
  answer:
    provider: "gemini"
    model: "gemini/gemini-2.0-flash"
    api_key_env: "DEEPDOC_CHAT_API_KEY"
  embeddings:
    backend: "litellm"
    provider: "gemini"
    model: "gemini/text-embedding-004"
    api_key_env: "DEEPDOC_EMBED_API_KEY"

Fully local embeddings with fastembed:

chatbot:
  embeddings:
    backend: "fastembed"
    fastembed_model: "nomic-ai/nomic-embed-text-v1.5"
    fastembed_batch_size: 4

Anthropic:

chatbot:
  answer:
    provider: "anthropic"
    model: "claude-3-5-sonnet-20241022"
    api_key_env: "DEEPDOC_CHAT_API_KEY"
  embeddings:
    provider: "openai"                        # Anthropic doesn't offer embedding models
    model: "text-embedding-3-large"
    api_key_env: "DEEPDOC_EMBED_API_KEY"

How Chatbot Indexing Works

During deepdoc generate, six corpora are built and stored in .deepdoc/chatbot/:

Corpus	Source	Description
Code chunks	All parsed source files	Code split by line count with overlap, tagged with symbols and file paths
Artifact chunks	Config files (Dockerfile, package.json, OpenAPI specs, etc.)	Non-code project files split similarly
Doc summary chunks	Generated documentation pages	First sections extracted from each generated MDX page
Doc full chunks	Generated documentation pages	Section-level chunks from the full generated MDX pages
Repo doc chunks	Repo-authored docs such as README.md, docs/, design notes, and notebooks	Raw repo documentation kept separate from generated pages
Relationship chunks	Imports, symbols, call graph, and graph-neighbor summaries	Lightweight graph-style retrieval context

deepdoc update incrementally syncs the chatbot indexes from the same commit-based update run — only changed files, repo-doc candidates, and regenerated doc pages are re-indexed.

Chatbot Query Pipeline

When a user asks a question, the backend runs a mode-aware retrieval pipeline:

1. Query expansion — In default/deep/code-aware mode, the LLM can generate alternative search queries to improve recall. Fast mode disables this by default.
2. Embedding — All queries are embedded using the configured embedding model.
3. Hybrid retrieval — FAISS similarity search and exact-match lexical search both gather candidates from each corpus.
4. Follow-up retrieval — The backend can derive focused second-pass searches and pull linked files/docs via graph-neighbor expansion. Fast mode can skip follow-up queries for lower latency.
5. Chunk stitching — Exact-match code hits can pull adjacent code windows from the same file so larger implementations survive chunk boundaries.
6. Reranking — In default/deep/code-aware mode, the LLM can rerank candidates for relevance. Fast mode disables this by default.
7. Prompt assembly — Query-type-aware budgets reserve space for the most important evidence types within the character budget.
8. Answer generation + continuity guard — The answer LLM produces a grounded response, and if the output appears truncated (for example ending on a dangling heading), DeepDoc retries with a continuation prompt so the response finishes cleanly.

POST /deep-research uses the same indexed corpora first, but it can also inspect a small bounded set of live repo files when exact-match evidence is missing from the index. This fallback respects the repo's exclude rules, skips oversized/binary files, and is only used in deep research mode.

POST /code-deep uses a code-heavy retrieval profile and returns an explicit file inventory plus step trace so users can see where evidence came from while answering file-oriented questions such as “where is auth defined?”.

POST /query, POST /deep-research, and POST /code-deep return response_mode in the payload (fast, deep, code_deep, or default) so clients can confirm which retrieval profile generated the result.

Chatbot API Endpoints

The generated chatbot_backend/ exposes five endpoints:

Health check:

GET /health → { "status": "ok" }

Query:

POST /query
{
  "question": "How does authentication work?",
  "history": [
    { "role": "user", "content": "What endpoints exist?" },
    { "role": "assistant", "content": "..." }
  ]
}

The response includes the answer text, code citations (file path + line range), artifact citations, and links to relevant generated doc pages.

/query is optimized for speed: it runs retrieval in fast mode (no LLM query expansion/rerank by default) and returns an answer plus citations.

Code-aware deep query:

POST /code-deep
{
  "question": "Where is authentication defined?",
  "history": [],
  "max_rounds": 4
}

/code-deep returns a code-aware answer plus trace and file_inventory fields so clients can show reasoning progress and files considered.

Code-aware live stream (SSE):

POST /code-deep/stream
{
  "question": "Where is authentication defined?",
  "history": [],
  "max_rounds": 4
}

/code-deep/stream emits trace events while researching, then a final result event and done.

Retrieve context only (no answer generation):

POST /query-context
{
  "question": "Where is reshipping implemented?",
  "history": []
}

/query-context returns selected citations/chunks only. Use this endpoint to inspect retrieval quality independently from answer generation.

Deploying the Chatbot

For local development, deepdoc serve handles everything automatically. For production:

1. Deploy the Fumadocs static site (site/out/) to any static host.
2. Deploy chatbot_backend/ separately to a Python-capable host.
3. Set chatbot.backend.base_url in .deepdoc.yaml to point at the deployed backend URL.
4. Rebuild the site so the frontend picks up the new backend URL: deepdoc deploy.

Cleaning Up Chatbot Files

deepdoc clean          # Removes chatbot_backend/, .deepdoc/chatbot/, and other generated state
deepdoc clean --yes    # Skip confirmation prompt

---

Supported Languages & Frameworks

Parsing (tree-sitter AST + regex fallback):

Language	Extensions	Extracts
Python	.py	Functions, classes, decorators, imports
JavaScript	.js, .jsx, .mjs, .cjs	Functions, classes, arrow functions, imports
TypeScript	.ts, .tsx	Same as JS + interfaces, type aliases
Go	.go	Functions, methods, structs, interfaces
PHP	.php	Functions, classes, methods, namespaces
Vue	.vue	SFC script symbols, props/emits/slots, router/store usage

High-confidence framework support (fixture-backed):

Framework	Language	Proven patterns
Laravel	PHP	Route::get(), grouped prefixes, middleware, resource expansion
Django / DRF	Python	path(), re_path(), @api_view, as_view(), DRF routers, @action
Express	JS/TS	Mounted routers via app.use(), nested prefixes, chained route() calls
Fastify	JS/TS	Plugin register(..., { prefix }), shorthand methods, route({ ... }), schema hints
Falcon	Python	app.add_route(), responder classes, imported resources, app middleware
Vue	Vue SFC	Component detection, defineProps, defineEmits, defineModel, defineSlots, router/store signals

Supported but not headline-high-confidence yet:

Framework	Language	Current coverage
Gin / Echo / Fiber	Go	Common route helpers (GET, POST, HandleFunc)

Runtime/background surface extraction:

Surface	Current coverage
Celery	Tasks, retry/queue hints, beat schedules, producers
Django	Management commands, signal receivers, Channels websocket consumers
Laravel	Queued jobs, listeners, events, scheduler registrations
JS/TS	node-cron, queue workers, agenda jobs, Socket.IO / websocket consumers
Go	Goroutine workers, AddFunc cron registrations, scheduler .Every(...).Do(...) patterns
Generic cron	Python crontab(...) style schedule declarations

---

Architecture

The current system is bucket-based.

Planner bucket types:

Type	Purpose
system	Architecture, setup, testing, deployment/ops, auth, shared middleware, observability
feature	Business workflows like checkout, refunds, order status, onboarding
endpoint	Endpoint-family or resource-level API docs
endpoint_ref	Legacy or OpenAPI-backed single-endpoint page; scanned runtime endpoints are grouped into endpoint-family docs by default
integration	Third-party systems like payment, warehouse, delivery, webhook providers
database	Cross-cutting database/schema/data-layer documentation

Five implemented phases:

1. Repository scan/indexing
   • Parse supported source files
   • Detect endpoints, config files, setup artifacts, runtime surfaces, integrations, and OpenAPI specs
   • Record file sizes, symbols, imports, config impacts, and raw scan summaries
2. Multi-step planning
   • Classify repo artifacts
   • Propose system/feature/endpoint/integration/database buckets
   • Assign files, symbols, and artifacts into the final plan
3. Generation engine
   • Build evidence packs for buckets
   • Generate pages in batches with parallel workers
   • Enrich grouped endpoint-family pages with scanned endpoint details and stage OpenAPI-backed API pages when a spec exists
   • Validate output for file, route, runtime, config, and integration grounding
   • Degrade gracefully on failures
   • Persist quality status so invalid/degraded pages are visible after a run
4. Persistence
   • Persist plan, file map, scan cache, and generation ledger in .deepdoc/
   • Keep enough state for updates, staleness detection, and cleanup
5. Smart update
   • Choose incremental update vs targeted replan vs full replan
   • Refresh only stale docs when safe
   • Rebuild affected docs after structural repo changes

---

Generated Files

After running deepdoc generate, you'll find:

your-repo/
├── .deepdoc.yaml              # Config
├── .deepdoc/                  # Canonical persisted state
│   ├── plan.json               # Bucket plan
│   ├── scan_cache.json         # Lightweight scan snapshot
│   ├── ledger.json             # Generated-page ledger
│   ├── file_map.json           # file → bucket/page mapping
│   ├── state.json              # last synced commit + update status
│   └── sync_receipt.json       # latest update/generate sync receipt
├── .deepdoc_manifest.json     # Legacy source hash manifest
├── .deepdoc_plan.json         # Legacy compatibility plan file
├── .deepdoc_file_map.json     # Legacy compatibility file map
├── docs/                       # Generated MDX pages
│   ├── index.mdx
│   ├── architecture.mdx
│   ├── setup-and-configuration.mdx
│   ├── orders-api.mdx
│   ├── get-api-v1-orders.mdx
│   └── ...
└── site/                       # Generated Fumadocs app
    ├── app/
    ├── components/
    ├── lib/
    ├── openapi/                # Staged OpenAPI assets (when a spec exists)
    ├── public/
    └── out/                    # Static export after `deepdoc deploy`

---

GitHub Actions CI/CD

Use GitHub Pages Actions when you want to publish the already-generated static export from site/out/:

# .github/workflows/github-pages.yml
name: Deploy GitHub Pages

on:
  push:
    branches: [main]
  workflow_dispatch:

permissions:
  contents: read
  pages: write
  id-token: write

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/configure-pages@v5
      - run: |
          test -d site/out
          test -f site/out/index.html
      - uses: actions/upload-pages-artifact@v3
        with:
          path: site/out

  deploy:
    needs: build
    runs-on: ubuntu-latest
    environment:
      name: github-pages
    steps:
      - id: deployment
        uses: actions/deploy-pages@v4

This workflow publishes the committed site/out/ directory directly, so GitHub Pages does not need a docs/ folder.

If you want to regenerate docs on every push before deploying them, use a separate workflow like this:

# .github/workflows/docs.yml
name: Update And Deploy Documentation

on:
  push:
    branches: [main]
  workflow_dispatch:

jobs:
  update-docs:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pages: write
      id-token: write

    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0    # Full history needed for git diff

      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - uses: actions/setup-node@v4
        with:
          node-version: "20"

      - name: Install dependencies
        run: |
          pip install deepdoc   # or: pip install "deepdoc[chatbot]" if you use chatbot features

      - name: Update and deploy docs
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          deepdoc update --strict-quality --deploy

      - name: Upload Pages artifact
        uses: actions/upload-pages-artifact@v3
        with:
          path: site/out

  deploy:
    needs: update-docs
    runs-on: ubuntu-latest
    environment:
      name: github-pages
    steps:
      - id: deployment
        uses: actions/deploy-pages@v4

Add your API key to repo Settings → Secrets → Actions → ANTHROPIC_API_KEY.

If you self-host and just want docs kept fresh without deploying to GitHub Pages, use this minimal variant instead — it commits the refreshed docs/ and .deepdoc/ state back to the branch:

# .github/workflows/deepdoc-refresh.yml
name: Refresh Docs

on:
  push:
    branches: [main]

jobs:
  refresh:
    runs-on: ubuntu-latest
    permissions:
      contents: write

    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0    # Full history needed for git diff

      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - uses: actions/setup-node@v4
        with:
          node-version: "20"

      - name: Install deepdoc
        run: pip install deepdoc

      - name: Refresh docs
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: deepdoc update --strict-quality

      - name: Commit updated docs
        run: |
          git config user.name "github-actions[bot]"
          git config user.email "github-actions[bot]@users.noreply.github.com"
          git add docs/ .deepdoc/
          git diff --cached --quiet || git commit -m "chore: refresh docs [skip ci]"
          git push

The [skip ci] tag on the commit message prevents the workflow from triggering itself again.

---

Contributing & Releases

Release flow for the Python package and the VS Code extension, plus repo layout and test-running notes, lives in CONTRIBUTING.md. Both tracks publish automatically when you push a version bump to main.

---

Typical Workflow

First time:

cd your-repo
deepdoc init --provider anthropic
export ANTHROPIC_API_KEY=sk-ant-...
deepdoc generate
deepdoc serve                      # Preview at localhost:3000
deepdoc deploy                     # Export a static site to site/out/

Every time you update code:

git add . && git commit -m "feat: new feature"
deepdoc update                     # Only regenerates affected pages
deepdoc deploy                     # Or use --deploy flag with update

Full refresh after planner / prompt / generator changes:

deepdoc generate --force

Wipe docs and rebuild from zero:

deepdoc generate --clean --yes

Switch LLM mid-project:

deepdoc config set llm.provider openai
deepdoc config set llm.model gpt-4o
export OPENAI_API_KEY=sk-...
deepdoc generate --force           # Full regen with new model

---

Requirements

• Python 3.10+
• Node 18+ on PATH — used at generate time for MDX compile validation, and at serve/deploy time for the Fumadocs site
• Git (for deepdoc update and deepdoc deploy)
• An LLM API key (or Ollama running locally)

---

License

MIT