hld-generator 0.3.1

Language-agnostic High-Level Design generator powered by Tree-sitter + LLM

HLD Generator

Language-agnostic High-Level Design document generator powered by Tree-sitter + LLM.

Point it at any codebase — Python, JavaScript, TypeScript, Java, Go, Rust, C/C++, Ruby, C#, Kotlin, Swift, and more — and get a complete HLD with architecture diagrams, component breakdowns, and dependency analysis.

How It Works

Codebase → Scanner → Tree-sitter AST Parser → Code Graph → LLM Analysis → HLD Document
                          ↓ (fallback)
                     Regex Parser

1. Scan — Discovers source files, filters out vendor/generated code
2. Parse — Extracts classes, functions, imports, endpoints using Tree-sitter (with regex fallback). Parsing runs in parallel using a thread pool
3. Graph — Builds a dependency graph with NetworkX, identifies entry points and hub modules
4. Analyse — Sends structured context to an LLM (Claude or GPT) for semantic understanding, or runs static fallback analysis with --provider none
5. Render — Generates Markdown report, interactive HTML viewer, and/or structured JSON with a React-based viewer

Installation

# Install from PyPI (coming soon) or directly from GitHub
pip install git+https://github.com/harsh-vishnoi/hld-generator.git

# With LLM support (Anthropic Claude / OpenAI GPT)
pip install "hld-generator[llm] @ git+https://github.com/harsh-vishnoi/hld-generator.git"

# For development
git clone https://github.com/harsh-vishnoi/hld-generator.git
cd hld-generator
pip install -e ".[dev,llm]"

Requirements: Python 3.9+

hld not found after install? pip may install the script to a directory not on your PATH (e.g. ~/.local/bin or ~/Library/Python/3.x/bin). Either add that directory to your PATH:

# macOS
echo 'export PATH="$PATH:$HOME/Library/Python/3.9/bin"' >> ~/.zshrc && source ~/.zshrc

# Linux
echo 'export PATH="$PATH:$HOME/.local/bin"' >> ~/.bashrc && source ~/.bashrc

Or run via Python directly:

python -m hld_generator ./my-project --provider none

Updating to latest version:

pip install --upgrade git+https://github.com/harsh-vishnoi/hld-generator.git

Quick Start

# With Anthropic Claude (recommended)
export ANTHROPIC_API_KEY=sk-ant-...
hld ./my-project

# With OpenAI
export OPENAI_API_KEY=sk-...
hld ./my-project --provider openai

# Without LLM (graph-only static analysis, no API key needed)
hld ./my-project --provider none

# Generate interactive HTML viewer
hld ./my-project --html

# Generate JSON data + React viewer
hld ./my-project --json

# Generate all output formats
hld ./my-project --html --json

# Scan a single file
hld ./main.py --provider none

# Custom output directory
hld ./my-project -o ./docs/architecture

# Include test files + verbose logging
hld ./my-project --include-tests -v

# Disable plugin auto-loading
hld ./my-project --no-plugins

Output

By default, the tool generates three files in ./hld_output/:

File	Description
hld_report.md	Complete HLD document with overview, components, data flow, tech stack, and architecture diagram
architecture.mmd	Standalone Mermaid diagram file (renderable in GitHub, VS Code, etc.)
graph_summary.md	Raw code graph statistics — modules, packages, entry points, hub modules, API endpoints

Interactive Viewers

HTML Viewer (--html)

Generates a single self-contained HTML file with an interactive dashboard:

hld ./my-project --html
open ./hld_output/hld_viewer.html

• Zero dependencies — works offline, just open in a browser
• Tabbed interface: Report view + Architecture diagram
• Mermaid diagrams with pan/zoom
• Dark mode toggle
• Collapsible sections and search

React Viewer (--json)

Generates structured JSON data plus a full React-based interactive viewer:

hld ./my-project --json
open ./hld_output/viewer.html

Output File	Description
hld_data.json	Raw JSON data (for programmatic use, CI/CD pipelines, external tools)
viewer.html	Interactive React viewer with data embedded
assets/	JS/CSS bundles for the viewer

The React viewer includes:

• Mind Map — expandable tree with click-to-expand nodes, side panel details, and keyboard navigation
• File View — Mermaid architecture diagram with pan/zoom
• Package View — Package-level dependency diagram
• Full keyboard accessibility (arrow keys, Enter/Space, Tab focus trap)
• Dark mode support
• Search with keyboard shortcut (/)

Note: The React viewer is bundled with the pip package — no Node.js or npm required.

CLI Reference

hld <target> [options]

Flag	Default	Description
target	(required)	Path to a source file or directory to analyse
-o, --output	./hld_output	Output directory
--provider	anthropic	LLM provider: anthropic, openai, none, or a plugin-registered name
--model	auto	LLM model name (defaults: claude-sonnet-4-20250514 / gpt-4o)
--format	markdown	Output format: built-in markdown or a plugin-registered renderer name
--html	off	Generate an interactive HTML viewer (hld_viewer.html)
--json	off	Generate structured JSON (hld_data.json) + React viewer (viewer.html)
--include-tests	off	Include test files in the analysis
--max-files	500	Maximum number of files to scan
--max-file-size	512000	Maximum file size in bytes
--plugins-dir	none	Directory containing plugin .py files to load
--no-plugins	off	Disable all plugin loading (explicit and auto-discovery)
-v, --verbose	off	Enable debug logging
--version	—	Show version

Exit codes: 0 = success, 1 = no files found, 2 = degraded (LLM failed but fallback output was produced).

API keys: Set via environment variables ANTHROPIC_API_KEY or OPENAI_API_KEY. The --api-key flag is deprecated (visible in process list).

Supported Languages

Language	Tree-sitter	Regex Fallback	Endpoint Detection
Python	yes	yes	Flask, FastAPI, Django
JavaScript	yes	yes	Express
TypeScript	yes	yes	Express
Java	yes	yes	Spring Boot
Go	yes	yes	net/http, Gin, Chi
Rust	yes	yes	—
C	yes	yes	—
C++	yes	yes	—
Ruby	yes	yes	—
C#	—	yes	—
Kotlin	—	yes	—
Swift	—	yes	—
PHP	—	yes	—
Scala	—	yes	—

Languages without Tree-sitter grammars automatically use the regex fallback parser. Even completely unknown languages get basic extraction via generic patterns.

Testing

# Run the full test suite (170 tests)
python -m pytest tests/ -q

# Run with verbose output
python -m pytest tests/ -v

# Lint check
python -m ruff check hld_generator/

You can also run a full end-to-end validation:

# Static analysis (no API key needed)
hld ./my-project --provider none --json -v

# Verify output files were created
ls -la ./hld_output/

# Open the interactive viewer
open ./hld_output/viewer.html

Test Coverage

Suite	Tests	Scope
test_quick.py	7	Language map, config defaults, regex parser (Python/JS/Go/Java), file scanner
test_comprehensive_core.py	—	Config, scanner, regex parser, tree-sitter parser, graph builder, LLM fallback, renderer
test_comprehensive_fixes.py	—	Audit fixes, plugin system, entry point detection, integration, edge cases
test_json_renderer.py	16	JSON renderer output validation, edge cases, Unicode, graph serialization
Total	170	Full-stack coverage: parsing, graph, analysis, rendering, plugins

Architecture

hld_generator/
├── __init__.py            # Package version
├── __main__.py            # python -m hld_generator entry point
├── cli.py                 # CLI entry point & pipeline orchestrator
├── config.py              # Configuration, language maps, constants
├── scanner.py             # File discovery & filtering
├── parsers/
│   ├── base.py            # Data structures (ParsedFile, ParsedEntity, ImportInfo)
│   ├── manager.py         # Parser facade (auto-selects tree-sitter or regex, parallel parse_all)
│   ├── tree_sitter_parser.py   # Tree-sitter AST parser
│   └── regex_parser.py    # Regex fallback parser
├── graph.py               # NetworkX dependency graph builder
├── llm.py                 # LLM client (Anthropic + OpenAI + plugin dispatch)
├── fallback.py            # Static fallback analyser (no LLM needed)
├── renderer.py            # Markdown + Mermaid output renderer
├── html_renderer.py       # Self-contained interactive HTML viewer
├── json_renderer.py       # JSON data + React viewer output
├── plugins.py             # Plugin registry & hook system
├── _networkx_shim/        # Lightweight NetworkX fallback for offline use
│   └── __init__.py
└── viewer/                # Bundled React frontend (built from frontend/)
    ├── index.html
    ├── favicon.svg
    └── assets/            # JS/CSS bundles

Frontend Development

The React viewer source lives in frontend/ and is built with Vite + React + TypeScript + Tailwind CSS. To develop the frontend:

cd frontend
npm install
npm run dev          # Start dev server at http://localhost:5173
npm run build        # Build for production → dist/

After building, copy the output to the Python package:

rm -rf hld_generator/viewer/assets
cp -R frontend/dist/* hld_generator/viewer/

Plugin System

HLD Generator supports plugins for custom parsers, LLM providers, renderers, and pipeline hooks.

Loading Plugins

Plugins are loaded from:

1. --plugins-dir <path> — explicitly specified directory
2. .hld_plugins/ — auto-discovered next to the target directory

Each .py file in the plugin directory is loaded automatically (files starting with _ are skipped).

Use --no-plugins to disable all plugin loading.

Plugin Types

Custom parser — add support for a new language:

from hld_generator.plugins import registry

@registry.register_parser("swift")
class SwiftParser:
    def parse_file(self, file_path, language):
        # Return a ParsedFile
        ...

Custom LLM provider — use a different LLM backend:

@registry.register_llm_provider("ollama")
class OllamaProvider:
    def call(self, context: str, system_prompt: str) -> str:
        # Return raw LLM response text
        ...

Then use it: hld ./project --provider ollama

Custom renderer — output in a different format:

@registry.register_renderer("html")
class HTMLRenderer:
    def render(self, analysis, code_graph, output_dir) -> list[Path]:
        # Write files and return their paths
        ...

Then use it: hld ./project --format html

Pipeline hooks — modify data between pipeline stages:

@registry.register_hook("post_parse")
def enrich(parsed_files):
    # Modify and return parsed_files
    return parsed_files

Available Hook Points

Hook	Receives	Returns
pre_scan	config	config
post_scan	scanned_files	scanned_files
pre_parse	scanned_files	scanned_files
post_parse	parsed_files	parsed_files
pre_graph	parsed_files	parsed_files
post_graph	code_graph	code_graph
pre_llm	code_graph	code_graph
post_llm	analysis	analysis
pre_render	analysis, code_graph	(analysis, code_graph)
post_render	output_files	output_files

Other Extension Points

Custom endpoint patterns:

registry.register_endpoint_pattern(
    r'@MyFramework\.route\("([^"]+)"\)',
    name="MyFramework"
)

Custom language file extensions:

registry.register_language(".hx", "haxe")

Extending (Without Plugins)

Add a new language:

1. Add extension mapping in config.py → LANGUAGE_MAP
2. Add regex patterns in parsers/regex_parser.py → _PATTERNS
3. (Optional) Add tree-sitter grammar to config.py → TREE_SITTER_GRAMMARS and queries to tree_sitter_parser.py → _QUERIES

Add framework endpoint detection: Add patterns to parsers/regex_parser.py → _ENDPOINT_PATTERNS

License

MIT