cjm-nbdev-overview 0.0.1

Automated documentation and visualization tools for nbdev projects.

cjm-nbdev-overview

How to use

Automatic Module Documentation

This project includes functionality to automatically update your index.ipynb with comprehensive module documentation. You can either:

1. Use the CLI command:

   nbdev-overview update-index
   

2. Use the Python API:

   from cjm_nbdev_overview.api_docs import update_index_module_docs
   update_index_module_docs()
   

This will add a “Module Overview” section to your index.ipynb containing detailed documentation for all modules in your project.

Project Structure

nbs/
├── 00_core.ipynb            # Core utilities and data models for nbdev project overview generation
├── 01_parsers.ipynb         # Parse notebook metadata, content, and extract function/class signatures with docments
├── 02_tree.ipynb            # Generate tree visualizations for nbdev project structure
├── 03_api_docs.ipynb        # Generate module overviews with formatted signatures for nbdev projects
├── 04_dependencies.ipynb    # Analyze cross-notebook imports and generate Mermaid.js dependency diagrams
├── 05_generators.ipynb      # Auto-generate folder_name.ipynb notebooks for nbdev project organization
└── 06_cli.ipynb             # CLI commands for nbdev project overview generation and analysis

Total: 8 notebooks

Module Dependencies

graph LR
    core[core<br/>Core Utilities]
    parsers[parsers<br/>Notebook and Module Parsing]
    tree[tree<br/>Directory Tree Visualization]
    api_docs[api_docs<br/>API Documentation Generation]
    dependencies[dependencies<br/>Dependency Analysis and Visualization]
    generators[generators<br/>Auto-generation Utilities]
    cli[cli<br/>Command-Line Interface]

    parsers --> core
    parsers --> tree
    tree --> core
    api_docs --> core
    api_docs --> parsers
    api_docs --> dependencies
    api_docs --> tree
    dependencies --> core
    dependencies --> parsers
    generators --> tree
    generators --> core
    cli --> tree
    cli --> api_docs
    cli --> parsers
    cli --> dependencies

    classDef default fill:#f9f9f9,stroke:#333,stroke-width:2px

15 cross-module dependencies detected

CLI Reference

nbdev-overview Command

usage: nbdev-overview [-h]
                      {tree,api,deps,overview,update-index,update-comprehensive}
                      ...

Generate comprehensive overviews for nbdev projects

positional arguments:
  {tree,api,deps,overview,update-index,update-comprehensive}
                        Available commands
    tree                Generate directory tree visualization
    api                 Generate API documentation
    deps                Analyze module dependencies
    overview            Generate complete project overview
    update-index        Update index.ipynb with module documentation
    update-comprehensive
                        Comprehensive update of index.ipynb with all sections

options:
  -h, --help            show this help message and exit

For detailed help on any command, use nbdev-overview <command> --help.

Module Overview

Detailed documentation for each module in the project:

Core Utilities (00_core.ipynb)

Core utilities and data models for nbdev project overview generation

Functions

def get_notebook_files(path: Path = None,           # Directory to search (defaults to nbs_path)
                      recursive: bool = True        # Search subdirectories
                      ) -> List[Path]:              # List of notebook paths
    "Get all notebook files in a directory"

def get_subdirectories(path: Path = None,           # Directory to search (defaults to nbs_path)
                      recursive: bool = False       # Include all nested subdirectories
                      ) -> List[Path]:              # List of directory paths
    "Get subdirectories in a directory"

def read_notebook(path: Path                    # Path to notebook file
                 ) -> Dict[str, Any]:           # Notebook content as dict
    "Read a notebook file and return its content"

def get_cell_source(cell: Dict[str, Any]        # Notebook cell
                   ) -> str:                    # Cell source as string
    "Get source from a notebook cell"

Classes

@dataclass
class NotebookInfo:
    "Information about a single notebook"
    
    def relative_path(self) -> Path:       # Path relative to nbs directory
        "Get path relative to nbs directory"

@dataclass
class DirectoryInfo:
    "Information about a directory in the nbs folder"
    
    def total_notebook_count(self) -> int:          # Total notebooks including subdirs
        "Get total notebook count including subdirectories"

Notebook and Module Parsing (01_parsers.ipynb)

Parse notebook metadata, content, and extract function/class signatures with docments

Functions

def extract_docments_signature(node: ast.FunctionDef,          # AST function node
                              source_lines: List[str]          # Source code lines
                              ) -> str:                        # Function signature
    "Extract function signature with docments-style comments"

def parse_function(node: ast.FunctionDef,               # AST function node
                  source_lines: List[str],              # Source code lines
                  is_exported: bool = False             # Has #| export
                  ) -> FunctionInfo:                    # Function information
    "Parse a function definition from AST"

def parse_class(node: ast.ClassDef,                    # AST class node
               source_lines: List[str],                # Source code lines
               is_exported: bool = False               # Has #| export
               ) -> ClassInfo:                         # Class information
    "Parse a class definition from AST"

def parse_variable(node: Union[ast.Assign, ast.AnnAssign],    # AST assignment node
                  source_lines: List[str],                     # Source code lines
                  is_exported: bool = False                    # Has #| export
                  ) -> List[VariableInfo]:                     # Variable information
    "Parse variable assignments from AST"

def parse_code_cell(cell: Dict[str, Any]                       # Notebook code cell
                   ) -> Tuple[List[FunctionInfo], List[ClassInfo], List[VariableInfo], List[str]]:  # TODO: Add return description
    "Parse a notebook code cell for functions, classes, variables, and imports"

def parse_notebook(path: Path                           # Path to notebook
                  ) -> ModuleInfo:                      # Module information
    "Parse a notebook file for module information"

def parse_python_file(path: Path                        # Path to Python file
                     ) -> ModuleInfo:                   # Module information
    "Parse a Python file for module information"

Classes

@dataclass
class FunctionInfo:
    "Information about a function"

@dataclass
class ClassInfo:
    "Information about a class"

@dataclass
class VariableInfo:
    "Information about a module-level variable"

@dataclass
class ModuleInfo:
    "Information about a module (notebook or Python file)"

Directory Tree Visualization (02_tree.ipynb)

Generate tree visualizations for nbdev project structure

Functions

def generate_tree_lines(path: Path,                         # Directory to visualize
                       prefix: str = "",                    # Line prefix for tree structure
                       is_last: bool = True,                # Is this the last item in parent
                       show_notebooks_only: bool = False,   # Only show notebooks, not directories
                       max_depth: Optional[int] = None,     # Maximum depth to traverse
                       current_depth: int = 0,              # Current depth in traversal
                       exclude_index: bool = True           # Exclude index.ipynb from tree
                       ) -> List[str]:                      # Lines of tree output
    "Generate tree visualization lines for a directory"

def generate_tree(path: Path = None,                    # Directory to visualize (defaults to nbs_path)
                 show_notebooks_only: bool = False,     # Only show notebooks, not directories
                 max_depth: Optional[int] = None,       # Maximum depth to traverse
                 exclude_index: bool = True             # Exclude index.ipynb from tree
                 ) -> str:                              # Tree visualization as string
    "Generate a tree visualization for a directory"

def extract_notebook_info(path: Path                    # Path to notebook file
                         ) -> NotebookInfo:             # Notebook information
    "Extract title and description from a notebook"

def generate_tree_with_descriptions(path: Path = None,              # Directory to visualize
                                   show_counts: bool = True,        # Show notebook counts for directories
                                   max_depth: Optional[int] = None, # Maximum depth to traverse
                                   exclude_index: bool = True       # Exclude index.ipynb from tree
                                   ) -> str:                        # Tree with descriptions
    "Generate tree visualization with descriptions from notebooks"

def _generate_nested_tree_lines(path: Path,                         # Directory to process
                               prefix: str = "",                    # Line prefix
                               show_counts: bool = True,            # Show notebook counts
                               max_depth: Optional[int] = None,     # Maximum depth
                               current_depth: int = 0,              # Current depth
                               exclude_index: bool = True           # Exclude index.ipynb from tree
                               ) -> List[str]:                      # Tree lines
    "Generate tree lines for nested directory structure"

def generate_subdirectory_tree(subdir_path: Path,               # Path to subdirectory
                              show_descriptions: bool = True    # Include notebook descriptions
                              ) -> str:                         # Tree visualization
    "Generate tree visualization for a specific subdirectory showing all notebooks"

def _generate_subdirectory_lines(item: Path,                    # Item to process
                                prefix: str,                    # Line prefix
                                is_last: bool,                  # Is last item
                                is_dir: bool,                   # Is directory
                                show_descriptions: bool,        # Show descriptions
                                depth: int                      # Current depth
                                ) -> List[str]:                 # Tree lines
    "Generate tree lines for subdirectory visualization"

def get_tree_summary(path: Path = None              # Directory to analyze
                    ) -> str:                       # Summary string
    "Get summary statistics for notebooks in directory tree"

API Documentation Generation (03_api_docs.ipynb)

Generate module overviews with formatted signatures for nbdev projects

Functions

def format_function_doc(func: FunctionInfo,             # Function informationFor the `add_cli_reference_section` function, remember that while we are using this project's functionality on itself, it is meant to be used on other nbdev projects as well. Therefore, we should first check the project's `settings.ini` file to see if the `console_scripts` value has anything, we can do that with `cfg = get_config(); cfg.console_scripts` (e.g., 'nbdev-overview=cjm_nbdev_overview.cli:main'). I believe we should then be able to use that to get the CLI reference info. Rather than hardcoding the CLI reference, we should programmatically generate it to account for changes.  
                       indent: str = ""                 # Indentation prefix
                       ) -> str:                        # Formatted documentation
    "Format a function with its signature for documentation"

def format_class_doc(cls: ClassInfo                     # Class information
                    ) -> str:                           # Formatted documentation
    "Format a class with its signature and methods for documentation"

def format_variable_doc(var: VariableInfo               # Variable information
                       ) -> str:                        # Formatted documentation
    "Format a variable for documentation"

def generate_module_overview(module: ModuleInfo,        # Module information
                           show_all: bool = False       # Show all items including private
                           ) -> str:                    # Module overview markdown
    "Generate a markdown overview for a module"

def generate_project_api_docs(path: Path = None,        # Project path (defaults to nbs_path)
                            show_all: bool = False      # Show all items including private
                            ) -> str:                   # Full API documentation
    "Generate API documentation for all modules in a project"

def update_index_module_docs(index_path: Path = None,   # Path to index.ipynb (defaults to nbs/index.ipynb)
                           start_marker: str = "## Module Overview",  # Marker to identify module docs section
                           ) -> None:                    # Updates index.ipynb in place
    "Update the module documentation section in index.ipynb"

def add_project_structure_section(index_path: Path = None,      # Path to index.ipynb
                                 marker: str = "## Project Structure",  # Section marker
                                 exclude_index: bool = True     # Exclude index.ipynb from tree
                                 ) -> str:                       # Generated structure content
    "Generate project structure tree content for index.ipynb"

def add_dependencies_section(index_path: Path = None,           # Path to index.ipynb
                           marker: str = "## Module Dependencies", # Section marker
                           direction: str = "LR"                # Diagram direction
                           ) -> str:                            # Generated dependencies content
    "Generate module dependencies diagram content for index.ipynb"

def add_cli_reference_section(marker: str = "## CLI Reference"  # Section marker
                            ) -> str:                           # Generated CLI content
    "Generate CLI reference content for index.ipynb based on project's console scripts"

def update_index_comprehensive(index_path: Path = None,         # Path to index.ipynb
                              include_structure: bool = True,  # Include project structure
                              include_dependencies: bool = True, # Include module dependencies
                              include_cli: bool = True,         # Include CLI reference
                              include_modules: bool = True      # Include module documentation
                              ) -> None:                        # Updates index.ipynb in place
    "Comprehensively update index.ipynb with project structure, dependencies, CLI, and modules"

Dependency Analysis and Visualization (04_dependencies.ipynb)

Analyze cross-notebook imports and generate Mermaid.js dependency diagrams

Functions

def extract_project_imports(import_str: str,            # Import statement
                           project_name: str            # Project package name
                           ) -> Optional[ModuleDependency]:  # Dependency if internal
    "Extract project-internal imports from an import statement"

def analyze_module_dependencies(module: ModuleInfo,     # Module to analyze
                               project_name: str        # Project package name
                               ) -> List[ModuleDependency]:  # Dependencies found
    "Analyze a module's imports to find project-internal dependencies"

def build_dependency_graph(path: Path = None,           # Project path
                          project_name: Optional[str] = None  # Override project name
                          ) -> DependencyGraph:         # Dependency graph
    "Build a dependency graph for all modules in a project"

def generate_mermaid_diagram(graph: DependencyGraph,    # Dependency graph
                           direction: str = "TD",       # Diagram direction (TD/LR)
                           show_imports: bool = False   # Show imported names
                           ) -> str:                    # Mermaid diagram code
    "Generate a Mermaid.js dependency diagram from a dependency graph"

def generate_dependency_matrix(graph: DependencyGraph   # Dependency graph
                              ) -> str:                 # Markdown table
    "Generate a dependency matrix showing which modules depend on which"

Classes

@dataclass
class ModuleDependency:
    "Represents a dependency between modules"

@dataclass
class DependencyGraph:
    "Dependency graph for a project"
    
    def add_module(
            self,
            module: ModuleInfo  # TODO: Add description
        ): # Add a module to the graph - TODO: Add type hint
        "Add a module to the dependency graph"
    
    def add_dependency(
            self,
            dep: ModuleDependency  # TODO: Add description
        ): # Add a dependency - TODO: Add type hint
        "Add a dependency to the graph"
    
    def get_module_dependencies(self, module_name: str  # Module to query
                                   ) -> List[ModuleDependency]:  # Dependencies
        "Get all dependencies for a specific module"
    
    def get_module_dependents(self, module_name: str    # Module to query
                                 ) -> List[ModuleDependency]:  # Dependents
        "Get all modules that depend on a specific module"

Auto-generation Utilities (05_generators.ipynb)

Auto-generate folder_name.ipynb notebooks for nbdev project organization

Functions

def create_folder_notebook(folder_path: Path,           # Path to folder
                          title: str,                   # Notebook title
                          description: str              # Folder description
                          ) -> List[NbCell]:            # List of notebook cells
    "Create cells for a folder notebook with proper nbdev structure"

def generate_folder_notebook(folder_path: Path,         # Path to folder
                           title: Optional[str] = None, # Custom title
                           description: Optional[str] = None, # Custom description
                           overwrite: bool = False      # Overwrite existing
                           ) -> Path:                   # Path to created notebook
    "Generate a folder_name.ipynb notebook for a folder"

def generate_all_folder_notebooks(base_path: Path = None, # Base path (defaults to nbs)
                                 recursive: bool = True,  # Include nested folders
                                 overwrite: bool = False, # Overwrite existing
                                 dry_run: bool = False    # Just show what would be created
                                 ) -> List[Path]:         # Created notebook paths
    "Generate folder notebooks for all folders that don't have them"

def interactive_folder_notebook_generator(base_path: Path = None  # Base path
                                        ) -> List[Path]:          # Created notebooks
    "Interactively generate folder notebooks with custom titles and descriptions"

Command-Line Interface (06_cli.ipynb)

CLI commands for nbdev project overview generation and analysis

Functions

def tree_cmd(args):                                     # Command line arguments
    "Generate tree visualization for nbdev project"
    # Get project path
    path = Path(args.path) if args.path else None
    
    # Determine exclude_index flag (default True, but --include-index overrides)
    exclude_index = not getattr(args, 'include_index', False)
    
    if args.basic
    "Generate tree visualization for nbdev project"

def api_cmd(
    args  # TODO: Add type hint and description
): # Command line arguments - TODO: Add type hint
    "Generate API documentation for nbdev project"

def deps_cmd(
    args  # TODO: Add type hint and description
): # Command line arguments - TODO: Add type hint
    "Analyze and visualize module dependencies"

def overview_cmd(
    args  # TODO: Add type hint and description
): # Command line arguments - TODO: Add type hint
    "Generate complete project overview"

def update_index_cmd(
    args  # TODO: Add type hint and description
): # Command line arguments - TODO: Add type hint
    "Update index.ipynb with module documentation"

def update_comprehensive_cmd(
    args  # TODO: Add type hint and description
): # Command line arguments - TODO: Add type hint
    "Comprehensively update index.ipynb with all sections"

def main()
    "Main CLI entry point for nbdev-overview"