cjm-nbdev-overview 0.0.18

Automated documentation and visualization tools for nbdev projects.

cjm-nbdev-overview

Install

pip install cjm-nbdev-overview

How to use

Automatic Module Documentation

This project includes functionality to automatically update your index.ipynb with comprehensive module documentation. You can either:

1. Use the CLI command:

   nbdev-overview update-index
   

2. Use the Python API:

   from cjm_nbdev_overview.api_docs import update_index_module_docs
   update_index_module_docs()
   

This will add a “Module Overview” section to your index.ipynb containing detailed documentation for all modules in your project.

Project Structure

nbs/
├── api_docs.ipynb     # Generate module overviews with formatted signatures for nbdev projects
├── cli.ipynb          # CLI commands for nbdev project overview generation and analysis
├── core.ipynb         # Core utilities and data models for nbdev project overview generation
├── dependencies.ipynb # Analyze cross-notebook imports and generate Mermaid.js dependency diagrams
├── generators.ipynb   # Auto-generate folder_name.ipynb notebooks for nbdev project organization
├── parsers.ipynb      # Parse notebook metadata, content, and extract function/class signatures with docments
└── tree.ipynb         # Generate tree visualizations for nbdev project structure

Total: 7 notebooks

Module Dependencies

graph LR
    api_docs[api_docs<br/>API Documentation Generation]
    cli[cli<br/>Command-Line Interface]
    core[core<br/>Core Utilities]
    dependencies[dependencies<br/>Dependency Analysis and Visualization]
    generators[generators<br/>Auto-generation Utilities]
    parsers[parsers<br/>Notebook and Module Parsing]
    tree[tree<br/>Directory Tree Visualization]

    api_docs --> parsers
    api_docs --> tree
    api_docs --> dependencies
    api_docs --> core
    cli --> tree
    cli --> api_docs
    cli --> parsers
    cli --> dependencies
    dependencies --> dependencies
    dependencies --> parsers
    dependencies --> core
    generators --> tree
    generators --> core
    parsers --> tree
    parsers --> core
    tree --> core

16 cross-module dependencies detected

CLI Reference

nbdev-overview Command

usage: nbdev-overview [-h]
                      {tree,api,deps,overview,update-index,update-comprehensive}
                      ...

Generate comprehensive overviews for nbdev projects

positional arguments:
  {tree,api,deps,overview,update-index,update-comprehensive}
                        Available commands
    tree                Generate directory tree visualization
    api                 Generate API documentation
    deps                Analyze module dependencies
    overview            Generate complete project overview
    update-index        Update index.ipynb with module documentation
    update-comprehensive
                        Comprehensive update of index.ipynb with all sections

options:
  -h, --help            show this help message and exit

For detailed help on any command, use nbdev-overview <command> --help.

Module Overview

Detailed documentation for each module in the project:

API Documentation Generation (api_docs.ipynb)

Generate module overviews with formatted signatures for nbdev projects

Import

from cjm_nbdev_overview.api_docs import (
    format_function_doc,
    format_class_doc,
    format_variable_doc,
    generate_module_overview,
    generate_project_api_docs,
    update_index_module_docs,
    add_project_structure_section,
    add_dependencies_section,
    add_cli_reference_section,
    update_index_comprehensive
)

Functions

def format_function_doc(func: FunctionInfo,             # Function information
                       indent: str = ""                 # Indentation prefix
                       ) -> str:                        # Formatted documentation
    "Format a function with its signature for documentation"

def format_class_doc(cls: ClassInfo                     # Class information
                    ) -> str:                           # Formatted documentation
    "Format a class with its signature and methods for documentation"

def format_variable_doc(var: VariableInfo               # Variable information
                       ) -> str:                        # Formatted documentation
    "Format a variable for documentation"

def _generate_module_header(module: ModuleInfo          # Module information
                          ) -> List[str]:               # Header lines
    "Generate module title and description lines"

def _generate_import_statement(module: ModuleInfo       # Module information
                             ) -> List[str]:            # Import statement lines
    "Generate import statement lines for a module"

def _filter_module_items(module: ModuleInfo,            # Module information
                        show_all: bool = False          # Show all items including private
                        ) -> tuple:                     # (functions, classes, variables)
    "Filter module items based on show_all and is_exported flags"

def _generate_functions_section(functions: List[FunctionInfo]   # List of functions
                              ) -> List[str]:                   # Section lines
    "Generate the functions section of module documentation"

def _generate_classes_section(classes: List[ClassInfo]          # List of classes
                            ) -> List[str]:                     # Section lines
    "Generate the classes section of module documentation"

def _generate_variables_section(variables: List[VariableInfo]   # List of variables
                              ) -> List[str]:                   # Section lines
    "Generate the variables section of module documentation"

def generate_module_overview(module: ModuleInfo,        # Module information
                           show_all: bool = False       # Show all items including private
                           ) -> str:                    # Module overview markdown
    "Generate a markdown overview for a module"

def generate_project_api_docs(path: Path = None,        # Project path (defaults to nbs_path)
                            show_all: bool = False      # Show all items including private
                            ) -> str:                   # Full API documentation
    "Generate API documentation for all modules in a project"

def _filter_cells_removing_sections(cells: List,               # List of notebook cells
                                   start_marker: str            # Section marker to remove
                                   ) -> List:                   # Filtered cells
    "Remove all cells from a section marked by start_marker until the next ## section"

def _sort_notebooks_by_prefix(notebooks: List[Path]             # List of notebook paths
                             ) -> List[Path]:                   # Sorted notebook paths
    "Sort notebooks by their numeric prefix, putting non-numbered notebooks at the end"

def _get_notebooks_with_exports(notebooks: List[Path]          # List of notebook paths
                               ) -> List[Path]:                 # Notebooks with exported content
    "Filter notebooks to only include those with exported content"

def _generate_module_overview_cells(notebooks: List[Path]      # List of notebook paths
                                   ) -> List:                   # List of notebook cells
    "Generate markdown cells containing module overview documentation"

def update_index_module_docs(index_path: Path = None,          # Path to index.ipynb (defaults to nbs/index.ipynb)
                           start_marker: str = "## Module Overview"  # Marker to identify module docs section
                           ) -> None:                          # Updates index.ipynb in place
    "Update the module documentation section in index.ipynb"

def add_project_structure_section(index_path: Path = None,      # Path to index.ipynb
                                 marker: str = "## Project Structure",  # Section marker
                                 exclude_index: bool = True     # Exclude index.ipynb from tree
                                 ) -> str:                       # Generated structure content
    "Generate project structure tree content for index.ipynb"

def add_dependencies_section(index_path: Path = None,           # Path to index.ipynb
                           marker: str = "## Module Dependencies", # Section marker
                           direction: str = "LR"                # Diagram direction
                           ) -> str:                            # Generated dependencies content
    "Generate module dependencies diagram content for index.ipynb"

def add_cli_reference_section(marker: str = "## CLI Reference"  # Section marker
                            ) -> str:                           # Generated CLI content
    "Generate CLI reference content for index.ipynb based on project's console scripts"

def update_index_comprehensive(index_path: Path = None,         # Path to index.ipynb
                              include_structure: bool = True,  # Include project structure
                              include_dependencies: bool = True, # Include module dependencies
                              include_cli: bool = True,         # Include CLI reference
                              include_modules: bool = True      # Include module documentation
                              ) -> None:                        # Updates index.ipynb in place
    "Comprehensively update index.ipynb with project structure, dependencies, CLI, and modules"

Command-Line Interface (cli.ipynb)

CLI commands for nbdev project overview generation and analysis

Import

from cjm_nbdev_overview.cli import (
    tree_cmd,
    api_cmd,
    deps_cmd,
    overview_cmd,
    update_index_cmd,
    update_comprehensive_cmd,
    main
)

Functions

def tree_cmd(args:argparse.Namespace  # Parsed command line arguments
            ):                         # None
    "Generate tree visualization for nbdev project"

def api_cmd(args:argparse.Namespace  # Parsed command line arguments
           ):                         # None
    "Generate API documentation for nbdev project"

def deps_cmd(args:argparse.Namespace  # Parsed command line arguments
            ):                         # None
    "Analyze and visualize module dependencies"

def overview_cmd(args:argparse.Namespace  # Parsed command line arguments
                ):                         # None
    "Generate complete project overview"

def update_index_cmd(args:argparse.Namespace  # Parsed command line arguments
                    ):                         # None
    "Update index.ipynb with module documentation"

def update_comprehensive_cmd(args:argparse.Namespace  # Parsed command line arguments
                            ):                         # None
    "Comprehensively update index.ipynb with all sections"

def main():  # None
    "Main CLI entry point for nbdev-overview"

Core Utilities (core.ipynb)

Core utilities and data models for nbdev project overview generation

Import

from cjm_nbdev_overview.core import (
    NotebookInfo,
    DirectoryInfo,
    get_notebook_files,
    get_subdirectories,
    read_notebook,
    get_cell_source
)

Functions

def get_notebook_files(path: Path = None,           # Directory to search (defaults to nbs_path)
                      recursive: bool = True        # Search subdirectories
                      ) -> List[Path]:              # List of notebook paths
    "Get all notebook files in a directory"

def get_subdirectories(path: Path = None,           # Directory to search (defaults to nbs_path)
                      recursive: bool = False       # Include all nested subdirectories
                      ) -> List[Path]:              # List of directory paths
    "Get subdirectories in a directory"

def read_notebook(path: Path                    # Path to notebook file
                 ) -> Dict[str, Any]:           # Notebook content as dict
    "Read a notebook file and return its content"

def get_cell_source(cell: Dict[str, Any]        # Notebook cell
                   ) -> str:                    # Cell source as string
    "Get source from a notebook cell"

Classes

@dataclass
class NotebookInfo:
    "Information about a single notebook"
    
    path: Path  # Path to the notebook file
    name: str  # Notebook filename without extension
    title: Optional[str]  # H1 title from first cell
    description: Optional[str]  # Blockquote description from first cell
    export_module: Optional[str]  # Module name from default_exp
    
    def relative_path(self) -> Path:       # Path relative to nbs directory
        "Get path relative to nbs directory"

@dataclass
class DirectoryInfo:
    "Information about a directory in the nbs folder"
    
    path: Path  # Path to the directory
    name: str  # Directory name
    notebook_count: int = 0  # Number of notebooks in directory
    description: Optional[str]  # Description from folder's main notebook
    subdirs: List[DirectoryInfo] = field(...)  # Subdirectories
    notebooks: List[NotebookInfo] = field(...)  # Notebooks in this directory
    
    def total_notebook_count(self) -> int:          # Total notebooks including subdirs
        "Get total notebook count including subdirectories"

Dependency Analysis and Visualization (dependencies.ipynb)

Analyze cross-notebook imports and generate Mermaid.js dependency diagrams

Import

from cjm_nbdev_overview.dependencies import (
    ModuleDependency,
    DependencyGraph,
    extract_project_imports,
    analyze_module_dependencies,
    build_dependency_graph,
    generate_mermaid_diagram,
    generate_dependency_matrix
)

Functions

def extract_project_imports(import_str: str,            # Import statement
                           project_name: str            # Project package name
                           ) -> Optional[ModuleDependency]:  # Dependency if internal
    "Extract project-internal imports from an import statement"

def analyze_module_dependencies(module: ModuleInfo,     # Module to analyze
                               project_name: str        # Project package name
                               ) -> List[ModuleDependency]:  # Dependencies found
    "Analyze a module's imports to find project-internal dependencies"

def build_dependency_graph(path: Path = None,           # Project path
                          project_name: Optional[str] = None  # Override project name
                          ) -> DependencyGraph:         # Dependency graph
    "Build a dependency graph for all modules in a project"

def generate_mermaid_diagram(graph: DependencyGraph,    # Dependency graph
                           direction: str = "TD",       # Diagram direction (TD/LR)
                           show_imports: bool = False   # Show imported names
                           ) -> str:                    # Mermaid diagram code
    "Generate a Mermaid.js dependency diagram from a dependency graph"

def generate_dependency_matrix(graph: DependencyGraph   # Dependency graph
                              ) -> str:                 # Markdown table
    "Generate a dependency matrix showing which modules depend on which"

Classes

@dataclass
class ModuleDependency:
    "Represents a dependency between modules"
    
    source: str  # Source module name
    target: str  # Target module name
    import_type: str  # Type of import (from/import)
    imported_names: List[str] = field(...)  # Specific names imported

@dataclass
class DependencyGraph:
    "Dependency graph for a project"
    
    modules: Dict[str, ModuleInfo] = field(...)  # Module name -> ModuleInfo
    dependencies: List[ModuleDependency] = field(...)  # All dependencies
    
    def add_module(self,
                       module:ModuleInfo  # Module to add to the graph
                       ):                  # None
        "Add a module to the dependency graph"
    
    def add_dependency(self,
                           dep:ModuleDependency  # Dependency to add
                           ):                     # None
        "Add a dependency to the graph"
    
    def get_module_dependencies(self, module_name: str  # Module to query
                                   ) -> List[ModuleDependency]:  # Dependencies
        "Get all dependencies for a specific module"
    
    def get_module_dependents(self, module_name: str    # Module to query
                                 ) -> List[ModuleDependency]:  # Dependents
        "Get all modules that depend on a specific module"

Auto-generation Utilities (generators.ipynb)

Auto-generate folder_name.ipynb notebooks for nbdev project organization

Import

from cjm_nbdev_overview.generators import (
    create_folder_notebook,
    generate_folder_notebook,
    generate_all_folder_notebooks,
    interactive_folder_notebook_generator
)

Functions

def create_folder_notebook(folder_path: Path,           # Path to folder
                          title: str,                   # Notebook title
                          description: str              # Folder description
                          ) -> List[NbCell]:            # List of notebook cells
    "Create cells for a folder notebook with proper nbdev structure"

def generate_folder_notebook(folder_path: Path,         # Path to folder
                           title: Optional[str] = None, # Custom title
                           description: Optional[str] = None, # Custom description
                           overwrite: bool = False      # Overwrite existing
                           ) -> Path:                   # Path to created notebook
    "Generate a folder_name.ipynb notebook for a folder"

def generate_all_folder_notebooks(base_path: Path = None, # Base path (defaults to nbs)
                                 recursive: bool = True,  # Include nested folders
                                 overwrite: bool = False, # Overwrite existing
                                 dry_run: bool = False    # Just show what would be created
                                 ) -> List[Path]:         # Created notebook paths
    "Generate folder notebooks for all folders that don't have them"

def interactive_folder_notebook_generator(base_path: Path = None  # Base path
                                        ) -> List[Path]:          # Created notebooks
    "Interactively generate folder notebooks with custom titles and descriptions"

Notebook and Module Parsing (parsers.ipynb)

Parse notebook metadata, content, and extract function/class signatures with docments

Import

from cjm_nbdev_overview.parsers import (
    FunctionInfo,
    VariableInfo,
    ClassInfo,
    ModuleInfo,
    extract_docments_signature,
    parse_function,
    parse_class,
    parse_variable,
    parse_code_cell,
    parse_notebook,
    parse_python_file
)

Functions

def extract_docments_signature(node: Union[ast.FunctionDef, ast.AsyncFunctionDef],  # AST function node
                              source_lines: List[str]                               # Source code lines
                              ) -> str:                                             # Function signature
    "Extract function signature with docments-style comments"

def _parse_decorators(node: Union[ast.ClassDef, ast.FunctionDef, ast.AsyncFunctionDef]  # AST node with decorators
                     ) -> List[str]:                                                    # List of decorator names
    "Parse decorators from an AST node"

def parse_function(node: Union[ast.FunctionDef, ast.AsyncFunctionDef],  # AST function node
                  source_lines: List[str],                             # Source code lines
                  is_exported: bool = False                            # Has #| export
                  ) -> FunctionInfo:                                   # Function information
    "Parse a function definition from AST"

def _parse_class_methods(node: ast.ClassDef,           # AST class node
                        source_lines: List[str],        # Source code lines
                        is_exported: bool = False       # Has #| export
                        ) -> List[FunctionInfo]:        # List of method information
    "Parse methods from a class definition"

def _parse_dataclass_attributes(node: ast.ClassDef,    # AST class node
                               source_lines: List[str], # Source code lines
                               is_exported: bool = False # Has #| export
                               ) -> List[VariableInfo]: # List of attribute information
    "Parse dataclass attributes from a class definition"

def _generate_class_signature(node: ast.ClassDef,      # AST class node
                             methods: List[FunctionInfo] # List of class methods
                             ) -> str:                  # Class signature
    "Generate a class signature including __init__ if present"

def parse_class(node: ast.ClassDef,                    # AST class node
               source_lines: List[str],                # Source code lines
               is_exported: bool = False               # Has #| export
               ) -> ClassInfo:                         # Class information
    "Parse a class definition from AST"

def parse_variable(node: Union[ast.Assign, ast.AnnAssign],    # AST assignment node
                  source_lines: List[str],                     # Source code lines
                  is_exported: bool = False                    # Has #| export
                  ) -> List[VariableInfo]:                     # Variable information
    "Parse variable assignments from AST"

def parse_code_cell(cell: Dict[str, Any]                       # Notebook code cell
                   ) -> Tuple[List[FunctionInfo], List[ClassInfo], List[VariableInfo], List[str]]:  # (functions, classes, variables, imports)
    "Parse a notebook code cell for functions, classes, variables, and imports"

def parse_notebook(path: Path                           # Path to notebook
                  ) -> ModuleInfo:                      # Module information
    "Parse a notebook file for module information"

def parse_python_file(path: Path                        # Path to Python file
                     ) -> ModuleInfo:                   # Module information
    "Parse a Python file for module information"

Classes

@dataclass
class FunctionInfo:
    "Information about a function"
    
    name: str  # Function name
    signature: str  # Full signature with docments
    docstring: Optional[str]  # Function docstring
    decorators: List[str] = field(...)  # List of decorators
    is_exported: bool = False  # Has #| export
    is_async: bool = False  # Is an async function
    source_line: Optional[int]  # Line number in source

@dataclass
class VariableInfo:
    "Information about a module-level variable"
    
    name: str  # Variable name
    value: Optional[str]  # String representation of value
    type_hint: Optional[str]  # Type annotation if present
    comment: Optional[str]  # Inline comment
    is_exported: bool = False  # Has #| export

@dataclass
class ClassInfo:
    "Information about a class"
    
    name: str  # Class name
    signature: str  # Class signature with __init__
    docstring: Optional[str]  # Class docstring
    methods: List[FunctionInfo] = field(...)  # Class methods
    decorators: List[str] = field(...)  # Class decorators
    attributes: List[VariableInfo] = field(...)  # Class attributes (for dataclasses)
    is_exported: bool = False  # Has #| export
    source_line: Optional[int]  # Line number in source

@dataclass
class ModuleInfo:
    "Information about a module (notebook or Python file)"
    
    path: Path  # Path to module
    name: str  # Module name
    title: Optional[str]  # H1 title from notebook
    description: Optional[str]  # Module description
    functions: List[FunctionInfo] = field(...)  # Functions in module
    classes: List[ClassInfo] = field(...)  # Classes in module
    variables: List[VariableInfo] = field(...)  # Variables in module
    imports: List[str] = field(...)  # Import statements

Directory Tree Visualization (tree.ipynb)

Generate tree visualizations for nbdev project structure

Import

from cjm_nbdev_overview.tree import (
    ALIGNMENT_BUFFER,
    strip_markdown_links,
    generate_tree_lines,
    generate_tree,
    extract_notebook_info,
    generate_tree_with_descriptions,
    generate_subdirectory_tree,
    get_tree_summary
)

Functions

def _directory_has_notebooks(path: Path,                        # Directory to check
                            exclude_index: bool = True          # Exclude index.ipynb from check
                            ) -> bool:                          # True if contains notebooks
    "Check if a directory contains any notebooks (directly or in subdirectories)"

def strip_markdown_links(text:str  # Text that may contain Markdown links
                         ) -> str:  # Text with links removed, keeping link text
    "Strip Markdown links from text, keeping only the link text"

def generate_tree_lines(path: Path,                         # Directory to visualize
                       prefix: str = "",                    # Line prefix for tree structure
                       is_last: bool = True,                # Is this the last item in parent
                       show_notebooks_only: bool = False,   # Only show notebooks, not directories
                       max_depth: Optional[int] = None,     # Maximum depth to traverse
                       current_depth: int = 0,              # Current depth in traversal
                       exclude_index: bool = True,          # Exclude index.ipynb from tree
                       exclude_empty: bool = True           # Exclude empty directories
                       ) -> List[str]:                      # Lines of tree output
    "Generate tree visualization lines for a directory"

def generate_tree(path: Path = None,                    # Directory to visualize (defaults to nbs_path)
                 show_notebooks_only: bool = False,     # Only show notebooks, not directories
                 max_depth: Optional[int] = None,       # Maximum depth to traverse
                 exclude_index: bool = True,            # Exclude index.ipynb from tree
                 exclude_empty: bool = True             # Exclude empty directories
                 ) -> str:                              # Tree visualization as string
    "Generate a tree visualization for a directory"

def extract_notebook_info(path: Path                    # Path to notebook file
                         ) -> NotebookInfo:             # Notebook information
    "Extract title and description from a notebook"

def generate_tree_with_descriptions(path: Path = None,              # Directory to visualize
                                   show_counts: bool = True,        # Show notebook counts for directories
                                   max_depth: Optional[int] = None, # Maximum depth to traverse
                                   exclude_index: bool = True,       # Exclude index.ipynb from tree
                                   exclude_empty: bool = True        # Exclude empty directories
                                   ) -> str:                        # Tree with descriptions
    "Generate tree visualization with descriptions from notebooks"

def _generate_nested_tree_lines(path: Path,                         # Directory to process
                               prefix: str = "",                    # Line prefix
                               show_counts: bool = True,            # Show notebook counts
                               max_depth: Optional[int] = None,     # Maximum depth
                               current_depth: int = 0,              # Current depth
                               exclude_index: bool = True,          # Exclude index.ipynb from tree
                               exclude_empty: bool = True           # Exclude empty directories
                               ) -> List[str]:                      # Tree lines
    "Generate tree lines for nested directory structure"

def generate_subdirectory_tree(subdir_path: Path,               # Path to subdirectory
                              show_descriptions: bool = True,   # Include notebook descriptions
                              exclude_empty: bool = True,       # Exclude empty directories
                              exclude_index: bool = True        # Exclude index.ipynb
                              ) -> str:                         # Tree visualization
    "Generate tree visualization for a specific subdirectory showing all notebooks"

def _generate_subdirectory_lines(item: Path,                    # Item to process
                                prefix: str,                    # Line prefix
                                is_last: bool,                  # Is last item
                                is_dir: bool,                   # Is directory
                                show_descriptions: bool,        # Show descriptions
                                depth: int,                     # Current depth
                                max_length: int = 0,            # Max length for alignment (calculated externally)
                                exclude_empty: bool = True,     # Exclude empty directories
                                exclude_index: bool = True      # Exclude index.ipynb
                                ) -> List[str]:                 # Tree lines
    "Generate tree lines for subdirectory visualization"

def get_tree_summary(path: Path = None              # Directory to analyze
                    ) -> str:                       # Summary string
    "Get summary statistics for notebooks in directory tree"

Variables

ALIGNMENT_BUFFER = 1