Netwlens - Network Topology Visualization Tool

A Python package for network topology visualization and analysis that integrates with IP Fabric. Features comprehensive data validation, automatic root device selection, intelligent topology discovery, and robust error handling with clear feedback.

🏗️ Development Setup

Prerequisites

• Python 3.11 or higher
• IP Fabric instance with API access
• Graphviz (for visualization)

Installation (end users)

# From PyPI (recommended)
pip install netwlens
# or with uv
uv pip install netwlens

Installation for Development

# Clone the repository
git clone https://gitlab.com/ip-fabric/integrations/netwlens.git
cd netwlens

# Sync the project + dev dependencies into a local .venv (uv-managed)
uv sync --group dev

# Or with pip (creates .venv yourself)
python -m venv .venv && . .venv/bin/activate
pip install -e ".[dev]"   # see [dependency-groups] in pyproject.toml

# Install Graphviz
# macOS: brew install graphviz
# Ubuntu/Debian: sudo apt-get install graphviz
# Windows: Download from https://graphviz.org/download/

Environment Configuration

Create a .env file in the project root:

IPF_URL=https://your-ipfabric-instance.com
IPF_TOKEN=your-api-token
SNAPSHOT_ID=your-snapshot-id # or $last
SITE_NAME=your-site-name # Optional: default site for data collection
# Optional: IPF_VERIFY=false  # for self-signed certificates

Development Workflow

# Activate the uv-managed virtual environment
. .venv/bin/activate

# Run the CLI tool directly from source
python -m netwlens.cli

# Or run the standalone scripts
python collect_data.py
python visualize_topology.py

Quick Test Run

# Test site-specific data collection
netwlens collect --site London-3X --protocol cdp --output test-site

# Test all inventory data collection
netwlens collect --protocol cdp --output test-all

# Test collection using SITE_NAME from .env (NEW!)
netwlens collect --protocol cdp --output test-env

# Test visualization with automatic root selection
netwlens visualize --input netwlens_data/test-site

# Test visualization with manual root selection
netwlens visualize --input netwlens_data/test-site --root s3xasw02

📁 Project Structure

netwlens/
├── netwlens/                     # Main package
│   ├── __init__.py              # Package exports (DataCollector, TopologyVisualizer)
│   ├── cli.py                   # CLI interface
│   ├── data_collector.py        # DataCollector class
│   ├── topology_visualizer.py   # TopologyVisualizer class
│   ├── docs/                    # Package documentation
│   │   └── topology_graphviz.md # Detailed API docs
│   └── src/                     # Core modules
│       ├── extract_ipfabric_data.py # IP Fabric data extraction
│       ├── topology_graphviz.py     # Graphviz visualization engine
│       └── validator.py             # Comprehensive validation system
├── collect_data.py              # Standalone data collection script
├── visualize_topology.py        # Standalone visualization script
├── pyproject.toml               # Package configuration (PEP 621 + hatchling)
├── uv.lock                      # Resolved dependency lockfile
└── README.md                    # This file

Output Data Structure

When users run the tool, data is organized as:

netwlens_data/
├── mynetwork/                    # Data collection output
│   ├── inventory.json            # Device inventory
│   ├── connections_raw.json      # Raw connections from IP Fabric
│   ├── connections_normalized.json # Processed connections (enriched + deduplicated)
│   └── topology/                 # Visualization output
│       ├── topology_hierarchical.svg
│       └── view_svg_topology.html

🏗️ Architecture

Core Classes

DataCollector (netwlens/data_collector.py)

• High-level interface for IP Fabric data collection with comprehensive validation
• IP Fabric connection validation (detects VPN/connectivity issues)
• Site name validation with helpful error messages
• Parameter precedence handling (CLI overrides ENV variables)
• Wraps the core extraction functions from src/extract_ipfabric_data.py
• Handles output directory management and data processing pipeline

TopologyVisualizer (netwlens/topology_visualizer.py)

• High-level interface for topology visualization
• Wraps the Graphviz engine from src/topology_graphviz.py
• Manages output formats and layout algorithms

🛡️ Enhanced Validation System

Netwlens includes comprehensive validation to prevent silent failures and provide clear feedback:

Connection Validation

• IP Fabric connectivity check - Detects VPN disconnections or API issues
• Early failure detection - Validates connection before data collection begins
• Clear error messages - Actionable feedback for connection problems

Site Validation

• Site existence verification - Checks if specified site exists in IP Fabric
• Available sites listing - Shows valid site names when validation fails
• Parameter precedence - CLI --site parameter overrides SITE_NAME environment variable

Data Validation

• Empty data detection - Validates collected inventory and connections
• Actionable error messages - Clear guidance when data collection fails
• Success confirmation - Visible feedback when validation passes

Error Handling

• Clean error output - Minimal, user-friendly error messages
• Fast failure - Stops processing early when validation fails
• No verbose tracebacks - Clean exit with helpful error information

API Design

from netwlens import DataCollector, TopologyVisualizer

# Collect site-specific data with validation
collector = DataCollector()
data_dir = collector.collect_data(
    site="London-3X",  # Automatically validated against IP Fabric
    protocol="cdp",
    output="my-network"
)

# Or collect all inventory data
data_dir = collector.collect_data(
    site=None,  # None = all inventory
    protocol="cdp",
    output="all-network"
)

# Create visualization with automatic root selection
visualizer = TopologyVisualizer()
viz_path = visualizer.create_visualization(
    input_dir=data_dir,
    # root_device automatically selected based on connection count
    output_format="svg"
)

# Or specify root device manually
viz_path = visualizer.create_visualization(
    input_dir=data_dir,
    root_device="s3xasw02",
    output_format="svg"
)

CLI Commands

Data Collection with Validation

# Site-specific data collection (validates site exists)
netwlens collect --site London-3X --protocol cdp

# All inventory data collection
netwlens collect --protocol cdp

# Use SITE_NAME from .env file (validates connection and site)
netwlens collect --protocol cdp --output env-site

# Custom output directory
netwlens collect --site London-3X --protocol cdp --output my-site

# Available protocols: cef, cdp, lldp
netwlens collect --site London-3X --protocol lldp

Visualization

# Basic visualization with automatic root device selection
netwlens visualize --input netwlens_data/London-3X

# With manual root device specification
netwlens visualize --input netwlens_data/London-3X --root s3xasw02

# Different formats and layouts
netwlens visualize --input netwlens_data/London-3X --output_format png --layout tree

Standalone Scripts

# Site-specific collection (with validation)
python collect_data.py --site London-3X --protocol cdp

# All inventory collection
python collect_data.py --protocol cdp

# Collection using SITE_NAME from environment
python collect_data.py --protocol cdp --output env-data

# Visualization with auto root selection
python visualize_topology.py --input netwlens_data/London-3X

# Visualization with manual root selection
python visualize_topology.py --input netwlens_data/London-3X --root s3xasw02

🛠️ Features

Data Collection with Validation

• IP Fabric Integration: Direct API connection to extract inventory and connections
• Connection Validation: Automatic IP Fabric connectivity check before data collection
• Site Validation: Verifies site names exist in IP Fabric with helpful error messages
• Parameter Precedence: CLI parameters override environment variables (SITE_NAME)
• Clean Error Handling: Minimal, actionable error messages without verbose tracebacks
• Protocol Support: CDP, LLDP, and CEF protocols
• Flexible Site Filtering: Extract data for specific sites or all inventory
• Data Processing: Automatic enrichment and normalization of connection data
• JSON Export: Clean, structured data files for analysis
• Comprehensive Data Validation: Multi-layer validation ensures data quality (NEW!)
• Backward Compatibility: Preserves existing API while adding new capabilities

Visualization

• Automatic Root Selection: Intelligently selects device with most connections as root
• Data Validation: Automatic validation before visualization with detailed error reporting (NEW!)
• Quality Assurance: Connectivity analysis, device relationship validation, protocol consistency checks (NEW!)
• Clean Diagrams: Orthogonal layouts with 90-degree connections only
• Multiple Formats: SVG, PNG, and PDF output formats
• Interactive Viewers: HTML viewers for easy sharing and presentation
• Hierarchical Layouts: Root-based topology with proper device positioning
• Color Coding: Device types distinguished by colors and shapes
• Interface Labels: Clear labeling of connection interfaces
• Smart Logging: Detailed connection analysis and validation summary

Architecture

• Object-Oriented Design: Clean DataCollector and DataVisualizer classes
• Comprehensive Validation: Dedicated DataValidator class with multi-layer validation (NEW!)
• Modular Structure: Separate concerns for data collection, validation, and visualization
• CLI Integration: Modern subcommand-based CLI interface with built-in validation
• Package Installation: Proper Python package with pip install
• Extensible: Easy to add new protocols, formats, layouts, and validation rules

📖 API Reference

DataCollector Class

class DataCollector:
    def collect_site_data(site: str, protocol: str = "cef", output: str = None) -> str
    def extract_inventory(site: str) -> dict
    def extract_connections(site: str, protocol: str) -> dict
    def get_client_info() -> dict

TopologyVisualizer Class

class TopologyVisualizer:
    def create_topology(input_dir: str, root_device: str = None, format_type: str = "svg") -> str
    def create_hierarchical_topology(input_dir: str, root_device: str) -> str
    def create_tree_topology(input_dir: str) -> str
    def get_available_formats() -> list
    def get_available_layouts() -> list

🔧 Requirements

• Python: 3.11+
• Dependencies:
  • ipfabric>=7.0.0 - IP Fabric API client
  • loguru>=0.6.0 - Logging
  • graphviz>=0.20.0 - Graph visualization
  • python-dotenv>=1.0.0 - Environment variables
• System: Graphviz binaries installed
• Access: IP Fabric instance with API token
• Package Management: Poetry (recommended) or pip

🎨 Examples

Complete Workflow

# 1. Collect data for London-3X site using CDP
netwlens collect --site London-3X --protocol cdp --output london-network

# 2. Create hierarchical visualization with s3xasw02 as root
netwlens visualize --input netwlens_data/london-network --root s3xasw02 --output_format svg

# Output files:
# - netwlens_data/london-network/inventory.json
# - netwlens_data/london-network/connections_*.json
# - netwlens_data/london-network/topology/topology_hierarchical.svg
# - netwlens_data/london-network/topology/view_svg_topology.html

Python Integration

from netwlens import DataCollector, DataVisualizer
import os

# Collect data for multiple sites
collector = DataCollector()
sites = ["London-3X", "Paris-2X", "Berlin-1X"]

for site in sites:
    print(f"Collecting data for {site}...")
    data_dir = collector.collect_site_data(
        site=site,
        protocol="cdp",
        output=site.lower()
    )

    # Create visualization
    visualizer = DataVisualizer()
    viz_path = visualizer.create_topology(
        input_dir=data_dir,
        format_type="svg"
    )
    print(f"Visualization saved: {viz_path}")

🤝 Contributing

Contributions are welcome! Please feel free to submit issues and pull requests.

👨‍💻 Author

Milan Zapletal

📄 License

MIT License