nexusflowmeter 0.1.3

Convert PCAP files to flow-based analysis (CSV, JSON, Excel)

NexusFlowMeter

NexusFlowMeter is a high-performance network flow analysis tool that converts packet capture (PCAP) files into comprehensive flow-based insights. Unlike traditional packet-by-packet analysis, NexusFlowMeter aggregates network traffic into flows, providing deeper understanding of network behavior, communication patterns, and traffic characteristics.

  Contributions are welcome! Please feel free to submit pull requests, report bugs, or suggest features.

Architecture

graph LR
    %% Input Layer
    subgraph "Input Layer"
        A[Single PCAP File] 
        B[Directory of PCAPs]
        C[Large PCAPs >1GB]
    end

    %% Input Processing
    subgraph "Input Processing"
        D[File Size Check]
        E[PCAP Discovery]
        F[File Splitting]
        G[Chunk Generation]
    end

    %% Core Engine
    subgraph "Core Engine"
        H[PCAPToFlowConverter]
        I[Streaming Reader: PcapReader]
        J[Non-Streaming Reader: rdpcap]
        K[Flow Key Extractor]
        L[Packet Parser]
    end

    %% Flow Management
    subgraph "Flow Management"
        M[FlowKey Generator]
        N[NetworkFlow Objects]
        O[Flow Statistics]
        P[TCP State Analysis]
        Q[Timing Analysis]
    end

    %% Parallel Processing
    subgraph "Parallel Processing"
        R[ProcessPoolExecutor]
        S[Worker 1]
        T[Worker 2]
        U[Worker N]
        V[Results Merger]
    end

    %% Protocol Analysis
    subgraph "Protocol Analysis"
        W[TCP Handler]
        X[UDP Handler]
        Y[ICMP Handler]
        Z[ARP Handler]
        AA[DNS Handler]
    end

    %% Data Aggregation
    subgraph "Data Aggregation"
        BB[Flow Merger]
        CC[Statistics Calculator]
        DD[DataFrame Converter]
        EE[Protocol Filter]
        FF[Flow Limiter]
    end

    %% Output Processing
    subgraph "Output Processing"
        GG[CSV Exporter]
        HH[JSON Exporter]
        II[Excel Exporter]
        JJ[Report Generator]
        KK[Summary Statistics]
    end

    %% Batch Modes
    subgraph "Batch Modes"
        LL[Separate Files Mode]
        MM[Merged Mode]
        NN[Source File Tracking]
    end

    %% Output Layer
    subgraph "Output Layer"
        OO[Individual CSV Files]
        PP[Merged CSV File]
        QQ[JSON Output]
        RR[Excel Files]
        SS[Analysis Reports]
        TT[Processing Logs]
    end

    %% Flow connections
    A --> D
    B --> E
    C --> F
    D --> H
    E --> H
    F --> G
    G --> R
    H --> I --> L
    H --> J --> L
    L --> K --> M --> N
    N --> O & P & Q
    G --> S & T & U
    S --> V
    T --> V
    U --> V
    L --> W & X & Y & Z & AA
    V --> BB
    O & P & Q --> CC
    CC --> DD --> EE --> FF
    FF --> GG & HH & II
    DD --> JJ
    CC --> KK
    B --> LL & MM
    MM --> NN
    LL --> OO
    MM --> PP
    GG --> OO & PP
    HH --> QQ
    II --> RR
    JJ --> SS
    KK --> TT

    %% Styling
    classDef input fill:#d9edf7,stroke:#31708f,stroke-width:1px;
    classDef process fill:#e6e6fa,stroke:#6a1b9a,stroke-width:1px;
    classDef core fill:#fff2cc,stroke:#b36b00,stroke-width:1px;
    classDef flow fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px;
    classDef parallel fill:#fde0dc,stroke:#b71c1c,stroke-width:1px;
    classDef proto fill:#ede7f6,stroke:#4527a0,stroke-width:1px;
    classDef data fill:#f1f8e9,stroke:#33691e,stroke-width:1px;
    classDef outputProc fill:#f0f4c3,stroke:#827717,stroke-width:1px;
    classDef batch fill:#ffe0b2,stroke:#e65100,stroke-width:1px;
    classDef finalOut fill:#c8e6c9,stroke:#1b5e20,stroke-width:2px;

    class A,B,C input
    class D,E,F,G process
    class H,I,J,K,L core
    class M,N,O,P,Q flow
    class R,S,T,U,V parallel
    class W,X,Y,Z,AA proto
    class BB,CC,DD,EE,FF data
    class GG,HH,II,JJ,KK outputProc
    class LL,MM,NN batch
    class OO,PP,QQ,RR,SS,TT finalOut

Installation

From PyPI (Recommended)

pip install nexusflowmeter

From Source

git clone https://github.com/yourusername/NexusFlowMeter.git
cd NexusFlowMeter
pip install -r requirements.txt

Dependencies

• scapy - Packet manipulation library
• pandas - Data analysis and manipulation
• openpyxl - Excel file support (optional)

NexusFlowMeter CLI Usage

 nexusflowmeter [-h] [-d DIRECTORY] [-c OUTPUT_DIR] [--merge] [--protocols PROTOCOLS] [--max-flows MAX_FLOWS]
                      [--output-format {csv,json,xlsx}] [--quick-preview QUICK_PREVIEW] [--split-by-protocol]
                      [--stream] [--chunk-size CHUNK_SIZE] [--flow-timeout FLOW_TIMEOUT] [--max-workers MAX_WORKERS]
                      [--verbose]
                      [pcap_file] [output_file]

Convert PCAP files to flow-based analysis (CSV, JSON, Excel)

positional arguments:
  pcap_file                               Input PCAP file path (for single file processing)
  output_file                             Output file path (for single file processing)

options:
  -h, --help                              show this help message and exit
  -d DIRECTORY, --directory DIRECTORY     Input directory containing PCAP files (for batch processing)
                                      
  -c OUTPUT_DIR, --output-dir OUTPUT_DIR  Output directory (for batch processing)
                        
  --merge                                 Merge all PCAP files into single output (directory processing only)
  --protocols PROTOCOLS                   Comma-separated list of protocols to include (tcp,udp,icmp,arp,dns,all)
                        
  --max-flows MAX_FLOWS                   Maximum number of flows to analyze
                        
  --output-format {csv,json,xlsx}, -of {csv,json,xlsx}
                                          Output file format (default: csv)

  --quick-preview QUICK_PREVIEW           Show first N flows before conversion (single file only)
                        
  --split-by-protocol                     Create separate files for each protocol (single file only)
  --stream                                Use streaming mode (PcapReader) instead of loading the whole file into memory
  --chunk-size CHUNK_SIZE                 Chunk size in MB for large files (default: 1024MB = 1GB)
                        
  --flow-timeout FLOW_TIMEOUT             Flow timeout in seconds (default: 60)
                        
  --max-workers MAX_WORKERS               Maximum parallel workers for chunk processing (default: 4)
                        
  --verbose                               Enable verbose logging

---

Positional Arguments

Argument	Description
pcap_file	Input PCAP file path (for single file processing)
output_file	Output file path (for single file processing)
-d /folder_path/	Input folder path containing PCAPs (for multiple file processing)
-c /folder_path/	Output folder path for extracted CSVs (for multiple file processing)

Examples

    Single file conversion:
      nexusflowmeter capture.pcap flows.csv

    Directory processing:
      nexusflowmeter -d ./pcaps/ -c ./out/                    # Convert each PCAP in the folder to separate CSV
      nexusflowmeter -d ./pcaps/ -c ./out/ --merge            # Merge all PCAPs into one CSV
      nexusflowmeter -d ./pcaps/ -c ./out/ --output-format json  # Convert to JSON format

Options

Option	Description
-h, --help	Show help message and exit
-d DIRECTORY, --directory DIRECTORY	Input directory containing PCAP files (for batch processing)
-c OUTPUT_DIR, --output-dir OUTPUT_DIR	Output directory (for batch processing)
--merge	Merge all PCAP files into single output (directory processing only)
--protocols PROTOCOLS	Comma-separated list of protocols to include (tcp,udp,icmp,arp,dns,all)
--max-flows MAX_FLOWS	Maximum number of flows to analyze
--output-format {csv,json,xlsx}, -of {csv,json,xlsx}	Output file format (default: csv)
--quick-preview QUICK_PREVIEW	Show first N flows before conversion (single file only)
--split-by-protocol	Create separate files for each protocol (single file only)
--stream	Use streaming mode (PcapReader) instead of loading the whole file into memory (recommended for memory constrained systems)
--chunk-size CHUNK_SIZE	Chunk size in MB for large files (default: 1024MB = 1GB)
--flow-timeout FLOW_TIMEOUT	Flow timeout in seconds (default: 60)
--max-workers MAX_WORKERS	Maximum parallel workers for chunk processing (default: 4)
--verbose	Enable verbose logging

Examples

    nexusflowmeter capture.pcap flows.json --output-format json  OR
    nexusflowmeter capture.pcap flows.json -of json


    nexusflowmeter capture.pcap flows --split-by-protocol
    nexusflowmeter capture.pcap flows.csv --quick-preview 5
    nexusflowmeter capture.pcap tcp_flows.xlsx --output-format xlsx --protocols tcp

Flow Features Extracted

NexusFlowMeter generates 34 per-flow statistics useful for traffic analysis, anomaly detection, and ML feature extraction.

Feature	Description
flow_id	Unique identifier for the flow (src_ip:src_port <-> dst_ip:dst_port protocol)
src_ip, dst_ip	Source and destination IP addresses
src_port, dst_port	Source and destination ports
protocol	Transport or network protocol (TCP, UDP, ICMP, etc.)
start_time, end_time	Timestamps marking the beginning and end of the flow
duration_seconds	Duration of the flow
total_packets	Total number of packets in the flow
forward_packets, backward_packets	Packets in forward (src→dst) and backward (dst→src) directions
is_bidirectional	Indicates if packets exist in both directions
forward_backward_ratio	Ratio of forward to backward packet counts
total_bytes	Total bytes transferred in the flow
forward_bytes, backward_bytes	Bytes in forward and backward directions
total_payload_bytes	Total payload size excluding headers
forward_payload_bytes, backward_payload_bytes	Payload bytes split by direction
packets_per_second	Packet transmission rate
bytes_per_second	Byte transfer rate
min_packet_size, max_packet_size	Smallest and largest packet sizes observed
avg_packet_size	Average packet size
avg_inter_arrival_ms	Average packet inter-arrival time (milliseconds)
min_inter_arrival_ms, max_inter_arrival_ms	Minimum and maximum inter-arrival times
tcp_flags	Summary of TCP flags observed (e.g., SYN, FIN, ACK)
syn_count, fin_count, rst_count, ack_count	Count of specific TCP flags
connection_state	State of the TCP connection (e.g., Established, Closed)

---

What makes this tool unique?

NexusFlowMeter includes several productivity-focused options which makes using this tool very convinient and easy.

Directory & Batch Processing

• Feature: Process an entire folder of PCAP files at once.
• Example:

  convert all pcaps in the folder to individual csvs
    nexusflowmeter -d ./pcaps/ -c ./out/
  
  convert all pcaps in the folder into a single file
    nexusflowmeter -d ./pcaps/ -c ./out/ --merge
  

Protocol Filtering

• Feature: Focus only on specific protocols (TCP, UDP, ICMP, ARP, DNS, or all) i.e, extracted file contains flow with mentioned protocol only.
• Example:

  nexusflowmeter capture.pcap tcp_flows.csv --protocols tcp
  

Quick Preview

• Feature: View the first N flows before committing to full conversion.
• Example:

  nexusflowmeter capture.pcap flows.csv --quick-preview 5
  

  Prints the first 5 flows in the console.

Split by Protocol

• Feature: Automatically create separate output files for each protocol.
• Example:

  nexusflowmeter capture.pcap flows --split-by-protocol
  

  Generates files like flows_tcp.csv, flows_udp.csv, etc.

Streaming Mode

• Feature: Processes packets using a stream reader instead of loading the entire PCAP into memory (1000 pkts/sec). Useful for large captures and memory contrained systems.
• Example:

  nexusflowmeter capture.pcap flows.csv --stream
  

Chunked Processing

• Feature: Split very large PCAPs into chunks (in MB) for parallel or memory-efficient processing (default : 1024 MB).

• Example:

  nexusflowmeter capture.pcap flows.csv --chunk-size 512
  

  Processes the file in 512 MB chunks.

  nexusflowmeter capture.pcap flows.csv --chunk-size 512 --max-workers 5
  

  Processes the file in 512 MB chunks parallely , here 5 seperate processes.

  Output Formats

• Feature: Export results in CSV, JSON, or Excel (XLSX).

• Example:

   nexusflowmeter capture.pcap flows.json --output-format json
   nexusflowmeter capture.pcap flows.xlsx --output-format xlsx