MQTTNetLyzer

Message Queuing Telemetry Transport (MQTT) Network Traffic Analyzer

MQTTNetLyzer is an open-source Python library developed to extract features from MQTT-based network traffic. It analyzes all MQTT packets of a pcap file and generates a dataset, which is essential for developing an Intrusion Detection System (IDS).

MQTTNetLyzer is a session-based analyzer. In essence, it extracts features at the packet level and aggregates them to generate a session. There are several criteria for the aggregation. All packets in a session must have the same source/destination IP address, source/destination port number, client ID, and pub-sub topic. A session is initiated by a CONNECT packet and ended by either a DISCONNECT packet or implemented constraints like maximum session duration, inactivity timeout, etc. MQTTNetLyzer is a bidirectional session generator where the forward direction is determined by the CONNECT packet. MQTTNetLyzer extracts relevant features from an MQTT packet's Network and Transport layers as well.

Additionally, MQTTNetLyzer calculates new features other than raw features and adds them to the session. It also calculates statistical features for some of the features, like mean, median, variance, skewness, etc., in both directions. It is capable of extracting a total of 321 features from a pcap file.

Installation

First, clone this repository on your system by running the following command:

git clone git@github.com:adityaraj218/MQTTNetLyzer.git

To install the MQTTNetLyzer package, it is necessary to have the required libraries installed on your system.

Navigate to the package directory and run the following command to ensure it:

Make sure Python and pip are installed on your system

pip3 install -r requirements.txt

Depending on your OS, execute the following command to install the MQTTNetLyzer package.

On Windows

pip3 install .

On Linux

 python3 setup.py install

To ensure a successful installation, run mqttnetlyzer --version.

Execution

To directly run the analyzer without any configuration file, run the following command:

mqttnetlyzer -i path/to/pcap_file.pcap

You can run mqttnetlyzer -h for more supported commands

Configuration file

Since different users may want to run the program differently, users can alternatively use the configuration file to provide the PCAP file address, customise the analyzer parameters, default behaviours, etc.

To use the configuration file, modify the config.json file provided in the repository:

config.json

{
  "input_pcap_file": "path/to/pcap_file.pcap",
  "output_csv_file": "path/to/output.csv",
  "label": "normal",
  "batch_size": 10000,
  "display_batch_status": true,
  "display_batch_stats": false,
  "restricted_topics": ["admin", "$SYS"]
}

Parameters

• input_pcap_file: Address of your PCAP file

• output_csv_file: Desired address of the output file

• label: Type of packets

• batch_size: Number of packets to process at once

• display_batch_status: Displays the number of the batch being processed currently

• display_batch_stats: Displays stats like the total number of packets, number of MQTT packets, and Analyzing rate for each batch

• restricted_topics: List of topics to which clients were not supposed to publish or subscribe

Analyzer will count the number of times a client tried to access restricted topics

To run the analyzer using the config.json file, run the following command:

mqttnetlyzer -c path/to/config.json

Features

The MQTTNetLyzer can extract up to 321 features per session. Below is the visual representation of those features in the form of a sunburst chart:

Sunburst Chart

Taxonomy of features

The list of features the analyzer can extract are:

fwd: Forward direction bwd: Backward direction mean: mean skew: Skewness std: Standard Deviation var: Variance total: Sum

• MQTT

  • MQTTMsgLength
    • fwdMQTTMsgLength
      • meanFwdMQTTMsgLength
      • medianFwdMQTTMsgLength
      • varFwdMQTTMsgLength
      • stdFwdMQTTMsgLength
      • skewFwdMQTTMsgLength
      • minFwdMQTTMsgLength
      • maxFwdMQTTMsgLength
      • totalFwdMQTTMsgLength
    • bwdMQTTMsgLength
      • meanBwdMQTTMsgLength
      • medianBwdMQTTMsgLength
      • varBwdMQTTMsgLength
      • stdBwdMQTTMsgLength
      • skewBwdMQTTMsgLength
      • minBwdMQTTMsgLength
      • maxBwdMQTTMsgLength
      • totalBwdMQTTMsgLength
    • MQTTMsgLength
      • meanMQTTMsgLength
      • medianMQTTMsgLength
      • varMQTTMsgLength
      • stdMQTTMsgLength
      • skewMQTTMsgLength
      • minMQTTMsgLength
      • maxMQTTMsgLength
      • totalMQTTMsgLength
  • PublishMsgLen
    • meanPublishMsgLen
    • medianPublishMsgLen
    • varPublishMsgLen
    • stdPublishMsgLen
    • skewPublishMsgLen
    • minPublishMsgLen
    • maxPublishMsgLen
    • totalPublishMsgLen
  • InterPublishTime
    • meanInterPublishTime
    • medianInterPublishTime
    • varInterPublishTime
    • stdInterPublishTime
    • skewInterPublishTime
    • minInterPublishTime
    • maxInterPublishTime
    • totalInterPublishTime
  • InterSubscribeTime
    • meanInterSubscribeTime
    • medianInterSubscribeTime
    • varInterSubscribeTime
    • stdInterSubscribeTime
    • skewInterSubscribeTime
    • minInterSubscribeTime
    • maxInterSubscribeTime
    • totalInterSubscribeTime
  • SubackDelay
    • meanSubackDelay
    • medianSubackDelay
    • varSubackDelay
    • stdSubackDelay
    • skewSubackDelay
    • minSubackDelay
    • maxSubackDelay
    • totalSubackDelay
  • SubscribeDuration
    • meanSubscribeDuration
    • medianSubscribeDuration
    • varSubscribeDuration
    • stdSubscribeDuration
    • skewSubscribeDuration
    • minSubscribeDuration
    • maxSubscribeDuration
    • totalSubscribeDuration
  • QoS
  • DUPFlag
    • fwdDUPFlag
    • bwdDUPFlag
    • DUPFlag
  • topic
  • clientID
  • username
  • isComplete
  • keepAlive
  • cleanSession
  • willFlag
  • willQoS
  • willRetain
  • willmsgLen
  • willtopicLen
  • passwordFlag
  • reserved
  • protocolName
  • protocolNameLength
  • connackSessionPresent
  • connackReturnCode
  • topicLen
  • publishCount
  • pubackNotRecieved
  • subscribeCount
  • subackNotRecieved
  • subscriptionDenied
  • accessRestrictedTopics
  • pingreqCount
  • pingrespCount

• IP

  • srcIP
  • dstIP
  • srcPort
  • dstPort
  • DFFlag
    • fwdDFFlag
    • bwdDFFlag
    • DFFlag
  • MFFlag
    • fwdMFFlag
    • bwdMFFlag
    • MFFlag
  • RBFlag
    • fwdRBFlag
    • bwdRBFlag
    • RBFlag
  • TTL
    • fwdTTL
      • meanFwdTTL
      • medianFwdTTL
      • varFwdTTL
      • stdFwdTTL
      • skewFwdTTL
      • minFwdTTL
      • maxFwdTTL
      • totalFwdTTL
    • bwdTTL
      • meanBwdTTL
      • medianBwdTTL
      • varBwdTTL
      • stdBwdTTL
      • skewBwdTTL
      • minBwdTTL
      • maxBwdTTL
      • totalBwdTTL
    • TTL
      • meanTTL
      • medianTTL
      • varTTL
      • stdTTL
      • skewTTL
      • minTTL
      • maxTTL
      • totalTTL
  • IPLength
    • fwdIPLength
      • meanFwdIPLength
      • medianFwdIPLength
      • varFwdIPLength
      • stdFwdIPLength
      • skewFwdIPLength
      • minFwdIPLength
      • maxFwdIPLength
      • totalFwdIPLength
    • bwdIPLength
      • meanBwdIPLength
      • medianBwdIPLength
      • varBwdIPLength
      • stdBwdIPLength
      • skewBwdIPLength
      • minBwdIPLength
      • maxBwdIPLength
      • totalBwdIPLength
    • IPLength
      • meanIPLength
      • medianIPLength
      • varIPLength
      • stdIPLength
      • skewIPLength
      • minIPLength
      • maxIPLength
      • totalIPLength

• TCP

  • WindowSize
    • fwdWindowSize
      • meanFwdWindowSize
      • medianFwdWindowSize
      • varFwdWindowSize
      • stdFwdWindowSize
      • skewFwdWindowSize
      • minFwdWindowSize
      • maxFwdWindowSize
      • totalFwdWindowSize
    • bwdWindowSize
      • meanBwdWindowSize
      • medianBwdWindowSize
      • varBwdWindowSize
      • stdBwdWindowSize
      • skewBwdWindowSize
      • minBwdWindowSize
      • maxBwdWindowSize
      • totalBwdWindowSize
    • WindowSize
      • meanWindowSize
      • medianWindowSize
      • varWindowSize
      • stdWindowSize
      • skewWindowSize
      • minWindowSize
      • maxWindowSize
      • totalWindowSize
  • TCPSegmentLen
    • fwdTCPSegmentLen
      • meanFwdTCPSegmentLen
      • medianFwdTCPSegmentLen
      • varFwdTCPSegmentLen
      • stdFwdTCPSegmentLen
      • skewFwdTCPSegmentLen
      • minFwdTCPSegmentLen
      • maxFwdTCPSegmentLen
      • totalFwdTCPSegmentLen
    • bwdTCPSegmentLen
      • meanBwdTCPSegmentLen
      • medianBwdTCPSegmentLen
      • varBwdTCPSegmentLen
      • stdBwdTCPSegmentLen
      • skewBwdTCPSegmentLen
      • minBwdTCPSegmentLen
      • maxBwdTCPSegmentLen
      • totalBwdTCPSegmentLen
    • TCPSegmentLen
      • meanTCPSegmentLen
      • medianTCPSegmentLen
      • varTCPSegmentLen
      • stdTCPSegmentLen
      • skewTCPSegmentLen
      • minTCPSegmentLen
      • maxTCPSegmentLen
      • totalTCPSegmentLen
  • TCPLength
    • fwdTCPLength
      • meanFwdTCPLength
      • medianFwdTCPLength
      • varFwdTCPLength
      • stdFwdTCPLength
      • skewFwdTCPLength
      • minFwdTCPLength
      • maxFwdTCPLength
      • totalFwdTCPLength
    • bwdTCPLength
      • meanBwdTCPLength
      • medianBwdTCPLength
      • varBwdTCPLength
      • stdBwdTCPLength
      • skewBwdTCPLength
      • minBwdTCPLength
      • maxBwdTCPLength
      • totalBwdTCPLength
    • TCPLength
      • meanTCPLength
      • medianTCPLength
      • varTCPLength
      • stdTCPLength
      • skewTCPLength
      • minTCPLength
      • maxTCPLength
      • totalTCPLength
  • TCPFIN
    • fwdTCPFIN
    • bwdTCPFIN
    • TCPFIN
  • TCPSYN
    • fwdTCPSYN
    • bwdTCPSYN
    • TCPSYN
  • TCPRST
    • fwdTCPRST
    • bwdTCPRST
    • TCPRST
  • TCPPSH
    • fwdTCPPSH
    • bwdTCPPSH
    • TCPPSH
  • TCPACK
    • fwdTCPACK
    • bwdTCPACK
    • TCPACK
  • TCPURG
    • fwdTCPURG
    • bwdTCPURG
    • TCPURG
  • TCPECE
    • fwdTCPECE
    • bwdTCPECE
    • TCPECE
  • TCPCWR
    • fwdTCPCWR
    • bwdTCPCWR
    • TCPCWR
  • TCPReserved
    • fwdTCPReserved
    • bwdTCPReserved
    • TCPReserved

• Other

  • PacketLength
    • fwdPacketLength
      • meanFwdPacketLength
      • medianFwdPacketLength
      • varFwdPacketLength
      • stdFwdPacketLength
      • skewFwdPacketLength
      • minFwdPacketLength
      • maxFwdPacketLength
      • totalFwdPacketLength
    • bwdPacketLength
      • meanBwdPacketLength
      • medianBwdPacketLength
      • varBwdPacketLength
      • stdBwdPacketLength
      • skewBwdPacketLength
      • minBwdPacketLength
      • maxBwdPacketLength
      • totalBwdPacketLength
    • PacketLength
      • meanPacketLength
      • medianPacketLength
      • varPacketLength
      • stdPacketLength
      • skewPacketLength
      • minPacketLength
      • maxPacketLength
      • totalPacketLength
  • IAT
    • fwdIAT
      • meanFwdIAT
      • medianFwdIAT
      • varFwdIAT
      • stdFwdIAT
      • skewFwdIAT
      • minFwdIAT
      • maxFwdIAT
      • totalFwdIAT
    • bwdIAT
      • meanBwdIAT
      • medianBwdIAT
      • varBwdIAT
      • stdBwdIAT
      • skewBwdIAT
      • minBwdIAT
      • maxBwdIAT
      • totalBwdIAT
    • IAT
      • meanIAT
      • medianIAT
      • varIAT
      • stdIAT
      • skewIAT
      • minIAT
      • maxIAT
      • totalIAT
  • PacketsCount
    • fwdPacketsCount
    • bwdPacketsCount
    • PacketsCount
  • Throughput
    • fwdThroughput
    • bwdThroughput
    • Throughput
  • ID
  • startTime
  • endTime
  • duration
  • label

Sample Output

A PCAP file with two sessions was analysed with MQTTNetLyzer

Citation&Copywrite2024

ProjectTeamMembers

• Aditya Raj: