Skip to main content

A flexible network data analysis framework

Project description

nfstream: a flexible network data analysis framework

nfstream is a Python package providing fast, flexible, and expressive data structures designed to make working with online or offline network data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world network data analysis in Python. Additionally, it has the broader goal of becoming a common network data processing framework for researchers providing data reproducibility across experiments.

Latest Release latest release
Downloads downloads
Supported Platforms Linux MacOS
Supported Versions python3 pypy3
Build Status Github WorkFlows
Documentation Status ReadTheDocs
Code Quality Quality
Code Coverage Coverage
Discussion Channel Gitter

Main Features

  • Performance: nfstream is designed to be fast (x10 faster with pypy3 support) with a small CPU and memory footprint.
  • Layer-7 visibility: nfstream deep packet inspection engine is based on nDPI. It allows nfstream to perform reliable encrypted applications identification and metadata extraction (e.g. TLS, QUIC, TOR, HTTP, SSH, DNS).
  • Flexibility: add a flow feature in 2 lines as an NFPlugin.
  • Machine Learning oriented: add your trained model as an NFPlugin.

How to use it?

  • Dealing with a big pcap file and just want to aggregate it as network flows? nfstream make this path easier in few lines:
   from nfstream import NFStreamer
   my_awesome_streamer = NFStreamer(source="facebook.pcap") # or network interface (source="eth0")
   for flow in my_awesome_streamer:
       print(flow)  # print it, append to pandas Dataframe or whatever you want :)!
    NFEntry(
        id=0,
        first_seen=1472393122365,
        last_seen=1472393123665,
        version=4,
        src_port=52066,
        dst_port=443,
        protocol=6,
        vlan_id=0,
        src_ip='192.168.43.18',
        dst_ip='66.220.156.68',
        total_packets=19,
        total_bytes=5745,
        duration=1300,
        src2dst_packets=9,
        src2dst_bytes=1345,
        dst2src_packets=10,
        dst2src_bytes=4400,
        expiration_id=0,
        master_protocol=91,
        app_protocol=119,
        application_name='TLS.Facebook',
        category_name='SocialNetwork',
        client_info='facebook.com',
        server_info='*.facebook.com,*.facebook.net,*.fb.com,*.fbcdn.net,*.fbsbx.com,*.m.facebook.com,*.messenger.com,*.xx.fbcdn.net,*.xy.fbcdn.net,*.xz.fbcdn.net,facebook.com,fb.com,messenger.com',
        j3a_client='bfcc1a3891601edb4f137ab7ab25b840',
        j3a_server='2d1eb5817ece335c24904f516ad5da12'
    )
  • From pcap to Pandas DataFrame?
    import pandas as pd	
    streamer_awesome = NFStreamer(source='devil.pcap')
    data = []
    for flow in streamer_awesome:
       data.append(flow.to_namedtuple())
    my_df = pd.DataFrame(data=data)
    my_df.head(5) # Enjoy!
  • Didn't find a specific flow feature? add a plugin to nfstream in few lines:
    from nfstream import NFPlugin

    class my_awesome_plugin(NFPlugin):
        def on_update(self, obs, entry):
            if obs.length >= 666:
                entry.my_awesome_plugin += 1

   streamer_awesome = NFStreamer(source='devil.pcap', plugins=[my_awesome_plugin()])
   for flow in streamer_awesome:
      print(flow.my_awesome_plugin) # see your dynamically created metric in generated flows
  • More example and details are provided on the official documentation.

Prerequisites

    apt-get install libpcap-dev

Installation

Using pip

Binary installers for the latest released version are available:

    pip3 install nfstream

Build from source

If you want to build nfstream on your local machine:

    git clone https://github.com/aouinizied/nfstream.git
    cd nfstream
    python3 setup.py install

Contributing

Please read Contributing for details on our code of conduct, and the process for submitting pull requests to us.

Authors

Zied Aouini created nfstream and these fine people have contributed.

Ethics

nfstream is intended for network data research and forensics. Researchers and network data scientists can use these framework to build reliable datasets, train and evaluate network applied machine learning models. As with any packet monitoring tool, nfstream could potentially be misused. Do not run it on any network of which you are not the owner or the administrator.

License

This project is licensed under the GPLv3 License - see the License file for details

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for nfstream, version 3.2.0
Filename, size File type Python version Upload date Hashes
Filename, size nfstream-3.2.0-cp36-cp36m-manylinux1_x86_64.whl (968.9 kB) File type Wheel Python version cp36 Upload date Hashes View hashes
Filename, size nfstream-3.2.0-cp37-cp37m-macosx_10_15_x86_64.whl (324.2 kB) File type Wheel Python version cp37 Upload date Hashes View hashes
Filename, size nfstream-3.2.0-cp37-cp37m-manylinux1_x86_64.whl (968.9 kB) File type Wheel Python version cp37 Upload date Hashes View hashes
Filename, size nfstream-3.2.0-cp38-cp38-manylinux1_x86_64.whl (968.9 kB) File type Wheel Python version cp38 Upload date Hashes View hashes
Filename, size nfstream-3.2.0-pp36-pypy36_pp73-manylinux1_x86_64.whl (969.0 kB) File type Wheel Python version pp36 Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page