nfstream: a flexible network data analysis framework

nfstream is a Python package providing fast, flexible, and expressive data structures designed to make working with online or offline network data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world network data analysis in Python. Additionally, it has the broader goal of becoming a common network data processing framework for researchers providing data reproducibility across experiments.

Live Demo Notebook
Project Website
Latest Release
Citation
Downloads
Supported Platforms
Supported Versions
Build Status
Code Quality
Code Coverage
Discussion Channel

Main Features

Performance: nfstream is designed to be fast (x10 faster with pypy3 support) with a small CPU and memory footprint.
Layer-7 visibility: nfstream deep packet inspection engine is based on nDPI. It allows nfstream to perform reliable encrypted applications identification and metadata extraction (e.g. TLS, QUIC, TOR, HTTP, SSH, DNS, etc.).
Flexibility: add a flow feature in 2 lines as an NFPlugin.
Machine Learning oriented: add your trained model as an NFPlugin.

How to use it?

Dealing with a big pcap file and just want to aggregate it as network flows? nfstream make this path easier in few lines:

from nfstream import NFStreamer
my_awesome_streamer = NFStreamer(source="facebook.pcap") # or network interface (source="eth0")
for flow in my_awesome_streamer:
    print(flow)  # print it, append to pandas Dataframe or whatever you want :)!

NFEntry(
    id=0,
    bidirectional_first_seen_ms=1472393122365.661,
    bidirectional_last_seen_ms=1472393123665.163,
    src2dst_first_seen_ms=1472393122365.661,
    src2dst_last_seen_ms=1472393123408.152,
    dst2src_first_seen_ms=1472393122668.038,
    dst2src_last_seen_ms=1472393123665.163,
    version=4,
    src_port=52066,
    dst_port=443,
    protocol=6,
    vlan_id=4,
    src_ip='192.168.43.18',
    dst_ip='66.220.156.68',
    bidirectional_packets=19,
    bidirectional_raw_bytes=5745,
    bidirectional_ip_bytes=5479,
    bidirectional_duration_ms=1299.502197265625,
    src2dst_packets=9, src2dst_raw_bytes=1345,
    src2dst_ip_bytes=1219,
    src2dst_duration_ms=1299.502197265625,
    dst2src_packets=10,
    dst2src_raw_bytes=4400,
    dst2src_ip_bytes=4260,
    dst2src_duration_ms=997.125,
    expiration_id=0,
    master_protocol=91,
    app_protocol=119,
    application_name='TLS.Facebook',
    category_name='SocialNetwork',
    client_info='facebook.com',
    server_info='*.facebook.com,*.facebook.net,*.fb.com,*.fbcdn.net,\
                *.fbsbx.com,*.m.facebook.com,*.messenger.com,*.xx.fbcdn.net,\
                *.xy.fbcdn.net,*.xz.fbcdn.net,facebook.com,fb.com,messenger.com',
    j3a_client='bfcc1a3891601edb4f137ab7ab25b840',
    j3a_server='2d1eb5817ece335c24904f516ad5da12'
)

nfstream also extracts 60+ flow statistical features

from nfstream import NFStreamer
my_awesome_streamer = NFStreamer(source="facebook.pcap", statistics=True)
for flow in my_awesome_streamer:
    print(flow)

NFEntry(
    id=0,
    bidirectional_first_seen_ms=1472393122365.661,
    bidirectional_last_seen_ms=1472393123665.163,
    src2dst_first_seen_ms=1472393122365.661,
    src2dst_last_seen_ms=1472393123408.152,
    dst2src_first_seen_ms=1472393122668.038,
    dst2src_last_seen_ms=1472393123665.163,
    version=4,
    src_port=52066,
    dst_port=443,
    protocol=6,
    vlan_id=4,
    src_ip='192.168.43.18',
    dst_ip='66.220.156.68',
    bidirectional_packets=19,
    bidirectional_raw_bytes=5745,
    bidirectional_ip_bytes=5479,
    bidirectional_duration_ms=1299.502197265625,
    src2dst_packets=9,
    src2dst_raw_bytes=1345,
    src2dst_ip_bytes=1219,
    src2dst_duration_ms=1299.502197265625,
    dst2src_packets=10,
    dst2src_raw_bytes=4400,
    dst2src_ip_bytes=4260,
    dst2src_duration_ms=997.125,
    expiration_id=0,
    bidirectional_min_raw_ps=66,
    bidirectional_mean_raw_ps=302.36842105263156,
    bidirectional_stdev_raw_ps=425.53315715259754,
    bidirectional_max_raw_ps=1454,
    src2dst_min_raw_ps=66,
    src2dst_mean_raw_ps=149.44444444444446,
    src2dst_stdev_raw_ps=132.20354676701294,
    src2dst_max_raw_ps=449,
    dst2src_min_raw_ps=66,
    dst2src_mean_raw_ps=440.0,
    dst2src_stdev_raw_ps=549.7164925870628,
    dst2src_max_raw_ps=1454,
    bidirectional_min_ip_ps=52,
    bidirectional_mean_ip_ps=288.36842105263156,
    bidirectional_stdev_ip_ps=425.53315715259754,
    bidirectional_max_ip_ps=1440,
    src2dst_min_ip_ps=52,
    src2dst_mean_ip_ps=135.44444444444446,
    src2dst_stdev_ip_ps=132.20354676701294,
    src2dst_max_ip_ps=435,
    dst2src_min_ip_ps=52,
    dst2src_mean_ip_ps=426.0,
    dst2src_stdev_ip_ps=549.7164925870628,
    dst2src_max_ip_ps=1440,
    bidirectional_min_piat_ms=0.0029296875,
    bidirectional_mean_piat_ms=72.19456651475694,
    bidirectional_stdev_piat_ms=137.32250609970072,
    bidirectional_max_piat_ms=397.63720703125,
    src2dst_min_piat_ms=0.008056640625,
    src2dst_mean_piat_ms=130.3114013671875,
    src2dst_stdev_piat_ms=179.64644832489174,
    src2dst_max_piat_ms=414.4921875,
    dst2src_min_piat_ms=0.006103515625,
    dst2src_mean_piat_ms=110.79166666666669,
    dst2src_stdev_piat_ms=169.61844149451002,
    dst2src_max_piat_ms=0.531005859375,
    bidirectional_syn_packets=2,
    bidirectional_cwr_packets=0,
    bidirectional_ece_packets=0,
    bidirectional_urg_packets=0,
    bidirectional_ack_packets=18,
    bidirectional_psh_packets=9,
    bidirectional_rst_packets=0,
    bidirectional_fin_packets=0,
    src2dst_syn_packets=1,
    src2dst_cwr_packets=0,
    src2dst_ece_packets=0,
    src2dst_urg_packets=0,
    src2dst_ack_packets=8,
    src2dst_psh_packets=4,
    src2dst_rst_packets=0,
    src2dst_fin_packets=0,
    dst2src_syn_packets=1,
    dst2src_cwr_packets=0,
    dst2src_ece_packets=0,
    dst2src_urg_packets=0,
    dst2src_ack_packets=10,
    dst2src_psh_packets=5,
    dst2src_rst_packets=0,
    dst2src_fin_packets=0, 
    master_protocol=91,
    app_protocol=119,
    application_name='TLS.Facebook',
    category_name='SocialNetwork',
    client_info='facebook.com',
    server_info='*.facebook.com,*.facebook.net,*.fb.com,*.fbcdn.net,\
                *.fbsbx.com,*.m.facebook.com,*.messenger.com,*.xx.fbcdn.net,\
                *.xy.fbcdn.net,*.xz.fbcdn.net,facebook.com,fb.com,messenger.com',
    j3a_client='bfcc1a3891601edb4f137ab7ab25b840',
    j3a_server='2d1eb5817ece335c24904f516ad5da12'
)

From pcap to Pandas DataFrame?

my_dataframe = NFStreamer(source='devil.pcap').to_pandas()
my_dataframe.head(5)

Didn't find a specific flow feature? add a plugin to nfstream in few lines:

from nfstream import NFPlugin

class packet_with_666_size(NFPlugin):
    def on_init(self, pkt): # flow creation with the first packet
        if pkt.raw_size == 666:
            return 1
        else:
            return 0

    def on_update(self, pkt, flow): # flow update with each packet belonging to the flow
        if pkt.pkt.raw_size == 666:
            flow.packet_with_666_size += 1

streamer_awesome = NFStreamer(source='devil.pcap', plugins=[packet_with_666_size()])
for flow in streamer_awesome:
    print(flow.packet_with_666_size) # see your dynamically created metric in generated flows

More example and details are provided on the official documentation.
You can test nfstream without installation using our live demo notebook.

Installation

Using pip

Binary installers for the latest released version are available:

python3 -m pip install nfstream

Build from sources

If you want to build nfstream from sources on your local machine:

On Linux

sudo apt-get install autoconf automake libtool pkg-config libpcap-dev
git clone https://github.com/aouinizied/nfstream.git
cd nfstream
python3 -m pip install -r requirements.txt
python3 setup.py bdist_wheel

On MacOS

brew install autoconf automake libtool pkg-config
git clone https://github.com/aouinizied/nfstream.git
cd nfstream
python3 -m pip install -r requirements.txt
python3 setup.py bdist_wheel

Contributing

Please read Contributing for details on our code of conduct, and the process for submitting pull requests to us.

Authors

Zied Aouini created nfstream and these fine people have contributed.

Ethics

nfstream is intended for network data research and forensics. Researchers and network data scientists can use these framework to build reliable datasets, train and evaluate network applied machine learning models. As with any packet monitoring tool, nfstream could potentially be misused. Do not run it on any network of which you are not the owner or the administrator.

License

This project is licensed under the GPLv3 License - see the License file for details

Release files for nfstream 4.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distributions (wheels)

Table of built distributions (wheels) for nfstream 4.0.0
File	Interpreter	ABI	Platform	Reset
nfstream-4.0.0-pp36-pypy36_pp73-manylinux1_x86_64.whl 1.1 MB Apr 15, 2020	PyPy 3.6	PyPy 3.6 7.3	Linux glibc 2.5+ x86-64	Details
nfstream-4.0.0-cp38-cp38-manylinux1_x86_64.whl 1.1 MB Apr 15, 2020	CPython 3.8	CPython 3.8	Linux glibc 2.5+ x86-64	Details
nfstream-4.0.0-cp37-cp37m-manylinux1_x86_64.whl 1.1 MB Apr 15, 2020	CPython 3.7	CPython 3.7 pymalloc	Linux glibc 2.5+ x86-64	Details
nfstream-4.0.0-cp37-cp37m-macosx_10_15_x86_64.whl 451.4 kB Apr 15, 2020	CPython 3.7	CPython 3.7 pymalloc	macOS 10.15+ x86-64	Details
nfstream-4.0.0-cp36-cp36m-manylinux1_x86_64.whl 1.1 MB Apr 15, 2020	CPython 3.6	CPython 3.6 pymalloc	Linux glibc 2.5+ x86-64	Details

Total release size: 4.9 MB

Release files / nfstream-4.0.0-pp36-pypy36_pp73-manylinux1_x86_64.whl

Download URL	nfstream-4.0.0-pp36-pypy36_pp73-manylinux1_x86_64.whl
Size	1.1 MB
Tags	Linux glibc 2.5+ x86-64 PyPy 3.6 PyPy 3.6 7.3
SHA-256 checksum How to use checksums	`e2f39ccb119e835ea1bb71fe81f5c504dcde973aeb0ec83359a31305101b714c`
BLAKE2b-256 checksum How to use checksums	`77aee44d06facb86c3a4999cdbe3b98a6e1cd917cefda483aad8d261ceb84f67`
Upload date	Apr 15, 2020
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 PyPy/7.3.0`

Release files / nfstream-4.0.0-cp38-cp38-manylinux1_x86_64.whl

Download URL	nfstream-4.0.0-cp38-cp38-manylinux1_x86_64.whl
Size	1.1 MB
Tags	CPython 3.8 Linux glibc 2.5+ x86-64
SHA-256 checksum How to use checksums	`5052af96a7dd9bb75fe33a5d30c5823781322554bc16d2ad747a6828f76147ce`
BLAKE2b-256 checksum How to use checksums	`20a26176947a7418158a1017deede630bdc5682167fcb7944a671eaea7ee01dc`
Upload date	Apr 15, 2020
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2`

Release files / nfstream-4.0.0-cp37-cp37m-manylinux1_x86_64.whl

Download URL	nfstream-4.0.0-cp37-cp37m-manylinux1_x86_64.whl
Size	1.1 MB
Tags	CPython 3.7 CPython 3.7 pymalloc Linux glibc 2.5+ x86-64
SHA-256 checksum How to use checksums	`938e152b414fd419443dbe84a84f9a797c8aa49ed621907858b808d82510c93f`
BLAKE2b-256 checksum How to use checksums	`17b74c0df4e7ef15b681d565e5bce5a9caeb80a27982783130ed47001ec58f7a`
Upload date	Apr 15, 2020
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.7.6`

Release files / nfstream-4.0.0-cp37-cp37m-macosx_10_15_x86_64.whl

Download URL	nfstream-4.0.0-cp37-cp37m-macosx_10_15_x86_64.whl
Size	451.4 kB
Tags	CPython 3.7 CPython 3.7 pymalloc macOS 10.15+ x86-64
SHA-256 checksum How to use checksums	`3b2ce0e34ed873bc36626164bcb176f8598fbfbf39c82e50cf4658795863c8c6`
BLAKE2b-256 checksum How to use checksums	`d64318562dfb5e3eb4eac718479e0aebca2e68de394a584647b13f4e9fe31a34`
Upload date	Apr 15, 2020
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.7.6`

Release files / nfstream-4.0.0-cp36-cp36m-manylinux1_x86_64.whl

Download URL	nfstream-4.0.0-cp36-cp36m-manylinux1_x86_64.whl
Size	1.1 MB
Tags	CPython 3.6 CPython 3.6 pymalloc Linux glibc 2.5+ x86-64
SHA-256 checksum How to use checksums	`13c05680f2a3486b17126c1f174dab97b0a5ffc5bd39bee7cf62593b3e59c7be`
BLAKE2b-256 checksum How to use checksums	`906e249d473af19fde9d7ce3f43ad59ff141fc271f7a841079c33d17207b86ca`
Upload date	Apr 15, 2020
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/40.6.2 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.6.10`