network-pipeline 1.0.23

Distributed Network Packet Analysis Pipeline for Layer 2, 3 and 4 Frames

Python 3 AI-ready framework for recording network traffic in a data pipeline. Once recorded, you can train a deep neural network (DNN) to identify attack and non-attack traffic on your network. Included demo DNN has over 83% accuracy predicting attack vs non-attack records. Currently supports recording ethernet and arp (layer 2), ipv4, ipv6 and icmp (layer 3) and also tcp, udp frames (layer 4) frames and datagrams. Messages are auto-forwarded over to redis or rabbitmq for distributed processing in realtime. Why should I use this? This framework can help build, train and tune your own defensive machine learning models to help defend your own infrastructure at the network layer. Once the data is auto-saved as a csv file, then you can build models within Jupyter notebooks: https://github.com/jay-johnson/celery-connectors#running-jupyterhub-with-postgres-and-ssl or your ML/AI framework of choice. This pip also has an example for training a Keras Deep Neural Network model to predict attack and non-attack records using a captured and prepared dataset. There are test tools installed with this pip to quickly send mock: TCP, UDP, ARP and ICMP packets. This build currently utilizes scapy-python3 for packet recording: https://github.com/phaethon/scapy Future builds will utilize the multiprocessing engine included but does not filter src/dst ports correctly yet.The license will be full Apache 2 once that migration is done.