tcpfeature

A blazing-fast, cross-platform network traffic feature extraction tool specifically designed for Machine Learning and Cyber Security research. Built with Cython and linked directly to native libpcap/Npcap architectures to achieve maximum throughput while keeping logic completely compiled and secure.

🚀 Key Features

• Blazing Fast Performance: Powered by Cython C-Extensions to parse packets at the memory layer with zero Python overhead.
• ML-Ready Output: Extract advanced statistical features directly into structured formats compatible with standard network benchmarks.
• Cross-Platform Binary Wheels: Native support for Windows (AMD64), Linux (x86_64), and macOS (Apple Silicon M1/M2/M3/M4).
• Live & Offline Modes: Continuous streaming feature extraction from live network cards (using Python Generators) or batch processing from standard PCAP files.
• Integrated Alert Manager: High-speed Snort raw alert log parser with automated bitwise IP subnet checking to calculate directional network metrics (is_src_internal, is_dst_internal).

🛠️ System Requirements & Prerequisites

Since tcpfeature communicates directly with kernel-level packet capture facilities, the host system requires network capture libraries installed:

1. Linux (Ubuntu/Debian/CentOS)

# Ubuntu/Debian
sudo apt-get install libpcap-dev

# CentOS/RHEL/Fedora
sudo yum install libpcap-devel

2. macOS

macOS comes with libpcap pre-installed natively. No additional configuration is required.

3. Windows

Download and install the Npcap Driver from the official website: https://npcap.com/ (Ensure you check WinPcap API-compatible mode during installation).

The runtime environment automatically utilizes the optimized Windows WinSock2 stack.

📦 Installation

Install the compiled binary wheels directly via pip:

pip install tcpfeature

💻 Quick Start & Usage Examples

tcpfeature exposes a collection of high-performance, stateless Functional APIs at the package root level, making it seamlessly compatible with the standard Python Data Science ecosystem.

1. Offline Batch Feature Extraction (PCAP to ML Features)

Extract the comprehensive 29 network layer feature vectors from an offline packet capture file into structured arrays.

import tcpfeature
import json

# Define path to target pcap file
pcap_path = "data/pcaps/sample.pcap"

# Execute the high-speed Cython feature extraction core
dataset = tcpfeature.run_extraction(pcap_path, mode="offline", version="100")

if dataset:
    print(f"[+] Successfully extracted {len(dataset)} connection records.")
    # Print the first feature vector formatted as JSON
    print(json.dumps(dataset[0], indent=4))

2. Live Network Traffic Feature Streaming (Generator Mode)

Capture network packets directly from a physical network interface in real-time. The Cython engine evaluates moving traffic structures and yields ready-to-use data dictionaries through a continuous Python Generator.

import tcpfeature

# Automatically detect the optimal active network interface card
target_device = tcpfeature.get_best_device()

print(f"[*] Starting live traffic feature extraction on: {target_device}")
try:
    # Stream features continuously through the generator
    for ml_features in tcpfeature.run_extraction(target_device, mode="live", version="100"):
        proto = ml_features.get('protocol_type')
        src_port = ml_features.get('src_port')
        dst_port = ml_features.get('dst_port')
        src_bytes = ml_features.get('src_bytes', 0)
        count_2sec = ml_features.get('count', 0)
        
        print(f"[{proto}] {src_port}->{dst_port} | Bytes: {src_bytes} | Windowed Count (2s): {count_2sec}")
except KeyboardInterrupt:
    print("\n[+] Live feature streaming stopped securely.")

3. Basic Stream Reader (For Custom Feature Engineering)

If you need to engineer your own custom topological features, use the lightweight stream reader to pull raw isolated header fields from the memory stack sequentially.

import tcpfeature

pcap_file = "data/pcaps/sample.pcap"
total_bytes = 0
tcp_count = 0

# Loop through raw C-extracted layer-3/layer-4 boundary fields
for packet in tcpfeature.run_basic_capture(pcap_file, mode="offline"):
    frame_len = packet.get("frame_length", 0)
    proto = packet.get("protocol_type", "Others")
    
    # Apply custom real-time cumulative logic (Feature Engineering)
    total_bytes += frame_len
    if proto == "TCP":
        tcp_count += 1
        
    if tcp_count % 100 == 0:
        print(f"[Custom Metric] Accumulated Bytes: {total_bytes} | Total TCP Frames: {tcp_count}")

4. Automated End-to-End Dataset Labeling (Snort Integration)

Construct fully labeled classification training datasets (DARPA29F or HNET3A standards) by cross-referencing your network feature vectors with standard Snort alert system logs

import tcpfeature

feature_csv = "data/csv/pcap_features.csv"
snort_log = "data/alerts/alert.log"

# Step A: Parse raw Snort alerts and compute bitwise IP ranges (is_src_internal/is_dst_internal)
alert_csv_path = tcpfeature.convert_snort_log(snort_log)

# Step B: Match extracted traffic records with alert logs inside a 2-second moving time window
final_dataset_csv = "data/final/training_data_labeled.csv"
tcpfeature.build_labeled_dataset(
    feature_csv=feature_csv,
    alert_csv=alert_csv_path,
    output_csv=final_dataset_csv,
    time_window=2.0
)
print(f"[+] ML Training Dataset generated successfully at: {final_dataset_csv}")

📊 Extracted Feature Architecture

The core Cython engine evaluates network sessions and maps them into the standardized KDDCup'99 dataset features.

🎓 Citation & Academic Reference

If you use tcpfeature or the accompanying DARPA29F dataset in your research, academic publications, or permitted educational configurations, please cite the original peer-reviewed paper that introduced this methodology:

IEEE Format

Van Kha Nguyen, Minh Nhat Quang Truong, Van Lam Le, Quyet Thang Le, and Thanh Hai Nguyen, "A Novel Approach for Data Collection and Network Attack Warning," In 2019 11th International Conference on Knowledge and Systems Engineering (KSE), IEEE, 2019, pp. 1-6, doi: 10.1109/KSE.2019.8919494.

BibTeX (For LaTeX Researchers)

@inproceedings{nguyen2019novel,
  title={A Novel Approach for Data Collection and Network Attack Warning}, 
  author={Nguyen, Van Kha and Truong, Minh Nhat Quang and Le, Van Lam and Le, Quyet Thang and Nguyen, Thanh Hai},
  booktitle={2019 11th International Conference on Knowledge and Systems Engineering (KSE)},
  pages={1--6},
  year={2019},
  publisher={IEEE},
  doi={10.1109/KSE.2019.8919494},
  note={Funded by Can Tho Department of Science and Technology}
}

🛡️ License

This project is licensed under a Custom Proprietary Academic License.

• Academic & Research Use: Free to use for non-commercial research, academic publications, and educational purposes, provided that proper citation is given to the original paper.
• Commercial Use: Strictly prohibited. Any commercial deployment, production usage, or redistribution of this binary wheel requires a separate commercial license from the author.
• Source Code: The underlying Cython/C source code is proprietary and not open to the public. Reverse engineering or decompiling the binaries is strictly forbidden.

👥 Authors & Contact

Author: dzokha

Email: info@ipz.vn