cesnet-datazoo 0.1.5

A toolkit for large network traffic datasets

The goal of this project is to provide tools for working with large network traffic datasets and to facilitate research in the traffic classification area. The core functions of the cesnet-datazoo package are:

• A common API for downloading, configuring, and loading of three public datasets of encrypted network traffic.
• Extensive configuration options for:
  • Selection of train, validation, and test periods.
  • Selection of application classes and splitting classes between known and unknown.
  • Data transformations, such as feature scaling.
• Built on suitable data structures for experiments with large datasets. There are several caching mechanisms to make repeated runs faster, for example, when searching for the best model configuration.
• Datasets are offered in multiple sizes to give users an option to start the experiments at a smaller scale (also faster dataset download, disk space, etc.). The default is the S size containing 25 million samples.

:brain: :brain: See a related project CESNET Models providing pre-trained neural networks for traffic classification. :brain: :brain:

:notebook: :notebook: Example Jupyter notebooks are included in a separate CESNET Traffic Classification Examples repo. :notebook: :notebook:

Datasets

The following datasets are available in the cesnet-datazoo package:

Name	CESNET-TLS22	CESNET-QUIC22	CESNET-TLS-Year22
Protocol	TLS	QUIC	TLS
Published in	2022	2023	2023
Collection duration	2 weeks	4 weeks	1 year
Collection period	4.10.2021 - 17.10.2021	31.10.2022 - 27.11.2022	1.1.2022 - 31.12.2022
Application count	191	102	180
Available samples	141392195	153226273	507739073
Available dataset sizes	XS, S, M, L	XS, S, M, L	XS, S, M, L
Cite	https://doi.org/10.1016/j.comnet.2022.109467	https://doi.org/10.1016/j.dib.2023.108888
Zenodo URL	https://zenodo.org/record/7965515	https://zenodo.org/record/7963302
Related papers		https://doi.org/10.23919/TMA58422.2023.10199052

Installation

Install the package from pip with:

pip install cesnet-datazoo

or for editable install with:

pip install -e git+https://github.com/CESNET/cesnet-datazoo

Examples

Initialize dataset to create train, validation, and test dataframes

from cesnet_datazoo.datasets import CESNET_QUIC22
from cesnet_datazoo.config import DatasetConfig, AppSelection

dataset = CESNET_QUIC22("/datasets/CESNET-QUIC22/", size="XS")
dataset_config = DatasetConfig(
    dataset=dataset,
    apps_selection=AppSelection.ALL_KNOWN,
    train_period_name="W-2022-44",
    test_period_name="W-2022-45",
)
dataset.set_dataset_config_and_initialize(dataset_config)
train_dataframe = dataset.get_train_df()
val_dataframe = dataset.get_val_df()
test_dataframe = dataset.get_test_df()

The DatasetConfig class handles the configuration of datasets, and calling set_dataset_config_and_initialize initializes train, validation, and test sets with the desired configuration. Data can be read into Pandas DataFrames as shown here or via PyTorch DataLoaders. See CesnetDataset reference.

See more examples in the documentation.

Papers

• DataZoo: Streamlining Traffic Classification Experiments
  Jan Luxemburk and Karel Hynek
  CoNEXT Workshop on Explainable and Safety Bounded, Fidelitous, Machine Learning for Networking (SAFE), 2023

Acknowledgments

This project was supported by the Ministry of the Interior of the Czech Republic, grant No. VJ02010024: Flow-Based Encrypted Traffic Analysis.