A toolkit for large network traffic datasets
Project description
The goal of this project is to provide tools for working with large network traffic datasets and to facilitate research in the traffic classification area. The core functions of the cesnet-datazoo
package are:
- A common API for downloading, configuring, and loading of three public datasets of encrypted network traffic.
- Extensive configuration options for:
- Selection of train, validation, and test periods.
- Selection of application classes and splitting classes between known and unknown.
- Feature scaling.
- Built on suitable data structures for experiments with large datasets. There are several caching mechanisms to make repeated runs faster, for example, when searching for the best model configuration.
- Datasets are offered in multiple sizes to give users an option to start the experiments at a smaller scale (also faster dataset download, disk space, etc.). The default is the
S
size containing 25 million samples.
Datasets
The package is able to handle the following datasets:
Name | CESNET-TLS22 | CESNET-QUIC22 | CESNET-TLS-Year22 |
---|---|---|---|
Protocol | TLS | QUIC | TLS |
Published in | 2022 | 2023 | 2023 |
Collection duration | 2 weeks | 4 weeks | 1 year |
Collection period | 4.10.2021 - 17.10.2021 | 31.10.2022 - 27.11.2022 | 1.1.2022 - 31.12.2022 |
Application count | 191 | 102 | 182 |
Available samples | 141720670 | 153226273 | 507739322 |
Available dataset sizes | XS, S, M, L | XS, S, M, L | XS, S, M, L |
Cite | https://doi.org/10.1016/j.comnet.2022.109467 | https://doi.org/10.1016/j.dib.2023.108888 | |
Zenodo URL | https://zenodo.org/record/7965515 | https://zenodo.org/record/7963302 | |
Related papers | https://doi.org/10.23919/TMA58422.2023.10199052 |
Installation
Install the package from pip with:
pip install cesnet-datazoo
or for editable install with:
pip install -e git+https://github.com/CESNET/cesnet-datazoo
Examples
Initialize dataset to create train, validation, and test dataframes
from cesnet_datazoo.datasets import CESNET_QUIC22
from cesnet_datazoo.config import DatasetConfig, AppSelection
dataset = CESNET_QUIC22("/datasets/CESNET-QUIC22/", size="XS")
dataset_config = DatasetConfig(
dataset=dataset,
apps_selection=AppSelection.ALL_KNOWN,
train_period_name="W-2022-44",
test_period_name="W-2022-45",
)
dataset.set_dataset_config_and_initialize(dataset_config)
train_dataframe = dataset.get_train_df()
val_dataframe = dataset.get_val_df()
test_dataframe = dataset.get_test_df()
The DatasetConfig
class handles the configuration of datasets, and calling set_dataset_config_and_initialize
initializes train, validation, and test sets with the desired configuration.
Data can be read into Pandas DataFrames as shown here or via PyTorch DataLoaders. See CesnetDataset
reference.
See more examples in the documentation.
Acknowledgements
This work was supported by the Ministry of the Interior of the Czech Republic, grant No. VJ02010024: Flow-Based Encrypted Traffic Analysis.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for cesnet_datazoo-0.0.13-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 297552011b5f5d7b3b1c3407f11a360882deed99548aa93d31d74a954d8b2275 |
|
MD5 | 8f992b2c99c4d8b25b0e3661fd099377 |
|
BLAKE2b-256 | 08790947963d1010d2a036fd5288ebe423919e19c1cfe8b832d5c057c8b175c2 |