ids-evaluation-framework 0.4.0

A framework for evaluating intrusion detection systems for IoT botnets.

IDS Evaluation Framework

A comprehensive, modular, and configurable framework for evaluating Machine Learning-based Intrusion Detection Systems (IDS).

• IDS Evaluation Framework
  • Features
  • Installation
    • Prerequisites
    • Python Installation
    • Native Installation
    • Docker Installation
  • Quick Start
    • 1. Create a Configuration File
    • 2. Prepare Your Data
    • 3. Run Evaluation
  • Usage
    • CLI Commands
    • Evaluation Flags
    • Makefile Targets
  • Configuration
    • Key Configuration Sections
  • Output Structure
  • Plugin Development
  • Development
  • Tests
• Additional Information
  • BibTeX entry
  • Creating Issues
  • License
  • Programmier Praktikum

Features

• Modular Plugin Architecture: Easily extend the framework with custom IDS models, metrics, and adversarial attacks
• Flexible Data Pipeline: Load, preprocess, and split datasets with configurable preprocessing steps and feature selection
• Multiple Evaluation Modes: Support for intra-dataset, cross-dataset, and k-fold cross-validation evaluation
• Comprehensive Metrics: Built-in static metrics (accuracy, F1, precision, recall, ROC-AUC, etc.) and runtime metrics (CPU, RAM, training time)
• Adversarial Robustness Testing: Evaluate model robustness against adversarial attacks (FGSM, noise perturbation, junk data injection)
• Reproducible Results: Hash-based output organization ensures consistent experiment tracking
• Flexible Deployment: Run natively with Python or via Docker

Installation

Prerequisites

• Python 3.13+
• uv (recommended) or pip

Python Installation

pip3 install ids-evaluation-framework

Native Installation

# Install dependencies (uv should be in your $PATH)
uv sync

# Verify installation
uv run ids-eval version

Docker Installation

# Configure environment variables
cp .env.example .env
# Edit .env to set your data paths

# Build and run via Docker Compose
docker compose build
docker compose run --rm ids-eval version

The image is built locally from the provided Dockerfile.

Quick Start

1. Create a Configuration File

Copy the example configuration and adjust it to your needs:

cp examples/run_config/example.config.yml examples/run_config/my_config.yml

2. Prepare Your Data

Run the data preparation pipeline:

uv run ids-eval dataset <run_config>

3. Run Evaluation

Execute the evaluation pipeline:

uv run ids-eval evaluate <run_config>

Usage

CLI Commands

The framework provides two main commands:

Command	Description
ids-eval dataset <config.yml>	Run dataset pipeline
ids-eval evaluate <config.yml>	Run evaluation pipeline

Evaluation Flags

Flag	Description
--train-only	Only train models, skip testing phase
--force-train	Force retraining, ignore saved models
--force-model	Load saved models without config hash validation
--clear-checkpoints	Clear evaluation checkpoints before running

Makefile Targets

make dataset CONFIG=<config.yml>          # Run dataset pipeline
make evaluate CONFIG=<config.yml>         # Run evaluation pipeline
make docker-dataset CONFIG=<config.yml>   # Run dataset pipeline via Docker
make docker-evaluate CONFIG=<config.yml>  # Run evaluation via Docker
make help                                 # Show all available targets

Configuration

The framework uses YAML configuration files. See examples/run_config/example.config.yml for a fully documented example.

Key Configuration Sections

• general: Run name, paths, random seed
• data_manager: Dataset loading, preprocessing, feature selection, train/test split
• evaluation: IDS models, metrics, adversarial attacks

Output Structure

All outputs are organized in hash-based directories for reproducibility:

out/
├── processed_datasets/<hash>/    # Preprocessed datasets
├── saved_models/<hash>/          # Trained models
└── reports/<hash>/               # Evaluation reports
    ├── config.yaml               # Configuration used
    ├── dataset_report.yaml       # Dataset statistics
    ├── ids_report.yaml           # Detailed evaluation results
    └── evaluation_summary.yaml   # Aggregated summary

The configuration hash is displayed at startup:

Your config hash is: a1b2c3d4

Plugin Development

The framework supports four types of plugins:

Plugin Type	Directory	Base Class
IDS Models	plugin_ids/	AbstractIDSConnector
Static Metrics	plugin_static_metric/	AbstractStaticMetric
Runtime Metrics	plugin_runtime_metric/	AbstractRuntimeMetric
Adversarial Attacks	plugin_adversarial/	AbstractAdversarialAttack

See the existing plugins in each directory for implementation examples.

Development

make setup      # Install dependencies
make test       # Run tests
make lint       # Check code style
make format     # Format code

Tests

The test suite lives in tests/ and is run with pytest:

uv run pytest          # run all tests
make test              # equivalent target

tests/test_metrics.py contains two tests that verify the mathematical correctness of two static metrics:

Test	Checks
test_pr_auc_average_precision	PR-AUC (Average Precision) against a known reference value
test_robustness_index_normalized_area	Robustness Index equals the normalized area under the accuracy–perturbation curve

Run a single test:

uv run pytest tests/test_metrics.py -k pr_auc

Additional Information

BibTeX entry

Please cite this project using the following bibtex entry:

@inproceedings{}

Creating Issues

If you find any bugs, bad patterns, performance issues, etc. do not hesitate to open an issue.
Any new features which should be part of the evaluation has to be underlined by peer-reviewed publications. This counts for new examples as well. All examples are reproduced publications except baseline models.
If you want to write a publication for a new feature, fork this project and we will merge the pull request after acceptance.

License

See LICENSE for details.

Programmier Praktikum

This project was extended in the Programmier-Praktikum with a raw-pcap ingestion path (nfstream) and two metrics, the PR-AUC and the Robustness Index (RI) demonstrated on the Apollon MAB IDS.

Architecture A layered plugin pipeline setup entirely by a YAML run-config: dataset_pipeline builds the dataset(s) (CSV, or pcap via NFStream + time-window labeling), evaluation_pipeline trains the IDS plugins and runs the adversarial/robustness sweep, metrics_pipeline computes the metrics. IDS models, metrics and attacks are swappable plugins resolved through the registry via an abstract class.

Two example runs are provided and both evaluate Apollon (attacked via a trained surrogate) and report PR-AUC, the RI and further metrics under an FGSM sweep.

1. pcap run (single day). examples/run_config/PP_EXAMPLE.yml ingests the CICIDS2017 Friday capture directly from raw pcap via nfstream (+ time-window labeling):

cd examples/run_config
uv run ids-eval dataset  PP_EXAMPLE.yml
uv run ids-eval evaluate PP_EXAMPLE.yml

2. full-dataset run (all days, CSV). examples/run_config/PP_EXAMPLE_full.yml runs on the complete CICIDS2017, no nfstream. The much larger sample gives more robust results than the friday pcap run:

cd examples/run_config
uv run ids-eval dataset  PP_EXAMPLE_full.yml
uv run ids-eval evaluate PP_EXAMPLE_full.yml

For further information to correctly configure the run please refer to examples/run_config/example.config.yml.

Dataset.

• For run 1 download Friday-WorkingHours.pcap.
• For run 2 download the full MachineLearningCVE / GeneratedLabelledFlows CSV set (the whole labeled CICIDS2017).

Both are available from the CIC-IDS2017 dataset (https://www.unb.ca/cic/datasets/ids-2017.html). Place the files in examples/run_config/raw_data/cic_ids_2017_flow/, matching the base_path/subpath in each config. Both configs are run with examples/run_config/ as the working directory, so out/ and raw_data/ resolve inside that directory.