idsdata

Load intrusion-detection datasets by name and version, check them against recorded SHA-256 hashes, and get the right citation. ids-lint flags known quality problems in a dataset CSV file.

Papers that say "we used CIC-IDS2017" can mean several different files. idsdata makes the version part of the name, and refuses to load a file whose hash does not match.

Install

pip install idsdata

It needs Python 3.10 or newer and installs pandas and pyarrow.

Quickstart

idsdata list
idsdata info cic-ids2017 engelen2021
idsdata download cic-ids2017 engelen2021
idsdata verify cic-ids2017 engelen2021

import idsdata

flows = idsdata.load('cic-ids2017', 'engelen2021')
print(flows.shape)  # (2100814, 88)
print(flows['label'].value_counts())
print(idsdata.cite('cic-ids2017', 'engelen2021'))

load keeps every flow and every value as it is in the source files. It gives the columns clean names (Flow Bytes/s becomes flow_bytes_s), keeps the source label as label_raw, and adds four columns:

Keeping "Attempted" flows out of is_attack is a choice this package makes, following the corrected release. It means is_attack and the class name can disagree: a dos_hulk_attempted row has is_attack == False. Decide what you want before training a binary classifier:

flows = flows[~flows['is_attempted']]  # drop them
attack = flows['is_attack'] | flows['is_attempted']  # or count them as attacks

The first load reads the CSV files and writes a parquet cache next to them. Later loads read the cache.

Train/test splits

train, test = idsdata.split('cic-ids2017', 'engelen2021', 'time')

A split has a name, and the name always means the same rows. Both splits put 20% of the flows in the test set and keep every class on both sides.

Things to know:

• The cut is made per class, not once per day. Each attack runs in a short window, so a single cut-off time per day would leave most attack classes out of the test set.
• Timestamps have one-second resolution. Flows that share the second at the cut keep the order they have in the source file.
• A class with two or more flows always gets at least one test flow; a class with a single flow stays in train. Some classes are very small, so their test sets are tiny.
• A split is of the whole release. Filter afterwards, for example train[~train['is_attempted']].

Datasets

For the two original versions, fill in the form, take the zip from the CSVs folder and put it in the directory idsdata where cic-ids2017 original-flows (or original-ml) prints. verify checks the zip and load unpacks it.

Both original versions hold the same 2,830,743 flows in 15 classes. They are kept as published, and four things about them are worth knowing:

• The column Fwd Header Length appears twice in every file. Both are kept; the second is named fwd_header_length_1. In these files the two hold the same values.
• One file of original-flows holds 288,602 lines without any value. load leaves them out. This is the only case where lines of a source file are not returned, and the number is recorded and checked.
• The dash in the three web attack labels is not plain text. original-flows is read as Windows-1252, which makes it an en dash; in original-ml it is the Unicode replacement character. label_raw shows what the file holds, and label is the same in both (web_attack_xss).
• The timestamps of original-flows come in more than one format and without AM or PM, so only the random split is offered for the original versions.

Data policy

• The package contains no dataset files and this repository will never contain any.
• Nothing is mirrored. Files come from the official host, or you download them yourself.
• idsdata download works only where the host offers a direct link. It prints the terms and asks before it fetches anything (--yes skips the question).
• A release behind a request form is never fetched by the package. You fill in the form and place the files; the package checks them.
• Every recorded hash was computed from a real download, and the registry records the date of that download. If the host replaces a file, verification fails and says so. It also fails when a table is missing and the archive it comes from is not there.

Files are kept in ~/.cache/idsdata/<name>/<version>/. Set IDSDATA_DIR or pass --data-dir to use another place. idsdata where NAME VERSION prints the directory.

How to cite

To cite idsdata itself, use CITATION.cff (the "Cite this repository" button on GitHub).

Cite the papers behind the data you use. This prints the BibTeX entries:

idsdata cite cic-ids2017 engelen2021

For engelen2021 that is the paper that corrected the dataset and the paper that introduced it:

• G. Engelen, V. Rimmer and W. Joosen, "Troubleshooting an Intrusion Detection Dataset: the CICIDS2017 Case Study", 2021 IEEE Security and Privacy Workshops (SPW), 2021. https://doi.org/10.1109/SPW53761.2021.00009
• I. Sharafaldin, A. Habibi Lashkari and A. Ghorbani, "Toward Generating a New Intrusion Detection Dataset and Intrusion Traffic Characterization", ICISSP 2018. https://doi.org/10.5220/0006639801080116

ids-lint

ids-lint Friday-WorkingHours.csv
ids-lint --release cic-ids2017 engelen2021 *.csv
ids-lint --rules

The exit status is 1 when a rule of severity error fires. With --strict, warnings count as well. --release also checks that every label string in the file belongs to that release, and reads the file in the text encoding recorded for it.

ids-lint --rules prints the full reference for each source. The source of IDS001 is the authors' documentation page, which is not peer reviewed. Flood et al. describe near-duplicate flows; IDS004 reports only exact repeats. The sources of IDS003 describe flows that carry the wrong label; IDS003 can only see label strings that are missing or that a release does not define.

The linter reports and changes nothing. A warning is a prompt to decide, and what to do with the flagged rows or columns is up to you.

How it was tested

• Unit tests. The test suite runs on every pull request, on Python 3.10 and 3.12 (GitHub Actions, Linux), together with ruff and basedpyright. They use small made-up files only, never real data: checksums, the download rules, loading, the cache, label mapping, both splits and every lint rule. The random split is pinned to exact rows, so a change in pandas that moves rows between train and test is noticed. One test checks that the rule table above matches the code.
• Real data, by hand. The numbers in the registry (hashes, sizes, row counts, column names, label strings) were read from real downloads made on 2026-10-09, not typed in. The MD5 files the publisher ships with the original release match the two zips. These checks were then run on Windows 11 with Python 3.12, and are not part of CI because the data cannot be redistributed:

The class counts of the two original versions are equal label by label.

What was not tested: other operating systems and Python versions with real data, and idsdata download beyond the one release that has a direct link.

Not in this version

• A time split for the original versions.
• Other datasets.

Acknowledgements

The CI setup is adapted from DFAIR-LAB-Augusta/XSecIoT (Seth Barrett).