awesome-log-data

A pipeline that ingests security log datasets from multiple sources — Windows event logs, cloud audit trails, Linux auditd, network logs, and more — and normalizes them into one consistent record format. The goal is a single, straightforward way to pull structured, high-signal security log data into a downstream project (e.g. for embedding/ML work) without hand-rolling a parser per dataset.

Each dataset keeps its original field structure — records aren't remapped to a shared schema — but every record is wrapped the same way and indexed the same way, so consuming five different log formats looks the same from the calling code's perspective.

Datasets

"Labeled" means each record can be tied back to the specific attack technique that produced it (per the manifest's labeled field) — useful if you want ground truth for a specific behavior, not just a corpus of logs.

Each dataset's exact source URL and license are also recorded per-file in data/manifest.jsonl after ingestion (see Manifest below) — the table above is a summary, not the source of truth.

Requirements

• Python 3.14+
• uv

Install dependencies:

uv sync

Fetching raw data

scripts/fetch_datasets.sh list              # list available dataset_ids
scripts/fetch_datasets.sh <dataset_id>       # fetch one dataset
scripts/fetch_datasets.sh all                # fetch every dataset

This downloads each dataset's raw files into data/raw/<dataset_id>/ (gitignored — raw data isn't checked into the repo). Requires git and curl; ait_lds additionally requires jq, and splunk_attack_data additionally requires git-lfs (its raw samples are stored via LFS; the fetch script pulls a scoped subset of the ~23GB upstream repo rather than the whole thing — only files whose extension could plausibly be a format the adapter supports).

Ingesting a dataset

Once a dataset's raw files are on disk (via the fetch script, or placed there yourself), run the CLI:

uv run python -m awesome_log_data.cli <dataset_id> <path-to-raw-files>

For example:

uv run python -m awesome_log_data.cli otrf data/raw/otrf

This walks the raw files, extracts archives as needed, parses every recognized log file, and writes:

• data/manifest.jsonl — one entry per ingested source file (see Manifest).
• data/parsed/<dataset_id>/ — the parsed records themselves, sharded across fixed-size JSONL files with a shard_index.parquet (plus a small source_ids.jsonl sidecar) for random access (see Reading parsed data).

Re-running the same command is safe and incremental: files already recorded in the manifest (by checksum) are skipped, so only newly added raw files get parsed and appended.

Record format

Each shard file holds the bare parsed record on every line, in that dataset's native field structure — not remapped to a common schema, and not wrapped in any envelope, so the shard files can be read directly as the training corpus:

{"eventName": "GetObject", "...": "..."}

Provenance for each record lives alongside it in shard_index.parquet, not in the record itself (see Reading parsed data):

• source_id — an int identifying the raw file this record came from. It's an index into source_ids.jsonl (one JSON string per line, line number = id) rather than the string itself, to avoid repeating that string once per record — a source file's records are otherwise identical strings repeated thousands of times over. Look the string up via dataset.source_ids[source_id], then that string (<dataset_id>/<file_name>, or <dataset_id>/<file_name>#<hash> if two different files happened to share a file_name — see the manifest's collision handling) in data/manifest.jsonl for the file's source URL, license, checksum, etc.
• record_ref — an offset back into the original raw file (byte offset, array index, or Windows event record ID depending on the format) that can re-derive this exact record from source, independent of the parsed/sharded copy.

Manifest

data/manifest.jsonl has one entry per ingested raw file:

{
  "source_id": "otrf/ec2_proxy_s3_exfiltration/ec2_proxy_s3_exfiltration_2020-09-14011940.json",
  "dataset_id": "otrf",
  "file_name": "ec2_proxy_s3_exfiltration/ec2_proxy_s3_exfiltration_2020-09-14011940.json",
  "source_url": "https://github.com/OTRF/Security-Datasets",
  "license": "GPL-3.0",
  "ingested_at": "2026-08-09",
  "checksum_sha256": "...",
  "bytes": 106600,
  "record_count": 103,
  "labeled": true,
  "record_ref_type": "byte_offset",
  "notes": ""
}

Reading parsed data

Use ShardedDataset for indexed, random-access reads over a dataset's already-parsed records — no need to load the whole corpus into memory:

from awesome_log_data.sharded_dataset import ShardedDataset

dataset = ShardedDataset(Path("data/parsed/otrf"))

len(dataset)             # total record count
dataset[0]                # {"eventName": "GetObject", "...": "..."} - the bare parsed record
dataset.indices_for_source("otrf/ec2_proxy_s3_exfiltration/...json")  # all record indices from one raw file

Each record's provenance is a separate ShardIndexEntry at the same index in dataset.index — source_id (an int; resolve the string via dataset.source_ids, then cross-reference against data/manifest.jsonl) and record_ref (to re-derive the record from its original raw file):

from awesome_log_data.manifest import ManifestStore

manifest = ManifestStore(Path("data/manifest.jsonl"))
entry = manifest.get(dataset.source_ids[dataset.index[0].source_id])
entry.source_url, entry.license, entry.labeled

To re-derive a record directly from its original raw file instead of the parsed copy (e.g. to see the exact original bytes, not just the parsed fields), use the registered adapter's parser together with the manifest entry's record_ref_type and dataset.index[0].record_ref:

from awesome_log_data.adapters import get_adapter

adapter = get_adapter(entry.dataset_id)
source = next(
    s for s in adapter.discover(Path("data/raw/otrf"))
    if s.file_name == entry.file_name
)
source.parser.resolve(source.path, dataset.index[0].record_ref)

This requires the original raw files (data/raw/<dataset_id>/) still present on disk — ShardedDataset alone is sufficient for everything else.