DeepLogBot

Bot detection and traffic classification for scientific data repository logs.

Overview

DeepLogBot (CLI: deeplogbot) detects and classifies download patterns in scientific data repository logs, distinguishing between:

• Organic users — Human researchers with natural download patterns
• Bots — Automated scrapers, crawlers, and coordinated bot farms
• Download hubs — Legitimate mirrors, institutional pipelines, and data aggregators

Applied to the PRIDE Archive (159M download records), the system identified that 88% of traffic is bot-generated. After filtering, 19.1M clean downloads remain across 34,085 datasets and 213 countries.

Classification Categories

Each geographic location is classified into one of three categories:

• Bot — Automated scrapers, crawlers, and coordinated bot farms
• Hub — Legitimate automation: institutional mirrors, CI/CD pipelines, educational workshops
• Organic — Human researchers with natural download patterns

Classification Methods

DeepLogBot provides 2 classification methods:

Benchmarked on a 1M-record sample with manually curated ground truth.

Rule-Based (--classification-method rules)

Hierarchical threshold classification using YAML-configurable rules. Fast, interpretable, and requires no training. Best for production use with known patterns.

Deep Architecture (--classification-method deep)

Multi-stage learned pipeline:

1. Seed Selection — Identify high-confidence bot/organic/hub seeds from feature distributions
2. Organic VAE — Learn the normal-behavior manifold; score reconstruction error
3. Deep Isolation Forest — Non-linear anomaly detection on VAE latent space
4. Temporal Consistency — Modified z-score spike detection (no fixed thresholds)
5. Fusion Meta-Learner — Gradient-boosted combination of all anomaly signals

Additional components:

• Soft priors — Pre-filter signals encoded as continuous features (no hard lockout)
• Reconciliation — Override thresholds for cases where pipeline and pre-filter disagree
• Hub protection — Prevent legitimate automation from being classified as bots
• Post-classification — Hub protection and final label assignment

Installation

pip install -e .

Requirements

• Python 3.9+
• pandas, numpy, scikit-learn, scipy, duckdb
• Optional: torch (for deep method)

Usage

Command Line

# Rule-based classification (default)
deeplogbot -i data.parquet -o output/

# Deep architecture
deeplogbot -i data.parquet -o output/ -m deep

# With sampling for large datasets
deeplogbot -i data.parquet -o output/ -m deep --sample-size 1000000

Options:

Python API

from deeplogbot import run_bot_annotator

# Rule-based classification
results = run_bot_annotator(
    input_parquet='data.parquet',
    output_dir='output/',
    classification_method='rules'
)

# Deep architecture
results = run_bot_annotator(
    input_parquet='data.parquet',
    output_dir='output/',
    classification_method='deep'
)

print(f"Bots detected: {results['bot_count']}")
print(f"Hubs detected: {results['hub_count']}")

Project Structure

deeplogbot/
├── __init__.py                  # Package exports
├── main.py                      # CLI entry point and pipeline
├── config.py                    # Configuration loading
├── config.yaml                  # Main configuration file
│
├── features/                    # Feature extraction (~117 features)
│   ├── base.py                  # Base extractor class
│   ├── schema.py                # Log schema definitions
│   ├── registry.py              # Feature documentation registry
│   └── providers/
│       └── ebi/                 # EBI/PRIDE provider
│           ├── ebi.py           # Location feature extraction
│           ├── behavioral.py    # Behavioral features
│           ├── discriminative.py # Discriminative features
│           ├── timeseries.py    # Time series features
│           └── schema.py        # EBI-specific schema
│
├── models/
│   ├── isoforest/               # Isolation Forest anomaly detection
│   │   └── models.py
│   └── classification/          # Classification methods
│       ├── rules.py             # Rule-based hierarchical classifier
│       ├── deep_architecture.py # Deep pipeline orchestration
│       ├── seed_selection.py    # High-confidence seed identification
│       ├── organic_vae.py       # VAE + Deep Isolation Forest
│       ├── temporal_consistency.py # Modified z-score spike detection
│       ├── fusion.py            # Gradient-boosted meta-learner
│       ├── post_classification.py # Hub protection & label finalization
│       └── feature_validation.py  # Feature usage validation
│
├── reports/                     # Output generation
│   ├── reporting.py             # Text report generation
│   ├── annotation.py            # Parquet annotation
│   ├── statistics.py            # Summary statistics
│   ├── html_report.py           # Interactive HTML reports
│   └── visualizations.py        # Charts and plots
│
├── utils/                       # Utilities
│   └── geography.py             # Geographic lookups
│
└── providers/
    └── base_taxonomy.yaml       # Classification taxonomy

Configuration

Configuration is in deeplogbot/config.yaml:

isolation_forest:
  contamination: 0.15
  n_estimators: 200
  random_state: 42

classification:
  rule_based:
    bots:
      require_anomaly: true
      patterns:
        - downloads_per_user: {max: 100}
          unique_users: {min: 5000}
    hubs:
      require_anomaly: true
      patterns:
        - downloads_per_user: {min: 500}

deep_reconciliation:
  override_threshold: 0.7
  strict_threshold: 0.8

Classifying a Download Parquet File

Given a parquet file of download logs (one row per download event), DeepLogBot aggregates records by geographic location, extracts ~117 behavioral and discriminative features, classifies each location as bot/hub/organic, and writes a new annotated parquet with classification columns appended to every row.

Input format

The input parquet must contain at minimum:

Running classification

# Classify with the deep method (recommended) — writes <input>_annotated.parquet
deeplogbot -i downloads.parquet -o output/ -m deep

# Classify with rules (faster, no torch required)
deeplogbot -i downloads.parquet -o output/ -m rules

# For large files, sample first to speed up classification
deeplogbot -i downloads.parquet -o output/ -m deep --sample-size 5000000

The annotated parquet is written to the output directory with an _annotated suffix (e.g., output/downloads_annotated.parquet). You can also specify an explicit output path:

deeplogbot -i downloads.parquet -o output/ -m deep --output output/classified.parquet

Output strategies

Using the annotated parquet

import duckdb

conn = duckdb.connect()
df = conn.execute("""
    SELECT accession, country, year,
           is_bot, is_hub, is_organic,
           is_bot, is_hub, is_organic
    FROM read_parquet('output/downloads_annotated.parquet')
    LIMIT 10
""").df()

# Filter to clean (non-bot, non-hub) downloads
clean = conn.execute("""
    SELECT accession, country, COUNT(*) as downloads
    FROM read_parquet('output/downloads_annotated.parquet')
    WHERE is_bot = false AND is_hub = false
    GROUP BY accession, country
    ORDER BY downloads DESC
""").df()

Output Format

The annotated output parquet contains:

Reports generated:

• bot_detection_report.txt — Summary with counts and breakdowns
• location_analysis.csv — Per-location features and classifications
• Interactive HTML report (if enabled)

License

MIT