pirogue-mongoose 1.0.5

A Python daemon and library for collecting, enriching, storing and forwarding network events from Suricata and NFStream.

Mongoose

Collect, enrich, store and forward Suricata alerts and network flows.

Website | Documentation | GitHub | Support

Mongoose is a Python daemon and library for collecting, enriching, storing and forwarding network events from Suricata and NFStream.

It runs a thread-safe pub-sub pipeline (ProcessingQueue) where collectors publish raw events to topics and subscribers — enricher, SQLite store, forwarders — consume them concurrently.

Key features

• Modular collectors: Suricata EVE (alerts and netflow via Unix socket), NFStream (live packet capture from a network interface).
• Automatic enrichment: traffic direction (inbound / outbound / local), Community ID calculation, reverse DNS hostname lookup, event type classification, and flow risk scoring via a configurable severity cache.
• GeoIP enrichment: MaxMind (GeoLite2-ASN, GeoLite2-City, GeoLite2-Country) and IP66 databases, with daily automatic database updates.
• Pluggable forwarders: local file output, HTTP(S) webhooks (immediate, bulk or periodic modes with retry logic and multiple authentication methods), and Discord (rich embed formatting).
• Topic filtering: forwarders can be scoped to specific topics and filtered by event attributes.
• Drop-in webhook configuration: new webhook forwarders can be added at runtime by dropping a YAML file into a watched directory — no restart required.
• SQLite storage: enriched events are persisted with configurable history pruning by record count or age.
• Sharded LRU cache: thread-safe severity cache with optional TTL used for flow risk scoring.
• Singleton engine: a single Engine instance manages the full component lifecycle (start, stop, reload).
• Systemd and PiRogue integration: the CLI daemon supports sd_notify and reads the isolated interface from pirogue-admin-client when available.

Pipeline topics

Events flow through the following pub-sub topics:

Topic	Description
network-dpi	Raw DPI flows from NFStream
network-alert	Raw Suricata alerts
network-flow	Raw Suricata netflow records
enriched-network-dpi	Enriched DPI flows
enriched-network-alert	Enriched Suricata alerts
enriched-network-flow	Enriched netflow records

Installation

Install in a virtual environment and editable mode for development:

python -m venv .venv && source .venv/bin/activate
pip install -e .

CLI usage

The package installs a mongoosed daemon entry-point:

# show top-level help
mongoosed --help

# run with a configuration file
mongoosed --config /etc/mongoose/mongoose.yaml

# override the network interface used by NFStream
mongoosed --config mongoose.yaml --interface eth0

# set logging verbosity
mongoosed --config mongoose.yaml --logging-level DEBUG

Python library usage

import time
from mongoose.core.engine import Engine

engine = Engine("config.yaml")
engine.start()
time.sleep(6)
engine.stop()

Configuration

Configured via a YAML file. All keys live under a top-level configuration key.

configuration:
  collector:
    suricata:
      socket_path: "/run/suricata.socket"  # Suricata Unix socket
      collect_alerts: true                 # collect Suricata alerts
      collect_netflow: false               # collect Suricata netflow records
      enable: true

    nf_stream:
      interface: "eth0"          # network interface for live capture
      active_timeout: 120        # seconds before an active flow expires
      enable: false

  enrichment:
    geoip:
      source: "ip66"             # "ip66" (default) or "maxmind"
      enable: true

  forwarder:
    webhooks:
      - url: "https://hooks.example.com/ingest"
        auth_type: "bearer"      # none | basic | bearer | header
        auth_token: "${WEBHOOK_AUTH_TOKEN}"
        verify_ssl: true
        retry_count: 3
        retry_delay: 5.0
        timeout: 10.0
        mode: "immediate"        # immediate | bulk | periodic
        bulk_size: 10
        periodic_interval: 5.0
        periodic_rate: 10
        topics:
          - "enriched-network-dpi"
          - "enriched-network-alert"
        enable: true

  database_path: "mongoose.db"

  history:
    max_duration_days: 14  # keep records for at most 14 days
    max_records: null      # optional hard cap on row count per table
    enable: true

  cache:
    severity:
      max_size: 1024       # maximum entries in the severity LRU cache
      ttl_seconds: null    # optional TTL; null means entries never expire
      enable: true

  extra_configuration_dir: "/var/lib/mongoose"

Drop-in webhook configuration

Place a YAML file matching the WebhookForwarderConfiguration schema inside <extra_configuration_dir>/webhook.d/. The engine watches this directory and activates new forwarders when a file is created, and deactivates them when it is deleted.

Development

This project uses UV and Hatching has build tools.

Once checked out, set up the project development environment with the following command:

uv sync --all-groups

Pre-commit hooks

Install the pre-commit hooks:

uvx pre-commit install

These hooks can be run at any time with tox:

tox run -e fix

Create a new release

Hatchling uses the Debian changelog to define the Python package version and authors:

dch -i  # Increment the Debian package version
git tag -a "v1.x.x" -m "message"  # New tag to publish the Python package on PyPi