sys-inspector - eBPF-based System Inspector and Audit Tool

Language / Idioma: English | Português

Sys-Inspector is an advanced observability and forensic tool powered by eBPF (Extended Berkeley Packet Filter).

Unlike traditional tools that poll /proc periodically, Sys-Inspector hooks directly into the Linux Kernel to capture events (process execution, file I/O, network connections) in real-time.

Features (v1.4.0)

• Fixed in v1.4.0 - The signing identity survives a redeploy on a correctly configured agent: Three defects found on the bench inside the 1.3.0 fix: the inheritance failed on agents holding only the analyst public key, declaring half of the key pair regenerated the other half, and an agent without the analyst private key was refused at startup. verify_tls is now documented in conf/config.yaml.

• New in v1.4.0 - Finding the capture that matters: The navigation bar shows each capture's type, and a capture taken while the test scenario runs is marked as such however it was started. The SSH and container badges no longer light up on host processes, and the interface now says Hosts and Report, with capture (agent) distinct from collection (server).

• New in v1.3.0 - A signing identity that survives a redeploy: The key that signs captures now lives where state lives, next to the database, and is inherited from the old location rather than regenerated. Every capture carries the fingerprint of the key that signed it and says how that key got there. A new key is legitimate; being born silently is not, because the same agent signing with two keys and nothing explaining it is exactly what the opposing side looks for.

• New in v1.3.0 - An agent that can actually be idle: Off by default. When on, the heavy capture stops running every cycle and happens for a reason: the declared cadence, an analyst command, or the agent's first cycle. A cycle that does not capture writes down why, with the time remaining, so an idle agent never looks like a stuck one.

• New in v1.2.0 - The report as an exhibit: It downloads as a file named after the host and the moment of COLLECTION, always complete, built by the same code path that draws the screen, and without the navigation bar, because a dead button in a forensic document is worse than no button. The bar itself now moves between captures (previous, next, latest, "capture N of M"), and an arrow with nowhere to go stays visible and dimmed rather than vanishing: the edge of the collection is information.

• New in v1.2.0 - Absence answers, on screen: A filter matching nothing says how many processes it examined instead of just emptying the tree, and the jump to the process now appears on every finding that concerns one, not only on those whose reported path happens to be running. When the process is gone from the capture, the screen says so.

• New in v1.1.0 - Directed acquisition and referral: A suspect file or memory region is hashed, excerpted and, when it fits the declared byte budget, copied - always recording the hash SCOPE, so a partial hash is never read as identifying the whole object. Every finding can also state which bench analysis concludes what it cannot, why, and on which object: the tool collects enough to identify and direct, never the mass that proves, and that is only honest when it says who finishes the job.

• New in v1.1.0 - Rootkit hunting and container escape: Crosses the three kernel module lists with the taint flags, judges every library in /etc/ld.so.preload by package provenance, and follows mount and pivot_root - the step that turns a broken namespace into host access. TLS SNI completes "who with" where DNS does not reach (cache, fixed IP, DoH).

• New in v1.1.0 - A single execution path: Agent and server, nothing else. For one machine, install both on the same host. HTTPS is the only transport; the plaintext path was removed from the code rather than switched off.

• New in v1.1.0 - A report built for the tree: The inventory block collapses to give the process tree the screen, columns resize by dragging the divider in the header, and the tree scrolls in both axes with the header staying put. Every detail block is always present, in one of three states: a value, "looked and found nothing", or "not captured by this capture".

• New in v1.0.0 - Answer contract per finding: Every finding declares its confidence (confirmed / probable / heuristic), so a heuristic is never shown as a fact, and its custody (what was preserved of the artifact). The forensic report reads as an investigation: a "how to read" strip (Findings -> Processes -> ATT&CK), a severity legend with the operator action, tooltips on every evidence field, and clickable pivots in both directions between a finding and its ATT&CK technique.

• New in v1.0.0 - Distributed fleet: Pull-model agents forward encrypted captures to a central server (store-and-forward outbox, prioritized ingestion, audited command queue, per-agent capabilities, HTTPS). The Manager shows each command's progress as a live stepper.

• New in v1.0.0 - Runtime and anti-forensic detection: Hidden processes, thread-count divergence, W+X memory, on-disk binary replacement, untrusted libraries, and immutable files in writable directories.

• Forensic Findings: Every collector emits normalized findings on a single severity scale (Info to Critical), each carrying the source that produced it, the MITRE ATT&CK technique, the raw evidence and a recommended action.

• Persistence Enumeration: Answers the first question after a suspected compromise, how would an intruder survive a reboot: systemd units, cron/at jobs, startup and profile scripts, /etc/ld.so.preload, kernel module autoload, udev rules, PAM stacks and per-user authorized_keys. Baseline items stay informational; severity rises only on real indicators such as execution from user-writable paths, world-writable files, hidden names or recent modification.

• Hosts Dashboard (Manager): Monitor multiple infrastructure nodes from a single centralized web interface.

• Forensic Time Machine: Pause live execution and travel back in time to inspect historical snapshots stored in SQLite.

• Kernel-Level Visibility: Uses eBPF kprobes/tracepoints for zero-blindspot monitoring.

• Deep Forensics:

  • Real-time MD5 Hashes: Calculates hashes of executed binaries instantly.
  • Context Awareness: Detects SSH origin IPs, Sudo users, and Tmux sessions.
  • Recursive Alert Bubbling: Child process anomalies (e.g., Unsafe Libs, Net Errors) propagate warnings up to the parent process in the tree view.

• Topology & Infrastructure:

  • Storage Topology: Hierarchical view of Disks -> Partitions -> LVM -> Mount Points with HCTL info.
  • Network Topology: Auto-detection of Gateway, DNS servers, and Interfaces.

• Enterprise Reporting:

  • Generates self-contained, interactive HTML Dashboards.
  • Custom Logo Support: Embeds your organization's logo automatically.
  • Visual Badges: Instant identification of [SSH], [SUDO], [UNSAFE], [NET ERR].
  • Active-state Toolbar: The report toolbar highlights the sort and filter currently applied.

• Dashboard Security (optional):

  • HTTP Basic Authentication: PBKDF2-hashed credentials, working over HTTP and HTTPS.
  • HTTPS with auto self-signed certificate: Zero manual PKI; operator-provided certificates are honored.
  • Both disabled by default (see the "Dashboard Security" section below).

Requirements

• Linux Kernel 4.15+ (5.x+ recommended for BTF support).
• Root privileges (sudo).
• Python 3.6+.
• BCC Tools (python3-bcc).
• iproute2 (for tc command, required only for Chaos Maker).
• Additional Python libs: flask, cryptography, pyyaml.

Installation (PyPI)

Works on any Linux distribution with Python 3.6+.

    pip install sys-inspector

Installation (RPM / openSUSE)

You can install Sys-Inspector directly via zypper using the openSUSE Build Service repository.

1. Add the Repository:

    zypper addrepo https://download.opensuse.org/repositories/home:mariosergiosl:sys-inspector/15.6/home:mariosergiosl:sys-inspector.repo

1. Refresh and Accept GPG Key: During the refresh, you will be asked to trust the repository GPG key.

Fingerprint: 7CF0 5795 053C F397 8E00 948E 9F8D 1AC9 E2BE EABC

    zypper refresh
    # Type 'a' to trust always when prompted.

1. Install the Package:

    zypper install sys-inspector

1. Run: Once installed, the command is available globally:

    sys-inspector

Usage

Sys-Inspector is orchestrated via the main.py entry point (or globally as sys-inspector). It supports multiple execution modes.

The tool has a single path: an agent that collects and a server that receives and renders. For a single machine, install both on the same host.

1. Server (dashboard)

Receives captures from the agents and serves the Manager dashboard (the host list).

    sudo sys-inspector --mode server
    # Access the dashboard at https://localhost:8080
    # TLS is on by default; a self-signed pair is generated on first start,
    # so the browser will warn about the unknown issuer.

2. Agent (collector)

Collects, encrypts, stores and forwards to the server configured in daemon.server_ip.

    sudo sys-inspector --mode daemon

3. One-off run (no server)

A single capture cycle, kept locally. This is the shortest way to try the tool: no keys to exchange, no token, no second process.

    sudo sys-inspector --mode daemon --once

4. Custom Logo

To include your company logo in the report header, simply place a PNG file at the following path:

    /etc/sys-inspector/logo.png

The application will automatically detect, resize (max-height: 40px), encode it to Base64, and embed it in the HTML.

Dashboard Security (Authentication & HTTPS)

Both are optional and disabled by default, so existing deployments are unaffected. Configure them in conf/config.yaml (or /etc/sys-inspector/config.yaml) under the network section.

HTTP Basic Authentication

1. Generate a password hash (run it on the host that serves the dashboard, so the hash matches its werkzeug version):

    python3 tools/gen_password.py

2. Paste the result into config.yaml and enable it:

    network:
      auth:
        enabled: true
        username: "admin"
        password_hash: "pbkdf2:sha256:..."

Authentication works over both HTTP and HTTPS. If enabled without a hash, the server fails closed and rejects all requests.

HTTPS (TLS)

Enable TLS in config.yaml. If the certificate/key below are missing, a self-signed pair is generated automatically on first start (browsers will warn about the unknown issuer, which is expected):

    network:
      tls_enabled: true
      ssl_cert: "/etc/sys-inspector/server_cert.pem"
      ssl_key: "/etc/sys-inspector/server_key.pem"

To use your own PKI, place your certificate and key at the configured paths and they will be used instead of generating one.

See docs/en/dashboard_security.md for details.

Chaos Engineering (Testing Tool)

Included in tools/chaos_maker.sh is a stress testing tool designed to validate the inspector's detection capabilities.

⚠️ WARNING: DO NOT RUN ON PRODUCTION SYSTEMS. This script uses tc (Traffic Control) to purposefully degrade network quality (packet loss/latency) and consumes CPU/Disk resources.

Capabilities

• Network Degradation: Injects 100ms latency and 20% packet loss to trigger [NET ERR] alerts in the report.
• Process Anomalies: Hides processes in /dev/shm to trigger [WARN] alerts.
• Unsafe Library Loading: Forces loading of dynamic libraries from /tmp via a Python script to trigger [UNSAFE] alerts.
• Disk Stress: Generates high I/O throughput to test IO accounting.

How to Run

    sudo ./tools/chaos_maker.sh

To Stop: Press Ctrl+C. The script traps the signal and automatically cleans up the network rules (tc qdisc del) and temporary files.

Project Structure

    ├── conf/                  # Configuration and Cryptographic Keys
    ├── data/                  # SQLite Persistence and Agent IDs
    ├── docs/                  # Narrative documentation (docs/en, docs/pt-BR)
    ├── report/                # Standalone HTML Reports Output
    ├── scripts/               # Development helpers (formatting, venv, test runner)
    ├── src/
    │   ├── collectors/        # eBPF Engine and Process Tree Builders
    │   ├── controllers/       # Execution Modes (Daemon, Web, Snapshot)
    │   ├── core/              # Database and Crypto Logic
    │   ├── exporters/         # HTML and Web Assets
    │   ├── probes/            # C eBPF source code
    │   ├── storage/           # Storage interface and handlers
    │   └── utils/             # Configuration loaders
    ├── tests/                 # Automated test suite (pytest)
    ├── tools/                 # Operational tools (chaos_maker, setup_env, key/password generation)
    └── main.py                # Unified Entry Point

License

Sys-Inspector is free software distributed under the GNU Affero General Public License v3.0 only (AGPL-3.0-only). See LICENSE.md for the full text.

The AGPL was chosen because Sys-Inspector can be operated as a network service (multi-agent server and web dashboard). If you run a modified version and make it available to users over a network, you must offer those users the corresponding source of your modified version.

The license covers the source code only. "Sys-Inspector" and its logo are trademarks and are not licensed with the code; see TRADEMARK.md and NOTICE.