denet 0.6.0

A streaming process monitoring tool

denet: a streaming process monitor

denet /de.net/ v. 1. Turkish: to monitor, to supervise, to audit. 2. to track metrics of a running process.

Denet is a streaming process monitoring tool that provides detailed metrics on running processes, including CPU, memory, I/O, and thread usage. Built with Rust, with Python bindings.

Features

• Lightweight, cross-platform process monitoring
• Adaptive sampling intervals that automatically adjust based on runtime
• Memory usage tracking (RSS, VMS)
• CPU usage monitoring with accurate multi-core support
• I/O bytes read/written tracking
• Thread count monitoring
• GPU monitoring with NVIDIA NVML support (optional) — see docs/gpu.md
• eBPF-based profiling on Linux: off-CPU time and syscall tracking (optional)
• Recursive child process tracking
• Command-line interface with colorized output
• In-memory sample collection for Python API — see docs/python-api.md
• Analysis utilities for metrics aggregation, peak detection, and resource utilization
• Process metadata preserved in output files (pid, command, executable path)

Installation

pip install denet    # Python package
cargo install denet  # Rust binary

# For GPU monitoring support (requires NVIDIA drivers and CUDA)
pip install denet[gpu]  # Python package with GPU support
cargo install denet --features gpu  # Rust binary with GPU support

# For eBPF profiling support (Linux only, requires clang)
cargo install denet --features ebpf

See docs/gpu.md for GPU monitoring details and docs/dev.md for development setup requirements.

Usage

Understanding CPU Utilization

CPU usage is reported in a top-compatible format where 100% represents one fully utilized CPU core:

• 100% = one core fully utilized
• 400% = four cores fully utilized
• Child processes are tracked separately and aggregated for total resource usage
• Process trees are monitored by default, tracking all child processes spawned by the main process

This is consistent with standard tools like top and htop. For example, a process using 3 CPU cores at full capacity will show 300% CPU usage, regardless of how many cores your system has.

Understanding Network Metrics (sys_net_rx_bytes / sys_net_tx_bytes)

Network I/O is reported as namespace-wide totals, not per-process. The values come from /proc/<pid>/net/dev when it exists, falling back to /proc/net/dev. This means:

• Containers (Docker, Podman, etc.): each container gets its own network namespace, so sys_net_rx_bytes/sys_net_tx_bytes reflect only traffic on that container's interfaces. Values are accurate for the monitored workload.
• Conda environments / venvs / bare processes: these share the host network namespace. The numbers reflect all traffic on the host's interfaces, not just the monitored process. The sys_net_ prefix signals this: it is a system-level counter scoped to the network namespace, not a process-level counter.

If you need per-process socket-level attribution, eBPF (enabled with --features ebpf --enable-ebpf) tracks individual syscalls and can give more precise network activity signals via the syscall intensity metrics.

Understanding Disk I/O Metrics

denet reports three signals for disk activity on Linux: block-layer bytes (disk_read_bytes/disk_write_bytes), syscall bytes (syscall_read_bytes/syscall_write_bytes), and page-fault counts (page_faults_cached/page_faults_disk). They answer different questions — most importantly, disk_read_bytes shows 0 for cached reads and mmap access, which surprises users. See docs/disk-io.md for how to interpret and combine them.

Command-Line Interface

# Basic monitoring with colored output
denet run sleep 5

# Output as JSON (actually JSONL format with metadata on first line)
denet --json run sleep 5 > metrics.json

# Write output to a file
denet --out metrics.log run sleep 5

# Custom sampling interval (in milliseconds)
denet --interval 500 run sleep 5

# Specify max sampling interval for adaptive mode
denet --max-interval 2000 run sleep 5

# Monitor existing process by PID
denet attach 1234

# Monitor just for 10 seconds
denet --duration 10 attach 1234

# Quiet mode (suppress process output)
denet --quiet --json --out metrics.jsonl run python script.py

# Monitor a CPU-intensive workload (shows aggregated metrics for all children)
denet run python cpu_intensive_script.py

# Monitor a GPU workload (requires --features gpu or denet[gpu])
denet run python gpu_training_script.py

# Enable eBPF profiling — off-CPU time and syscall tracking (Linux only, requires root or CAP_BPF)
sudo denet --enable-ebpf run python io_bound_script.py

# Disable child process monitoring (only track the parent process)
denet --no-include-children run python multi_process_script.py

Python API

See docs/python-api.md for the full Python API reference, including ProcessMonitor, execute_with_monitoring, and analysis utilities.

Adaptive Sampling

Denet uses an intelligent adaptive sampling strategy to balance detail and efficiency:

1. First second: Samples at the base interval rate (fast sampling for short processes)
2. 1-10 seconds: Gradually increases from base to max interval
3. After 10 seconds: Uses the maximum interval rate

This approach ensures high-resolution data for short-lived processes while reducing overhead for long-running ones.

GPU Monitoring

Denet supports NVIDIA GPU monitoring via NVML when built with the gpu feature. See docs/gpu.md for features, requirements, usage examples, and the GPU data structure.

eBPF Profiling

Denet provides optional eBPF-based profiling on Linux for deeper insight into what processes are doing when they're not running on a CPU.

Features

• Off-CPU profiling: Captures every sched_switch event to measure how long threads are blocked — waiting for I/O, locks, or sleep. Useful for diagnosing latency in I/O-bound workloads.
• Syscall tracking: Counts syscall frequency by category (file I/O, memory, network, …) and classifies process behaviour (I/O-bound, CPU-bound, etc.).

Requirements

• Linux kernel 5.5+
• clang available at build time
• CAP_BPF + CAP_PERFMON capabilities, or root at runtime

Build

cargo build --features ebpf

Usage

# Monitor an I/O-bound workload
sudo denet --enable-ebpf run python io_bound_script.py

# With JSON output
sudo denet --enable-ebpf --json run sleep 5

# Set capabilities on the binary to avoid running as root every time.
# cap_dac_read_search is needed to read /sys/kernel/tracing/events/*/id
# (mode 0400, root-owned) when attaching syscall tracepoints.
sudo setcap cap_bpf,cap_perfmon,cap_dac_read_search=ep ./target/debug/denet
denet --enable-ebpf run sleep 5

Sample JSON output

{
  "ts_ms": 1714000000000,
  "cpu_usage": 12.5,
  "mem_rss_kb": 8192,
  "ebpf": {
    "offcpu": {
      "total_time_ns": 1500000000,
      "total_events": 30,
      "avg_time_ns": 50000000,
      "max_time_ns": 500000000,
      "top_blocking_threads": [
        { "pid": 1234, "tid": 1234, "time_ms": 500.0, "percentage": 33.33 }
      ]
    },
    "syscalls": {
      "total": 1500,
      "by_category": { "file_io": 900, "memory": 300, "time": 200, "other": 100 },
      "top_syscalls": [
        { "name": "read", "count": 450 },
        { "name": "write", "count": 350 }
      ]
    }
  }
}

Notes on stack traces

Stack symbolication uses /proc/{pid}/maps and addr2line. For best results:

• Build monitored programs with debug symbols (-g)
• JIT-compiled languages (Python, Java, Node.js) produce limited stack information
• See docs/offcpu.md for troubleshooting and architecture details

Development

For detailed developer documentation — including development requirements, project structure, workflow, testing, and release process — see docs/dev.md.

License

GPL-3

Acknowledgements

• sysinfo - Rust library for system information
• PyO3 - Rust bindings for Python