kubemon 2.2.0

No project description provided

kubemon

A tool for distributed container monitoring over Kubernetes.

Translations

• Português

Table of contents

• Environment Requirements
• Application Requirements
• Illustrations
• Main Functionalities
  • Collected Metrics
• Installation
• Running
  • Collector
  • Monitor
  • CLI
• References

Environment requirements

• Ubuntu 18.04
• Kubernetes v1.19
• Docker v19.03.13
• Python 3.8

Application requirements

• psutil
• requests
• docker-py
• virtualenv
• flask

Illustrations

Basic diagram

Main functionalities

• Collects metrics from operating system, Docker containers and processes created by the container
• Send the collected metrics to the collector module, which saves the data in a CSV file
• Can be controlled remotely by a basic CLI

Collected metrics

For more information about the collected metrics, please refer to:

• Operating System Metrics: These metrics are collected from linux /proc filesystem using both psutil Python API and /sys/block/<dev>/stat.
  • CPU: Miscellaneous kernel statistics in /proc/stat
  • Memory: proc - process information pseudo-filesystem: /proc/vmstat
  • Disk : Block layer statistics in /sys/block/<dev>/stat
  • Network: proc - process information pseudo-filesystem: /proc/net/dev
• Docker: These metrics are collected from linux cgroups.
  • CPU: cgroups - Linux control groups: cpu,cpuacct
  • Memory: cgroups - Linux control groups: memory
  • Disk: cgroups - Linux control groups: blkio
  • Network: proc - process information pseudo-filesystem: /proc/net/dev
• Docker Processes: These metrics are collected from linux /proc filesystem using psutil Python API.
  • CPU: Miscellaneous kernel statistics in /proc/stat
  • Memory: proc - process information pseudo-filesystem: /proc/vmstat
  • Disk: Block layer statistics in /sys/block/<dev>/stat
  • Network: proc - process information pseudo-filesystem: /proc/net/dev

Operating System

Type	Unit	Metric
CPU	Quantity Quantity Quantity Quantity Clock Ticks Clock Ticks Clock Ticks Clock Ticks Clock Ticks Clock Ticks Clock Ticks Clock Ticks Clock Ticks	Context Switches Interrupts Soft Interrupts Syscalls Times User Times System Times Nice Times Softirq Times IRQ Times IOWait Times Guest Times Guest Nice Times Idle
Memory	Quantity Quantity Quantity Quantity Quantity KB KB Quantity Quantity Quantity Quantity	Active (Anon) Inactive (Anon) Inactive (file) Active (file) Mapped Pages KB Paged In Since Boot (pgpgin) KB Paged Out Since Boot (pgpgout) Pages Free (pgfree) Page Faults (pgfault) Major Page Faults (pgmajfault) Pages Reused (pgreuse)
Disk	Requests Requests Sectors Milliseconds Requests Requests Sectors Milliseconds Requests Milliseconds Milliseconds Requests Requests Sectors Milliseconds Requests Milliseconds	Read I/O Read I/O Merged with In-queue I/O Read Sectors Total Wait Time for Read Requests Write I/O Write I/O Merged with In-Queue I/O Write Sectors Total Wait Time for Write Requests I/O in Flight Total Time This Block Device Has Been Active Total Wait Time for All Requests Discard I/O Processed Discard I/O Processed with In-Queue I/O Discard Sectors Total Wait Time for Discard Requests Flush I/O Processed Total Wait Time for Flush Requests
Network	Bytes Bytes Packets Packets	Sent Received Sent Received

Docker Processes

Type	Unit	Metric
CPU	Clock Ticks Clock Ticks Clock Ticks Clock Ticks Clock Ticks	User Time System Time Children User Children System IOWait
Memory	Pages Pages Pages Pages Pages Pages Pages	Total Program Size (size) Resident Set Size (resident) Resident Shared Pages (shared) Text (text) Library (lib) Data + Stack (data) Dirty Pages (dt)
Disk	Requests Requests Bytes Bytes Chars Chars	Read Write Read Write Read Write
Network	Bytes Bytes Packets Packets	Sent Received Sent Received

Docker

Type	Unit	Metric
CPU	Clock Ticks Clock Ticks Quantity Quantity Clock Ticks	User System Periods Throttled Throttled Time
Memory	Pages Pages Pages Pages Pages Pages Pages Pages Pages Pages Pages Pages	Resident Set Size (rss) Chached Mapped (mapped_file) Paged In (pgpgin) Paged Out (pgpgout) Page Faults (pgfault) Major Page Faults (pgmajfault) Active (active_anon) Inactive (inactive_anon) Active File (active_file) Inactive File (inactive_file) Unevictable
Disk	Bytes Bytes Bytes Bytes Bytes Bytes	Read Write Sync Async Discard Total
Network	Bytes Bytes Packets Packets	Sent Received Sent Received

Installation

Before installing the tool, make sure Kubernetes and Docker are properly installed in the system. Besides, the tool must be downloaded and extracted in each Kubernetes node to properly monitor the metrics of the cluster.

1. Download the latest version here: kubemon v2.0.0

2. Extract the zip file and go on the extracted directory

3. Create a virtual environment and activate it

   $ python3 -m venv venv 
   $ source venv/bin/activate
   

4. Install packages

   (venv) $ pip install .
   

Running

The collected metrics will be saved in the client machine (See Client in the diagram) by default in /tmp/data. This setting can be changed in ./kubemon/constants.py by changing ROOT_DIR variable value.

Example:

# Before
ROOT_DIR = "/tmp/data"

# After
ROOT_DIR = "/home/user/Documents/data"

In the further subsections will be teaching how to execute this tool.

A brief command list:

usage: kubemon [-h] [-l] [-t TYPE] [-H IP] [-p PORT] [-f FILE1 FILE2] [-c [COMMAND ...]] [-i INTERVAL]

Kubemon commands

optional arguments:
  -h, --help            show this help message and exit
  -l, --list            Lists all available modules
  -t TYPE, --type TYPE  Functionality of sys-monitor. E.g. collector, monitor, merge...
  -H IP, --host IP      Host that any of sys-monitor functions will be connecting
  -p PORT, --port PORT  Port of the host
  -f FILE1 FILE2, --files FILE1 FILE2
                        Files for merge
  -c [COMMAND ...], --command [COMMAND ...]
                        Command for be executing on CollectorClient
  -i INTERVAL, --interval INTERVAL
                        Data collection rate by monitors

Collector

(venv) $ make collector
[ Collector ] Started collector CLI at 0.0.0.0:9880
[ Collector ] Started collector at 0.0.0.0:9822

Monitor

Assuming that the collector IP is 192.168.0.3, let's connect the monitors to it.

There are three types of monitors:

1. OSMonitor - Collects Operating System metrics
2. DockerMonitor - Collects Docker metrics
3. ProcessMonitor - Collects container processes metrics

The tool must be executed as sudo because some metrics (e.g. network, disk, ...) are available only for super users.

In this case, let's run all the monitors at once:

(venv) $ sudo python -m kubemon -t all -H 192.168.0.3
Connected OSMonitor_192_168_0_3_node_0 monitor to collector
...

CLI

Assuming the same IP address for the collector in the section Monitor, the port to communicate with the collector via CLI is 9880.

There are two commands available so far:

1. /instances - Number of monitor instances connected to the collector object.
2. /start <output_dir> - Start the monitors and setting up the output directory to save the files

Checking how many instances are connected

(venv) $ python -m kubemon -t cli -H 192.168.0.3 -c /instances
Connected instances: 5

Starting monitors. In this case the CSV files will be saved at /tmp/data/test00/

(venv) $ python -m kubemon -t cli -H 192.168.0.3 -c /start "test00"
Started 5 monitors

References

• Block layer statistics
• /proc virtual file system
• Evaluation of desktop operating systems under thrashing conditions
• cgroups
• Docker runtime metrics