Ssh Unix System Monitoring
Project description
SshSysMon is a system/server monitoring tool that executes all of its operations over SSH without the need for installing agents across machines.
Its goal is to provide simple self-hosted monitoring and alerting for small numbers of lightweight servers without the traditional overhead of a monitoring system.
It monitors things in /proc and with simple command executions to monitor system vitals such as: memory, cpu load, drive space, swap, etc.
Setup
Installation
Via PyPi
pip install sshsysmon
Manually (No Install)
# Requires python 2.x and pip:
sudo apt-get install -y python python-pip python-dev
# Download the latest SshSysMon:
wget -O - https://github.com/zix99/sshsysmon/archive/master.tar.gz | tar xzv
# Make sure the dependencies are installed:
cd sshsysmon-master/
sudo pip install -r requirements.txt
# Test it out!
./sshmon summary examples/starter.yml
Setting up a ssh key pair
You only need to do this if you are monitoring a remote server.
The best way to connect to remote servers is with private key created and added to the authorized_hosts file on all systems you are interested in monitoring. While password authentication is supported, this is the easiest way to guarantee continued authentication to other hosts.
On debian-based linux systems, setting up a key-pair to use with SSH is easy. I would recommend you make a new linux user to only do monitoring on each machine, but it isn’t required.
# 1. Create a new SSH key if you don't already have one. Follow the prompts, but leave the password blank
ssh-keygen
# 2. Install it on a user on another machine that you want to monitor
ssh-copy-id username@remotehost
Running
The service has two commands, summary and check.
Summary
summary will print out a human-readable summary of all servers specified in the config. It is a great way to validate your config.
It can be executed with:
./sshmon.py summary examples/starter.yml
It also can be told to use various templates. See templating section below. Eg, to use the html template:
./sshmon.py -f html summary examples/starter.yml
Check
check is meant to be executed as part of a scheduled job, and will notify all channels in the config if a condition is unmet.
It can be excuted with:
./sshmon.py check <myconfig.yml>
Running Scheduled Job
The best way to run the service automatically is with a cron job.
Edit your cron jobs with
crontab -e
Add an entry that runs the script every few hours: (or minutes, whatever you like)
0 */4 * * * /path/to/sshmon.py check /path/to/config.yml
Configuration
Configuration is written in yaml and is a set of servers, with a list of monitors with alarms, notification channels and connection details.
See the Examples folder for more sample configs.
An example simple configuration might look something like this:
meta: #Meta section (Optional). Used by summary templates title: "My Cluster Summary" author: "Me" servers: "Name of server": driver: ssh config: host: myhostname.com username: myuser channels: # Notification targets - type: email config: toAddr: myemail@gmail.com subject: "Something went wrong on {server}" monitors: # All alerts and inspectors - type: memory alarms: "Low Swap": "swap_free.mb < 50" "Low Memory": "mem_free.mb < 5" - type: disk alarms: "Low Disk Space": "disk_free.gb < 5" summarize: false # Optional, use if you don't want a monitor to show up in the summary
You can often use YAML’s inheritance to simplify your config for more than 1 server. Each config section also has a corresponding + version to add more in addition to something merged in. eg. monitors+.
All servers are iterated through, and queried for given inspector types. The resulting metrics are compared to the alarms, and if any of them are unmet, a notification it sent to all configured channels.
Data Format
All sizes (that is, number of bytes), is enapsulated by the ByteSize class, which has helper methods for both friendly output, and size casting in the form of b, kb, mb, etc. eg, you can write mem_free.mb > 50.
Percentages will always be presented in their 0-100 form.
Application
Components
The applications is built on three components: Drivers, Inspectors, and Channels.
Each has its corresponding folder with abstract implementation. They are loaded dynamically with their name or path provided in the configuration.
Drivers
Drivers are classes that define how to read information from a server. By default, there are two drivers:
Local
The local driver is only for your local machine. There is no config for this driver.
SSH
The SSH driver is for reaching out to remote machines. There are several config paramters for this driver:
host - The hostname of the machine (IP or Domain)
username - The username to connect with
password - (Not recommended, use key instead) The ssh user’s password
key - The path to the private key to use to connect (Default: ~/.ssh/id_rsa)
port - The port to connect to the machine on (Default: 22)
path - The path which proc is located (Default: /proc)
–
Channels
Channels define what can happen if an alert fires. There a few built-in.
There are a few variables passed in that can be used to format part of the commands:
server - The server that the alert triggered on
alert - The alert that triggered
metric - The metric that triggered the alert
stdout
Writes tab-separated data to stdout. Can be appended to file with bash >> operator.
Arguments:
timeFormat - Either ctime or epoch, the format which time is output. Default: ctime
format - The format string used to write output. Default: {time}\t{server}\t{inspector}\t{alert}
command
Executes a shell command on the machine in which the script is running.
Arguments:
command - The shell command to execute
Sends an email via a SMTP server.
By default, it assumes a local SMTP server is setup. For more complex configs, such as how to use gmail, see the examples.
Arguments:
toAddr - The address to send the email to
fromAddr - The address the email should come from (default: username@hostname)
host - The SMTP host (default: localhost)
port - The SMTP port (default: 25)
subject - Subject line of email (has reasonable default)
username - Username to authenticate with smtp server (default: none)
password - Password to authenticate with smtp server (default: none)
tls - Should use tls (default: false)
ssl - Should use ssl (default: false)
–
Inspectors (Alert Types)
Inspects are parsers that know how to read data from a driver and make sense of it.
Memory (memory)
The memory driver returns metrics about the systems memory:
Metrics: mem_total, mem_free, cached, swap_total, swap_free
Disk Space (disk)
The Disk driver returns status of the disk space (in GB)
Config:
device - The name of the device (Optional, eg /dev/sda)
mount - The mount point of the device (default: /)
Metrics: size, used, available, percent_full
Load Average (loadavg)
The load average inspector returns the system’s current 1/5/15 minute load average.
Metrics: load_1m, load_5m, load_15m
Process Monitor (process)
This inspector will allow you monitor a process on the given machine.
It takes in one required config name. This will use wildcard matching with * and ?.
Metrics: user, pid, cpu, mem, tty
TCP (tcp)
The TCP inspector will try to establish a connection on a given port with the same remote as the driver. It’s important to note that this does not go over SSH, and will not verify anything more than that the port is willing to establish a connection.
Config:
ports: A list, single port, or CSV of ports to check
Metrics:
A dictionary of the requested ports, prefixed with port_, and true if they are open, otherwise false (eg port_22)
A special all metric which will be true if all ports are open
HTTP (http)
The Http connector will attempt to do a GET request on a http/https endpoint, and return the data if able.
Config:
path: The path to request on (default ‘/’)
port: The port to request at (default 80 for http, 443 for https)
https: True/false if https (default: http)
json: true/false if it should attempt to parse the response as json (Default: false)
match: A regex to match against (default: None)
Metrics:
success: A true/false whether the request returns a 2xx, and all requirements were met (matches, or parses)
match: Whether or not the regex matched. None if no match requested
json: The parsed json, if requested
url: The requested url
Custom Command (exec)
exec runs a custom command and returns stdout, stderr, and status (returncode).
Config:
command: The shell command to execute
Metrics:
stdout: The out string of the command
stderr: The err string of the command
status: The returncode of the command (0 means normal)
Templating
SshSysMon uses handlebars to template its summary output. See the templating for more information.
Writing Your Own Component
To learn how to write a specific type of component, visit its readme in the appropriate subfolder.
All components must define def create(args): as a well-known method to instantiate the class. args will be the configuration dict given in the configuration.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.