healthchecker 1.1.0

Check sites health and publish results in a file in Github, filesystem or an API

HealthChecker

A very simple python script to check the health of a service: make a GET request to it and if it answers it on time, then it's alive.
Additionally, it can write the result of the operation in a git repo hosted on Github or in a local file. Or notify dead services through a POST to some JSON API.

Actions are logged using Python's standard logger and displayed according to the log level set.

This app was thought mainly for our status page. It's currently being maintained in our repository.

Requirements

• Python 3.7+
• PyGithub 1.43+
• requests 2.21+

Usage

usage: healthchecker [-h] [--gh-repo GH_REPO] [--gh-filename GH_FILENAME]
                     [--gh-branch GH_BRANCH] [--gh-token GH_TOKEN]
                     [--gh-email GH_EMAIL] [--notify-url NOTIFY_URL]
                     [--notify-payload NOTIFY_PAYLOAD]
                     [--notify-header NOTIFY_HEADER] [--notify-json]
                     [--version] [--validation VALIDATION] [--output OUTPUT]
                     [url [url ...]]

HealthChecker v1.1.0 by HacKan (https://hackan.net) FOSS under GNU GPL v3.0 or
newer. Checks URLs through HTTP GET requests to verify their availability.
Optionally writes the status result to a file in Github. Using Github requires
the repository name, the filename and an API token. Besides the ones listed
below, the following env vars exist: HEALTHCHECKER_LOG_LEVEL sets the minimal
logging level and defaults to info (can be: debug, info, warning, error,
critical); HEALTHCHECKER_REQUESTS_TIMEOUT sets the amount of time in seconds
to wait for services to respond and defaults to 10 seconds (setting a very low
value might cause several false positives). Note: command-line parameters will
always supersede env vars.

positional arguments:
  url                   (HEALTHCHECKER_URLS (comma-separated)) URL to check

optional arguments:
  -h, --help            show this help message and exit
  --version             show version and exit
  --validation VALIDATION
                        (HEALTHCHECKER_URLS_VALIDATION (comma-separated))
                        string to find in the body of a request to an URL as a
                        validation, one per URL (or the last one is used for
                        the remaining URLs) (this parameter can be repeated as
                        needed)
  --output OUTPUT, -o OUTPUT
                        (HEALTHCHECKER_OUTPUT) store result in a file,
                        overwriting if exists (use `-` for standard output)

github options:
  --gh-repo GH_REPO     (HEALTHCHECKER_GITHUB_REPO) repository in the form of
                        <user|org>/<repo> (case insensitive), i.e.:
                        HacKanCuBa/b2rsum
  --gh-filename GH_FILENAME
                        (HEALTHCHECKER_GITHUB_FILENAME) filename to modify
                        (include path if it is in a subdir such as
                        path/to/file.ext)
  --gh-branch GH_BRANCH
                        (HEALTHCHECKER_GITHUB_BRANCH) branch where commits are
                        done (defaults to master)
  --gh-token GH_TOKEN   (HEALTHCHECKER_GITHUB_API_TOKEN) API token or
                        client_id,client_secret (bypasses the one supplied
                        through the environment)
  --gh-email GH_EMAIL   (HEALTHCHECKER_GITHUB_COMMITTER_EMAIL) git committer
                        email (the committer name is hardcoded to
                        HealthChecker)

notify options:
  --notify-url NOTIFY_URL
                        (HEALTHCHECKER_NOTIFY_URL) URL to POST the status
                        notification
  --notify-payload NOTIFY_PAYLOAD
                        (HEALTHCHECKER_NOTIFY_PAYLOAD) payload to send to the
                        notify URL: it is prepended to the comma-separated
                        list of URLs that failed validation, unless that it
                        contains the string HEALTHCHECKER_FAILED_URLS (case
                        sensitive), where it will replace that string by the
                        comma-separated list of URLs, and send the entire
                        payload
  --notify-header NOTIFY_HEADER
                        (HEALTHCHECKER_NOTIFY_HEADERS (comma-separated))
                        header to send to the notify URL, which must be
                        specified as name and value separated by a semicolon:
                        <header name>:<header value> (this parameter can be
                        repeated as needed)
  --notify-json         (HEALTHCHECKER_NOTIFY_JSON (true/false)) send the
                        payload JSON encoded (it also adds the proper Content-
                        Type header)

Environment variables

• HEALTHCHECKER_GITHUB_REPO: repository in the form of <user|org>/ (case insensitive), i.e.: HacKanCuBa/b2rsum.
• HEALTHCHECKER_GITHUB_FILENAME: filename to modify (include path if it is in a subdir such as path/to/file.ext).
• HEALTHCHECKER_GITHUB_BRANCH: branch where commits are done (defaults to master).
• HEALTHCHECKER_GITHUB_API_TOKEN: API token or client_id,client_secret (bypasses the one supplied through the environment).
• HEALTHCHECKER_GITHUB_COMMITTER_EMAIL: git committer email (the committer name is hardcoded to HealthChecker).
• HEALTHCHECKER_URLS: URLs to check, comma-separated.
• HEALTHCHECKER_LOG_LEVEL: minimal logging level, defaults to info (can be: debug, info, warning, error, critical).
• HEALTHCHECKER_REQUESTS_TIMEOUT: amount of time in decimal seconds to wait for services to respond and defaults to 10 seconds (setting a very low value might cause several false positives).
• HEALTHCHECKER_NOTIFY_URL: URL to send failed checks via POST as notification, comma-separated.
• HEALTHCHECKER_URLS_VALIDATION: comma-separated list of validations to run on given URLs.
• HEALTHCHECKER_NOTIFY_PAYLOAD: payload to send to the notify URL: it is prepended to the comma-separated list of URLs that failed validation, unless that it contains the string HEALTHCHECKER_FAILED_URLS (case sensitive), where it will replace that string by the comma-separated list of URLs, and send the entire payload.
  • Example 1: HEALTHCHECKER_NOTIFY_PAYLOAD=here comes the failed urls...
  • Example 2: HEALTHCHECKER_NOTIFY_PAYLOAD={"data": "HEALTHCHECKER_FAILED_URLS"}
• HEALTHCHECKER_NOTIFY_HEADERS: headers to send to the notify URL, which must be specified as name and value separated by a semicolon: header name:header value, and successive headers separated by comma.
  • Example 1: HEALTHCHECKER_NOTIFY_HEADERS=X-Auth:4c18a291d7d8e7946cb9db9cbb3e1f49
  • Example 2: HEALTHCHECKER_NOTIFY_HEADERS=Content-Type:application/json,X-MyVal:1
• HEALTHCHECKER_NOTIFY_JSON: (true/false) send the payload JSON encoded (it also adds the proper Content-Type header).
• HEALTHCHECKER_OUTPUT: store result in a file (use - for standard output).

Responses

HealthChecker reports two values for each service checked:

• alive: means that the service has responded (no timeout and status code lower than 500).
• ok: means that the service has responded and has a status code of 2xx; additionally, if checks are given for the service this means that checks passed.

The app will follow any 3xx response accordingly.

Examples

Simply print checks result:

:~$ healthchecker https://rlab.be adm.rlab.be
INFO 2019-04-09 01:06:37 Begin checking URL https://rlab.be...
INFO 2019-04-09 01:06:37 Begin checking URL http://adm.rlab.be...
INFO 2019-04-09 01:06:37 Finish checking URL https://rlab.be: alive and OK
INFO 2019-04-09 01:06:37 Finish checking URL http://adm.rlab.be: alive and OK

Or show output through stdout, hiding logs:

:~$ HEALTHCHECKER_LOG_LEVEL=critical healthchecker -o - https://rlab.be adm.rlab.be 2> /dev/null
[
  {
    "uid": "416ff55a",
    "url": "https://rlab.be",
    "alive": true,
    "ok": true
  },
  {
    "uid": "5d961608",
    "url": "http://adm.rlab.be",
    "alive": true,
    "ok": true
  }
]

That output can be parsed by jq or any other JSON tool! Additionally, it can be saved to a file for later reference:

:~$ healthchecker -o checks.json https://rlab.be adm.rlab.be
INFO 2019-04-09 01:06:37 Begin checking URL https://rlab.be...
INFO 2019-04-09 01:06:37 Begin checking URL http://adm.rlab.be...
INFO 2019-04-09 01:06:37 Finish checking URL https://rlab.be: alive and OK
INFO 2019-04-09 01:06:37 Finish checking URL http://adm.rlab.be: alive and OK
INFO 2019-04-09 01:06:37 Result stored as "/home/hackan/Workspace/healthchecker/checks.json"

:~$ cat checks.json
[
  {
    "uid": "416ff55a",
    "url": "https://rlab.be",
    "alive": true,
    "ok": true
  },
  {
    "uid": "5d961608",
    "url": "http://adm.rlab.be",
    "alive": true,
    "ok": true
  }
]

Notify failed services to an endpoint: healthchecker --notify-url https://eoc.rlab.be/api/v1/status/ https://rlab.be http://wiki.rlab.be

Write to a file in Github and be very verbose:

:~$ HEALTHCHECKER_LOG_LEVEL=debug healthchecker --gh-repo rlyehlab/sysadmins --gh-filename data/healthcheck.json --gh-token ab410...2cc https://git.rlab.be
INFO 2019-04-09 01:07:32 Begin checking URL https://git.rlab.be...
ERROR 2019-04-09 01:07:32 Error GETing data from/to https://git.rlab.be: ConnectionError(MaxRetryError("HTTPConnectionPool(host='git.rlab.be', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x6ac59ffbd001>: Failed to establish a new connection: [Errno -2] Name or service not known'))"))
DEBUG 2019-04-09 01:07:32 Request to https://git.rlab.be took 0.18 seconds
INFO 2019-04-09 01:07:32 Finish checking URL http://git.rlab.be: dead
INFO 2019-04-09 01:07:33 Getting repository information...
INFO 2019-04-09 01:07:38 File data/healthcheck.json updated: 0335a8088f5aff42f078a9396916c8adbcc1a6c3

Parameters can be passed through env vars and/or through command-line indistinctly (command-line parameters will always supersede env vars):

:~$ HEALTHCHECKER_URLS_VALIDATION="Services | Administration,Adventurous writings by R'lyeh Sysadmins" healthchecker --notify-url 127.0.0.1:8000 https://adm.rlab.be https://blog.adm.rlab.be
INFO 2019-04-09 00:59:44 Begin checking URL https://adm.rlab.be...
INFO 2019-04-09 00:59:44 Begin checking URL https://blog.adm.rlab.be...
INFO 2019-04-09 00:59:44 Finish checking URL https://blog.adm.rlab.be: alive and OK
INFO 2019-04-09 00:59:44 Finish checking URL https://adm.rlab.be: alive and OK

All checks were OK, but should one fail...:

HEALTHCHECKER_LOG_LEVEL=debug HEALTHCHECKER_URLS_VALIDATION="Services | Administration,Adventurous writings by R'lyeh Sysadmins" python -m healthchecker --validation "non-existent string" --notify-url 127.0.0.1:8000 https://adm.rlab.be https://blog.adm.rlab.be
INFO 2019-04-09 01:03:39 Begin checking URL https://adm.rlab.be...
INFO 2019-04-09 01:03:39 Begin checking URL https://blog.adm.rlab.be...
DEBUG 2019-04-09 01:03:40 Request to https://blog.adm.rlab.be took 0.18 seconds
INFO 2019-04-09 01:03:40 Finish checking URL https://blog.adm.rlab.be: alive but not OK
DEBUG 2019-04-09 01:03:40 Request to https://adm.rlab.be took 0.30 seconds
INFO 2019-04-09 01:03:40 Finish checking URL https://adm.rlab.be: alive but not OK
DEBUG 2019-04-09 01:03:40 Notifying http://127.0.0.1:8000 with headers: {} and payload: https://adm.rlab.be,https://blog.adm.rlab.be
ERROR 2019-04-09 01:03:40 Error POSTing data from/to http://127.0.0.1:8000: ConnectionError(MaxRetryError("HTTPConnectionPool(host='127.0.0.1', port=8000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f49d9767f28>: Failed to establish a new connection: [Errno 111] Connection refused'))"))
DEBUG 2019-04-09 01:03:40 Request to http://127.0.0.1:8000 took 0.00 seconds
ERROR 2019-04-09 01:03:40 Could not notify http://127.0.0.1:8000

Note that nobody was listening at 127.0.0.1:8000 so the notification failed as well.

Run

Can be run as a python module python -m healthchecker ... or directly as a CLI tool healthchecker .... Read below on how to run the docker image.

PyPi

HealthChecker is in PyPi and can be installed with any standard tool such as pip (pip install healthchecker) or poetry (poetry add healthchecker). To add it as a dependency in your project it's recommended to use the hash parameter for pip. Hashes (and more) are listed in every release.

Repo

Clone the repo and install requirements:

• installing requirements with poetry (recommended): poetry install. Then start virtualenv with poetry shell.

Docker

Build

Build locally with docker build --compress --pull --rm --tag registry.rlab.be/sysadmins/healthchecker:latest . or invoke build

Pull

You can pull from our registry with docker pull registry.rlab.be/sysadmins/healthchecker:latest

Run

Run with docker run --rm registry.rlab.be/sysadmins/healthchecker:latest ...

Alternatively, use env vars by creating an env file and passing it to docker:

cp sample.env .env
vim .env  # edit and populate vars
docker run --rm --env-file .env registry.rlab.be/sysadmins/healthchecker:latest

Deploy

This can be deployed in a server creating a SystemD service and timer (the image will be pulled by Docker on first run):

1. Service

Create an env file where ever you want, as in /srv/healthchecker/.env (protect access to it with linux permissions). It can be anywhere with any name, just point it in the parameter --env-file in the service file. Then create the service file /etc/systemd/system/healthchecker.service:

[Unit]
Description=HealthChecker Service
Requires=docker.service
After=network.target docker.service

[Service]
Type=simple
ExecStart=/usr/bin/docker run --rm --env-file /srv/healthchecker/.env registry.rlab.be/sysadmins/healthchecker:latest
User=root
Group=docker

Alternatively, you can skip env file usage and write every parameter in the ExecStart line, but writing the API token there means it will be visible in the process list which is usually not a good idea.

2. Timer:

Create the timer file /etc/systemd/system/healthchecker.timer (use the same name as the service but with the .timer extension):

[Unit]
Description=HealthChecker Service Timer

[Timer]
OnBootSec=600
OnUnitActiveSec=5m

[Install]
WantedBy=multi-user.target

This sample is set to run the service 5 minutes after boot and then every 5 minutes. Read the documentation if you need to set different parameters.

3. Reload SystemD services: systemctl daemon-reload
4. Enable the timer: systemctl enable healthchecker.timer
5. Start it: systemctl start healthchecker.timer

Tip: you can see the execution log with journalctl -u healthchecker (or the service name used). Check journalctl help for additional filtering options.

Thanks @snkisuke for your help with this section.

Developing and PRing

For more information and use cases, refer to DEVELOPERS.md.

Collaborators

Many thanks to those that collaborate with this project (in alphabetical order): @erus, @seykron, @snkisuke.

License

HealthChecker is made by HacKan under GNU GPL v3.0+. You are free to use, share, modify and share modifications under the terms of that license.

Copyright (C) 2019 HacKan (https://hackan.net)

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.