Caduceus

What is Caduceus?

Caduceus is that long stick with the intertwined snakes that Hermes used to carry around. It is also a service that will notify you if your scheduled tasks/cronjobs fail to run.

Motivation

You know how you set all these cronjobs to run, and added fancy error reporting and things, only to realize too late that this doesn't help you at all when the server has been down for a month and nobody noticed? Caduceus won't let this happen again.

Rather than trigger on failure, Caduceus triggers on absence of success. Services have to actively check in (by visiting a URL), and, if they don't, Caduceus notifies you by email that the task has failed. If the service starts working again, Caduceus will notify you of that as well.

Installation

To install Caduceus, you can just get it from PyPI:

pip install caduceus

Alternatively, you can pull the Docker image:

docker pull registry.gitlab.com/stavros/caduceus:latest

Usage

To run Caduceus, you need to configure it. This is done by placing a file called caduceus.toml in the directory you want to run Caduceus in. That directory is where the Caduceus SQLite database will be created.

If you installed Caduceus from the repo or with pip, just run it:

caduceus

It will load the configuration from the file, create its database and start running on http://localhost:5000/.

To run it via Docker:

docker run -v (pwd):/caduceus registry.gitlab.com/stavros/caduceus:latest

Configuration

Here's a sample configuration file (which is also available as caduceus.toml.example in the repository):

[config]
# An optional secret key to use for checking in.
secret_key = "somelongkey"

[alerting.email]
# Where you want the notification emails sent if services don't check in.
recipient_emails = [ "notifyme@example.com", "otherdev@example.com" ]

# SMTP server configuration, for sending email.
from_addr = "caduceus@example.com"
hostname = "example.com"
port = 25
username = "myuser"
password = "mypassword"
encryption = "none"  # Can also be "ssl" or "starttls".

[alerting.telegram]
apikey = "#############:####################"
chat_id = "99999999"

[alerts]
# How alerts will be sent by default.
default_channels = [ "telegram" ]

# Your alerts go here.
# An alert needs a short name (here, `cron`), and an interval it needs to check in by.
[alerts.cron]
every = "1h"
# You can override the alerting channels per-alert.
channels = [ "email" ]

[alerts.backups]
every = "1d"
channels = [ "email", "telegram" ]

# For alerts that use email, you can also override the recipient emails.
recipient_emails = [ "thirdemail@example.com" ]

[alerts.alwaysfail]
every = "1s"
# You can tell Caduceus to only notify every minute, instead of every second,
# to prevent spam.
notify_every = "1m"

The above config defines three services, raidscrub, backups and alwaysfail. raidscrub needs to check in every hour, backups needs to check in every day, and alwaysfail needs to check in every second. That's why it was called that.

However, as emailing you every second would get spammy, notify_every is set to one minute, so Caduceus will only email you once a minute, even though the alert will be considered failed if it doesn't check in once per second. You will get an initial email right when the failure is detected (there is a 10 second notification resolution) and then emails every minute after that.

Always leave a bit of leeway in your tasks, to account for running time. If a task starts at midnight one day and runs for an hour, it'll check in at 1am. If the next day it runs for 61 minutes, it will check in more than a day later, so you'll get a "failed" email. To avoid that, add an buffer of 10% or so to your alerts.

Checking in

Checking in is done by retrieving a URL on the server. The URL for checking in and resetting the alert timer is /reset/<alert name>/. For example, to check in to backups if you haven't specified a secret_key (and if Caduceus is running on example.com), you'd simply do:

curl http://example.com/reset/backups/

If you did specify a secret key, just include it:

curl http://example.com/reset/backups/?key=<your secret_key>

If your alert is set up for, say, one hour, and your task does not check in, you will get an email one hour after its last checkin, saying "your task has not checked in". If it still doesn't check in, you'll get another email an hour after that, then an hour after that, etc, until it does, at which point you'll get an email saying that the job is now fine.