Health Check Wrap

@readwithai - X - blog - machine-aided reading - 📖⚡️🖋️

Wrap a daemon in a periodic health check. Terminate the process and exit if the health check fails allowing your service manager like systemd to restart your process.

Motivation

I am using a service manager (specifically circusd) to manage some processes. One of these processes is intermittently breaking but the error is rare and in code that I did not write myself. I therefore want to periodically run a health check to see if my service is working and then restart the process if it is not.

However, I dislike the code of my application communicating with my service manager. It feels like it breaks a layer of abstraction, does not allow me to switch out the service manager, and in theory allows one part of the system to restart all processes. Instead, I want to combine the health check with command to run the service and restart the service when the health check dies.

Usage

To monitor 'service' and terminate it and exit when 'check1' completes with a non-zero exit code you can use the following command:

health-check-wrap --service service --health-check check1

This runs a health check every minute. Normally, you would run this command from a service manager such as systemd, supervisor or circusd which would then restart health-check-wrap and so the service.

You can use quotes to specify a full command with arguments to run:

You can run multiple health check commands using repeated --health-check arguments.

You can use the --interval argument to change how often a command runs.

Caveats

This is very much written with the single-server set up in mind. This is useful for home servers, single board (raspberry pi) projects, and any project that does not yet have users paying you money. Other tools provide tools for distributing services across machines together with health check functionality such as kubernetes but these are too heavy weight for many projects.

Alternatives and prior work

It may be better to merely fix the bug. This sort of layering can add complexity and it can be far simpler to have something work consistently rather than try to fix the results of it not working.

The general approach of "die if something breaks" is influenced by Erlang's concept of a supervision tree.

Related tools

I produce a stream of command-line tools like this which you may be interested in.

Most similar to this is killable sudo which allows you to run processes as another user, such as root, with sudo but kill the process with SIGTERM. Many service managers kill processes with SIGTERM.

env subset is a tool to help you control the environment uses to run a process. This is useful for debugging.

About me

I am @readwithai. I create tools for reading, research and agency sometimes using the markdown editor Obsidian.

I also create a stream of tools that are related to carrying out my work.

I write about lots of things - including tools like this - on X. My blog is more about reading and research and agency.