notificationforwarder

In this framework, two aspects are in the focus. How to transport a notification to the recipient system and in which format. In the beginning, Naemon or one of the other monitoring cores will execute a command line. The actual script and the individual command line parameters are defined in a command definition. Typical parameters are (i use the notation of Nagios macros) HOSTNAME, SERVICEDESC, SERVICESTATE, SERVICEOUTPUT. These snippets need to be put together to some kind of payload suitable for the receiving system. And then this payload must be transported to it. We call the two components formatter and forwarder. The formatter takes the raw input data and creates a payload and the forwarder transmits the payload to the destination. What the framework does for you behind the scenes: When forwarding to a recipient fails, the event is saved in a local sqlite database for a certain time and re-sent when the script is called next time and the recipient is available again. Logging of successful and of course failed deliveries is also done automatically.
Logging uses the framework's native setup with the standard logfile naming and format used by the built-in tools. There is also a component reporter which will rarely be used. It's purpose is to run additional code after a successful or failed delivery.

Let me list some of the formatter/forwarder combinations which are usually found in enterprise environments:

|markdown-like Nostr note|Nostr relays |

Of course json is not json, the attributes and values are different depending on the recipient.

For every notification recipient you need such a pair, practically it means, you have to write two python files. Imagine you have a command definition like this:

define command{
    command_name    notify-service-victorops
    command_line    $USER1$/notificationforwarder \
                        --forwarder myspecialreceiver \
                        --forwarderopt company_id='$_CONTACTCOMPANY_ID$' \
                        --forwarderopt company_key='$_CONTACTCOMPANY_KEY$' \
                        --forwarderopt routing_key='$_CONTACTROUTING_KEY$' \
...
                        --eventopt HOSTNAME='$HOSTNAME$' \
                        --eventopt HOSTSTATE='$HOSTSTATE$' \
                        --eventopt HOSTADDRESS='$HOSTADDRESS$' \
                        --eventopt SERVICEDESC='$SERVICEDESC$' \
                        --eventopt SERVICESTATE='$SERVICESTATE$' \
                        --eventopt SERVICEOUTPUT='$SERVICEOUTPUT$' \
                        --eventopt LONGSERVICEOUTPUT='$LONGSERVICEOUTPUT$' \
                    >> $USER4$/var/log/notificationforwarder_errors.log 2>&1
}

Your service notifications should be sent to some ticket tool. The notification script will talk to a REST api and upload a a well-formatted Json payload. Therefore the notifcation framework has two jobs. First, take the event attributes (all the --eventopt arguments) and transform them to a Json structure. Then, upload it with a POST request.

In your OMD site you create a folder ~/local/lib/python/notificationforwarder/myspecialreceiver and add two files, formatter.py and forwarder.py. A skeleton for the formatter.py looks like this:

from notificationforwarder.baseclass import NotificationFormatter

class MyspecialreceiverFormatter(NotificationFormatter):

    def format_event(self, event):
        json_payload = {}
        # fill the payload with whatever is required
        json_payload['hostname'] = event.eventopts['HOSTNAME']
        json_payload['remark'] = "here is a ticket for you, haha"
       
        event.payload = json_payload
        event.summary = "this is a one-line summary which will be used to write a log"
        if event.eventopts['NOTIFICATIONTYPE'] == "DOWNTIMEEND":
            event.discard()
        elif event.eventopts['NOTIFICATIONTYPE'] == "DOWNTIMECANCELLED":
            event.discard(silently=False)

The class name is, by default, derived from the argument provided to the --forwarder parameter with the first letter capitalized, followed by "Formatter". Alternatively, the parameter --formatter can be used. The formatter class must have a method format_event. This method is called with an event object, which has an attribute event.eventopts. This is a dictionary consisting of keys and values extracted from the --eventopt parameters of the \$USER1\$/notificationforwarder command. The method should then set the attributes payload and summary of the event object.
The formatter also has the ability to decide not forwarded an event at all. If if chooses to halt the processing of an event, it can invoke the method discard(). The event will then simply be dropped without any trace. Invoking discard with the parameter silently=False will log a discard message in the log file. If event.summary has not yet been created, a dump of the raw event will be written.

A skeleton for the forwarder.py looks like this:

import requests
from notificationforwarder.baseclass import NotificationForwarder, NotificationFormatter, timeout

class MyspecialreceiverForwarder(NotificationForwarder):
    def __init__(self, opts):
        super(self.__class__, self).__init__(opts)
        self.url = "https://alert.someapi.com/v1/tickets/"+self.company_id+"/alert/"+self.company_key+"/"+self.routing_key

    @timeout(30)
    def submit(self, event):
        try:
            logger.info("submit "+event.summary)
            response = requests.post(self.url, json=event.payload)
            if response.status_code != 200:
                logger.critical("POST returned "+str(response.status_code)+" "+response.text)
                return False
            else:
                logger.debug("POST returned "+str(response.status_code)+" "+response.text)
                return True
        except Exception as e:
            logger.critical("POST had an exception: {}".format(str(e)))
            return False

    def probe(self):
        r = requests.head(self.url)
        return r.status_code == 200

Again, the class name has to be the argument of the --forwarder parameter with the first letter in upper case, but this time with "Forwarder" appended. This class must have a method submit(), which gets the event object which was supplied with payload and summary in the formatting step. If submit() returns a False value, the framework will spool the event in a database. The next time Naemon is executing the notificationforwarder script for this receiver, it will try to submit the events which have been spooled so far. If the Forwarder class has an optional method probe(), it will first check if the receiver is now up again before it flushes the spooled events with the submit() method.

There are two special forwarderopt parameters.

• --forwarderopt logfile_backups=n
  Besides the default logfile var/log/notificationforwarder_... there will be n rotated archive files. (Rotation takes place after the logfile reaches 20MB, default is 3 archives)
• --forwarderopt max_spool_minutes=n
  When submitting an event fails, it will be spooled for n minutes. Within this time re-submitting will be attempted on every call to notificationforwarder. (Default is 5)

Forwarders/Formatters which come with the module

WebhookForwarder

This is a generic class, which is used to upload random json payloads (that's why there is no WebhookFormatter as there are so many possibilities) with a POST request to an Api. The parameters it takes are url, username and password for basic auth, headers to add to the post request. The latter can be used for token based authentication.

First the fowarder will make a plain, unauthorized post request.

    command_line    $USER1$/notificationforwarder \
                        --forwarder webhook \
                        --forwarderopt url=https://cm.consol.de/api/v2/crticket \
                        --eventopt HOSTNAME='$HOSTNAME$' \

Second, the same but with basic auth.

    command_line    $USER1$/notificationforwarder \
                        --forwarder webhook \
                        --forwarderopt url=https://cm.consol.de/api/v2/crticket \
                        --forwarderopt username=lausser \
                        --forwarderopt username=consol123 \
                        --eventopt HOSTNAME='$HOSTNAME$' \

And this one shows how to set additional headers.

    command_line    $USER1$/notificationforwarder \
                        --forwarder webhook \
                        --forwarderopt url=https://cm.consol.de/api/v2/crticket \
                        --forwarderopt headers='{"Authentication": "Bearer 0x00hex0der8ase64schlonz", "Max-Livetime": "10"}' \
                        --eventopt HOSTNAME='$HOSTNAME$' \

What's missing here is --formatter myownpayload, where you call a formatter specifically written for the payload format your api wants.

Demo setup

Let's configure sending notification to a public REST Api, where you can watch the incoming event live. First, open https://webhook.site in your browser and copy the random url you are presented. You need it in the argument url= in the following commands. If you don't care if anybody can see your events, then just use the one from the command definitions.

define command {
  command_name    notify-service-webhooksite
  command_line    $USER1$/notificationforwarder \
                     --forwarder webhook \
                     --forwarderopt url=https://webhook.site/3864baed-d861-4e33-a5d6-3d9104d696d2 \
                     --formatter vong \
                     --eventopt HOSTNAME='$HOSTNAME$' \
                     --eventopt HOSTSTATE='$HOSTSTATE$' \
                     --eventopt HOSTADDRESS='$HOSTADDRESS$' \
                     --eventopt SERVICEDESC='$SERVICEDESC$' \
                     --eventopt SERVICESTATE='$SERVICESTATE$' \
                     --eventopt SERVICEOUTPUT='$SERVICEOUTPUT$' \
                     --eventopt LONGSERVICEOUTPUT='$LONGSERVICEOUTPUT$' \
                     >> $USER4$/var/log/notificationforwarder_errors.log 2>&1
}

define command {
  command_name    notify-host-webhooksite
  command_line    $USER1$/notificationforwarder \
                     --forwarder webhook \
                     --forwarderopt url=https://webhook.site/3864baed-d861-4e33-a5d6-3d9104d696d2 \
                     --formatter vong \
                     --eventopt HOSTNAME='$HOSTNAME$' \
                     --eventopt HOSTSTATE='$HOSTSTATE$' \
                     --eventopt HOSTADDRESS='$HOSTADDRESS$' \
                     --eventopt HOSTOUTPUT='$HOSTOUTPUT$' \
                     >> $USER4$/var/log/notificationforwarder_errors.log 2>&1
}

The forwarder webhook is already builtin, we only need to write the formatter in ~/local/lib/python/notificationforwarder/vong/formatter.py

from notificationforwarder.baseclass import NotificationFormatter

class VongFormatter(NotificationFormatter):

    def format_event(self, event):
        json_payload = {
            'greeting': 'Halo i bims 1 eveng vong Naemon her',
            'host_name': event.eventopts["HOSTNAME"],
        }
        if "SERVICEDESC" in event.eventopts:
            json_payload['service_description'] = event.eventopts['SERVICEDESC']
            if event.eventopts["SERVICESTATE"] == "WARNING":
                json_payload['output'] = "dem {} vong {} is schlecht".format(event.eventopts['SERVICEDESC'], event.eventopts['HOSTNAME'])
            elif event.eventopts["SERVICESTATE"] == "CRITICAL":
                json_payload['output'] = "dem {} vong {} is vol kaputt".format(event.eventopts['SERVICEDESC'], event.eventopts['HOSTNAME'])
            else:
                json_payload['output'] = "i bim mit dem Serviz {} vong {} voll zufriedn".format(event.eventopts['SERVICEDESC'], event.eventopts['HOSTNAME'])
        else:
            json_payload['output'] = event.eventopts["HOSTOUTPUT"]
            if event.eventopts["HOSTSTATE"] == "DOWN":
                json_payload['output'] = "dem {} is vol kaputt".format(event.eventopts["HOSTNAME"])
            else:
                json_payload['output'] = "dem {} is 1 host mid Niceigkeit".format(event.eventopts["HOSTNAME"])

        event.payload = json_payload
        event.summary = "i hab dem post gepost"

After you added the two notification commands to your default contact (or created a new contact which is assigned to all hosts and services), you can watch the notifications appear on https://webhook.site. Also check the logfile var/log/notificationforwarder_webhook.log

SyslogForwarder

The SyslogForwarder class takes a simple event, where the payload is one line of text. It sends this text to a syslog server. The possible value for --forwarderopts are:

There is also a SyslogFormatter, which creates the log line as:
host: <HOSTNAME>, service: <SERVICEDESC>, state: <SERVICESTATE>, output: <SERVICEOUTPUT>

If you want a different format, then copy lib/python/notificationforwarder/syslog/formatter.py to local/lib/python/notificationforwarder/syslog/formatter.py and modify it like you want. Or, with --formatter, you can use whatever formatter is suitable, as long as it's payload attribute consists of a line of text.

Nostr

The Nostr plugin pair publishes an encrypted direct message with labeled lines and default monitoring tags so the alert appears in chat-oriented clients.

Install the optional Nostr support with pip install notificationforwarder[nostr].

The recipient is identified by a p tag containing the recipient's public key (npub). The p tag must be present in the tags list — the forwarder will raise an error if no recipient is found.

Static recipient

If all notifications go to the same recipient, pass the p tag via --forwarderopt tags:

$USER1$/notificationforwarder \
    --forwarder nostr \
    --forwarderopt relays='wss://relay.damus.io,wss://nostr-pub.wellorder.net' \
    --forwarderopt nsec='$NSEC$' \
    --forwarderopt tags='[["p", "npub1xyz..."]]' \
    --formatter nostr \
    --eventopt HOSTNAME='$HOSTNAME$' \
    --eventopt SERVICEDESC='$SERVICEDESC$' \
    --eventopt SERVICESTATE='$SERVICESTATE$' \
    --eventopt SERVICEOUTPUT='$SERVICEOUTPUT$'

Per-contact recipient

In a real monitoring setup, different contacts may have different Nostr public keys. Use a custom contact variable (e.g., $CONTACTNOSTRNPUB$) and pass it as an eventopt. The formatter then adds it as the p tag.

Contact definition:

define contact {
    contact_name    admin-team
    ...
    _NOSTRNPUB      npub1xyz...
}

Command definition:

define command {
    command_name    notify-service-nostr
    command_line    $USER1$/notificationforwarder \
                        --forwarder nostr \
                        --forwarderopt relays='wss://relay.damus.io' \
                        --forwarderopt nsec='$NSEC$' \
                        --formatter nostr_custom \
                        --eventopt HOSTNAME='$HOSTNAME$' \
                        --eventopt SERVICEDESC='$SERVICEDESC$' \
                        --eventopt SERVICESTATE='$SERVICESTATE$' \
                        --eventopt SERVICEOUTPUT='$SERVICEOUTPUT$' \
                        --eventopt NOSTR_PUBKEY='$_CONTACTNOSTRNPUB$' \
                    >> $USER4$/var/log/notificationforwarder_errors.log 2>&1
}

Custom formatter in ~/local/lib/python/notificationforwarder/nostr_custom/formatter.py:

from notificationforwarder.nostr.formatter import NostrFormatter

class NostrCustomFormatter(NostrFormatter):
    def format_event(self, event):
        super().format_event(event)
        pubkey = event.eventopts.get("NOSTR_PUBKEY", "")
        if pubkey:
            event.payload["tags"].append(["p", pubkey])

The p tag is appended to the tags list that the default Nostr formatter already builds with monitoring tags. The forwarder picks it up and uses it as the recipient for the encrypted DM.

The message content is encrypted before relay publication, so it shows up as a DM in clients that support NIP-04 direct messages.

Loggers

The framework uses a modular logging architecture similar to formatters, forwarders, and reporters. By default, notificationforwarder uses text format logging - you don't need to do anything, logging works exactly as it did before. The traditional text format is backward compatible with all existing installations.

Why JSON Logging?

In enterprise environments, the gateway from monitoring systems to incident management platforms like Remedy, ServiceNow, or other ITSM tools is crucial for operational reliability. For comprehensive monitoring and troubleshooting of this critical path, logs need to be ingested into log aggregation systems like Splunk for analysis, alerting, and correlation.

The JSON logger provides structured logging optimized for ingestion into Splunk and other log management systems. It outputs single-line JSON with:

• Splunk-friendly underscore field naming (e.g., event_host_name, event_service_name)
• Complete event details including state, output, and summary
• Operational metrics (queue length, spool counts, retry attempts)
• Structured exception traces
• Timezone-aware timestamps

Usage

Default (text logging):

$USER1$/notificationforwarder \
    --forwarder webhook \
    --forwarderopt url=https://api.example.com/tickets \
    --eventopt HOSTNAME='$HOSTNAME$' \
    --eventopt SERVICESTATE='$SERVICESTATE$'

JSON logging for Splunk ingestion:

$USER1$/notificationforwarder \
    --forwarder webhook \
    --forwarderopt url=https://api.example.com/tickets \
    --logger json \
    --eventopt HOSTNAME='$HOSTNAME$' \
    --eventopt SERVICESTATE='$SERVICESTATE$'

Custom logger:

$USER1$/notificationforwarder \
    --forwarder webhook \
    --logger mycustomlogger \
    --eventopt HOSTNAME='$HOSTNAME$'

Example Log Output

Text format (default):

2025-11-13 17:00:57,987 3468977 - INFO - forwarded dbserver02.example.com/MySQL: WARNING - Slow queries

JSON format:

{
  "timestamp": "2025-11-13T17:00:57.987487+01:00",
  "host_name": "oasch.example.com",
  "version": "2.9",
  "level": "INFO",
  "logger": "notificationforwarder_webhook",
  "omd_site": "demo_site",
  "event_host_name": "dbserver02.example.com",
  "event_service_name": "MySQL",
  "event_state": "WARNING",
  "event_notification_type": "PROBLEM",
  "event_service_output": "MySQL WARNING - Slow queries detected",
  "event_summary": "dbserver02.example.com/MySQL: WARNING - Slow queries",
  "msg": {
    "message": "forwarded",
    "status": "success"
  }
}

Custom Loggers

You can create custom loggers by:

1. Creating ~/local/lib/python/notificationforwarder/mylogger/logger.py
2. Inheriting from NotificationLogger base class
3. Implementing the log(level, message, context) method

from notificationforwarder.baseclass import NotificationLogger

class MyloggerLogger(NotificationLogger):
    def log(self, level, message, context=None):
        # Custom logging implementation
        pass

Runtime Architecture

The runtime keeps the historical CLI and plugin model, but the core flow is now split into clearer responsibilities instead of concentrating everything in one large class.

• baseclass.new(...): compatibility entrypoint that creates a configured forwarder instance.
• runtime_config.py: normalizes runtime options, default values, log naming, and spool-related paths.
• component_loader.py: resolves and loads forwarders, formatters, reporters, and loggers using the established naming rules.
• runtime_flow.py: enriches raw events and applies the common forwarding/reporting result flow.
• spool.py: owns sqlite spool persistence, replay batches, and flush-lock retry behavior.

Forwarding Flow

The runtime flow is:

1. Normalize runtime configuration and initialize logging.
2. Load the forwarder implementation.
3. Enrich the raw event with OMD/site metadata.
4. Load the formatter and build a FormattedEvent.
5. Attempt delivery.
6. If delivery fails, spool the raw event for retry unless it is a heartbeat.
7. If a reporter is configured, attach forwarding context and call the reporter.

Extension Contracts

Custom extensions still follow the existing filesystem and naming conventions:

• Forwarder: notificationforwarder/<name>/forwarder.py with class <Name>Forwarder
• Formatter: notificationforwarder/<name>/formatter.py with class <Name>Formatter
• Reporter: notificationforwarder/<name>/reporter.py with class <Name>Reporter
• Logger: notificationforwarder/<name>/logger.py with class <Name>Logger

If you pass a dotted class path instead of a simple name, the runtime uses it as an explicit module/class reference. Otherwise it derives the class name by capitalizing the plugin name and appending the expected suffix.

Runtime Guarantees

• Text logging remains the default and fallback logger.
• Unknown logger implementations fall back to the text logger and emit a warning.
• Failed deliveries are spooled in sqlite when possible.
• Spooled events are retried on later executions before the new event is sent, as long as they have not expired.
• Expired spool entries are dropped during flush and logged as outdated events.
• Concurrent flush attempts for the same forwarder are suppressed with a lock file.

Contributor Verification

When changing runtime code, run the relevant tests at minimum:

pytest tests/test_runtime_foundation.py tests/test_delivery_resilience.py tests/test_classes.py

For broader confidence, run the full suite:

pytest

Reporters

Like forwarder and formatter, a reporter is an instance of a NotificationReporter class defined in a file named reporter.py. There is one class coming with notificationforwarder, the NaemonlogReporter. It's purpose it to write a message to the Naemon logfile. When notificationforwarder is run as a standalone script (and not triggered as a notificationhandler by Naemon), the NaemonlogReporter can nevertheless leave a line in the Naemon log. Or you can write an extra log showing success or failure of the notification delivery.

define command{
    command_name    notify-service-servicenow
    command_line    $USER1$/notificationforwarder \
                        --forwarder webhook \
                        --forwarderopt username='$_CONTACTUSERNAME$' \
                        --forwarderopt password='$_CONTACTPASSWORD$' \
                        --forwarderopt url='$_CONTACTURL$' \
...
                        --eventopt HOSTNAME='$HOSTNAME$' \
                        --eventopt HOSTSTATE='$HOSTSTATE$' \
                        --eventopt HOSTADDRESS='$HOSTADDRESS$' \
                        --eventopt SERVICEDESC='$SERVICEDESC$' \
                        --eventopt SERVICESTATE='$SERVICESTATE$' \
                        --eventopt SERVICEOUTPUT='$SERVICEOUTPUT$' \
                        --eventopt LONGSERVICEOUTPUT='$LONGSERVICEOUTPUT$' \
....
                        --reporter naemonlog \
                    >> $USER4$/var/log/notificationforwarder_errors.log 2>&1
}

Spool Flushing

When forwarding to a recipient fails, the event is saved in a local sqlite database. The framework automatically attempts to re-submit these spooled events on every subsequent call to notificationforwarder for the same forwarder.

The flush mechanism runs inside forward_formatted() for every event — not just heartbeats or special cases. Before submitting the current event, the framework checks if there are spooled events waiting. If there are, it attempts to flush them by calling submit() for each one, oldest first. Events that are successfully submitted are deleted from the spool. The first event that fails again ends the flush: it and all later events stay in the queue for the next attempt. A receiver that does not accept one event will most likely not accept the next ones either, so a flush costs at most one submit timeout instead of one per spooled event.

If the receiver rejects a spooled event for good (for example because of an invalid payload), that event must not block the queue until max_spool_minutes have passed. When a flush stopped at an event and the current event is then accepted by the receiver, the receiver is evidently up. The stopped event is moved to the end of the queue and logged as "spooled event moved to the end of the spool". It keeps its original time, so it is still dropped when max_spool_minutes have passed, and the next flush can deliver the events behind it.

This means that in a busy monitoring environment where notifications flow regularly, spooled events get drained naturally — each new notification triggers a flush attempt on its way through.

To monitor the size of the spool, use the line "spooling queue length". It is logged with the whole number of spooled events every time an event is spooled (level WARNING) and at the end of every flush (level INFO), so the value also goes down again while the spool drains. With the JSON logger, the number can be read with the regular expression "message":"spooling queue length","queue_length":(\d+). The value is only logged when notificationforwarder runs, so it can be outdated when no notifications arrive. A check that does not depend on notifications can count the rows in the spool database directly:

sqlite3 $OMD_ROOT/var/tmp/notificationforwarder_<forwarder>_notifications.db "SELECT COUNT(*) FROM events_<forwarder>"

<forwarder> is the value of --forwarder, followed by _ and the value of --forwardertag if one is set (for example remedy_TEST).

Delivery Order

Spooled events are stored and re-submitted in the order in which they failed. As long as every delivery succeeds, the receiver gets the events in the order in which they were created. After delivery failures, this is not guaranteed. A newer event can reach the receiver before an older one, for example a RECOVERY before its PROBLEM:

• While one notificationforwarder process is flushing the spool, another process submits its own new event directly, ahead of the spooled ones.
• A spooled event was rejected while the receiver accepted the current event. The rejected event is moved to the end of the spool and is delivered, if at all, after newer events.
• The flush stopped early, or probe() returned False. The process still submits its own new event directly.
• A submit ran into the timeout, but the receiver got the event anyway. The event stays in the spool, and the receiver gets it a second time later, after newer events.

If the receiver depends on the order, let the formatter include the event's creation time (omd_originating_timestamp, set for every event) in the payload. The receiver or its integration can then ignore an event that is older than the state it already has.

Probe Method

Forwarders can implement an optional probe() method to optimize the flush behavior. The probe() method checks whether the receiving system is reachable before the framework attempts to flush spooled events via submit().

The framework uses probe() as a pre-check with three possible scenarios:

Without probe(), every flush attempt would call submit() for each spooled event even when the receiver is known to be down — wasting time and producing noisy error logs. With probe(), the framework can short-circuit: if the receiver is unreachable, the flush is skipped entirely and spooled events wait in the database until the next call.

Example implementation:

def probe(self):
    r = requests.head(self.url)
    return r.status_code == 200

Heartbeat Events

Formatters can mark an event as a heartbeat by setting event.is_heartbeat = True on the FormattedEvent object. Heartbeat events have two special properties:

1. They skip spooling on failure. If a heartbeat event cannot be delivered, it is NOT saved to the spool database. This prevents periodic check-in events from accumulating in the retry queue.

2. They guarantee regular flush attempts. While every event triggers the flush phase, heartbeat events ensure this happens at regular intervals — but only when scheduled. A cronjob or Nagios service using notificationforwarder as its check command provides this scheduling, creating a steady pulse of heartbeat events. Without such a schedule, heartbeats don't exist and spooled events could sit in the queue for hours waiting for the next real notification to trigger a flush.

Example formatter that sets the heartbeat flag:

class MyFormatter(NotificationFormatter):
    def format_event(self, event):
        notification_type = event.eventopts.get("NOTIFICATIONTYPE", "")
        event.is_heartbeat = notification_type == "HEARTBEAT"

        event.payload = {
            "hostname": event.eventopts.get("HOSTNAME", ""),
            "heartbeat": event.is_heartbeat,
        }
        event.summary = "heartbeat from {}".format(event.eventopts.get("HOSTNAME", ""))

Scheduling Heartbeats in Nagios/Naemon

Since notificationforwarder always exits with code 0, it can be used as a Nagios check command to schedule periodic heartbeats. The service will always be in OK status — it's using Nagios purely as a scheduling mechanism.

Command definition:

define command {
    command_name    notificationforwarder_heartbeat
    command_line    $USER1$/notificationforwarder \
                        --forwarder webhook \
                        --forwarderopt url=https://xxx.service-now.com/api/now/import/u_notification \
                        --formatter myformatter \
                        --eventopt HOSTNAME='$HOSTNAME$' \
                        --eventopt SERVICEDESC='$SERVICEDESC$' \
                        --eventopt NOTIFICATIONTYPE=HEARTBEAT \
                        --eventopt SERVICEOUTPUT='heartbeat from $HOSTNAME$' \
                    >> $USER4$/var/log/notificationforwarder_errors.log 2>&1
}

Service definition:

define service {
    host_name               monitoring-server
    service_description     notificationforwarder-heartbeat
    check_command           notificationforwarder_heartbeat
    max_check_attempts      1
    check_interval          5
    retry_interval          1
    notification_interval   0
    notification_options    n
    notification_period     24x7
}

The key settings:

• check_interval 5 — runs the heartbeat every 5 minutes
• notification_interval 0 — no Nagios-level notifications
• notification_options n — don't send notifications on state changes

The heartbeat goes through the normal notificationforwarder flow: the formatter sets is_heartbeat = True, the flush phase checks for spooled events (and calls probe() if available), and the heartbeat itself is submitted to the receiver. If the receiver accepts it, the receiving system knows the monitoring infrastructure is alive (dead-man switch). If the receiver is down, the heartbeat is simply dropped — it won't add to the spool queue.

The two use cases for periodic heartbeats:

• Dead-man switch: The receiving system (e.g., ServiceNow) gets a regular "I'm alive" signal. If heartbeats stop, it knows the monitoring system is down.
• Spool draining: Even when nothing breaks, the heartbeat triggers flush attempts at regular intervals, so spooled events are delivered quickly once the receiver recovers — without waiting for the next real notification.