arrakis-monitor

HTTP health sidecar for Arrakis Flight servers.

Overview

arrakis-monitor runs alongside each Arrakis server and exposes its health status over HTTP, allowing standard monitoring tools (Nagios/Icinga via NRPE + check_http) to check gRPC Flight servers without needing a gRPC-native plugin.

Installation

pip install arrakis-monitor

Usage

# basic health sidecar
arrakis-monitor --server grpc://localhost:31206

# with scope map domain validation
arrakis-monitor --server grpc://localhost:31206 --expect-domains H1,L1,V1

# with publisher liveliness/latency checks (online servers)
arrakis-monitor --server grpc://localhost:31206 --check-publishers

# custom bind address
arrakis-monitor --server grpc://localhost:31206 --host 127.0.0.1 --port 9090

Endpoints

Health and scope-map endpoints return JSON with an HTTP status code:

• 200 — OK or WARNING
• 503 — CRITICAL (server unreachable or scope map empty)
• 500 — UNKNOWN (unexpected error)

Response format

Both check endpoints include a latency_s field measuring the gRPC call round-trip time in seconds:

{
  "status": "ok",
  "message": "Server 0.12.0 (kafka)",
  "version": "0.12.0",
  "backend": "kafka",
  "domains": ["H1", "L1"],
  "capabilities": ["find", "count", "describe", "stream"],
  "latency_s": 0.042
}

Publisher checks

On online (Kafka-backed) servers, --check-publishers enables per-publisher liveliness and latency monitoring. For each publisher tagged in the server's scope map, the sidecar consumes the publisher's metadata topic from the local broker to select a surrogate channel, then probes the data topic for that channel's partition, waiting for a fresh message. The block-time latency of the freshest record batch is compared against the max_latency advertised in the channel metadata:

• OK — fresh data arrived within the latency budget
• WARNING — latency within --publisher-warn-margin (default 1 s) of max_latency
• CRITICAL — latency at/past max_latency, or no message within max_latency plus one block stride (reported with the age of the last retained message)

/publishers probes every publisher concurrently and reports the worst status; /publishers/{publisher} probes one, for fine-grained per-publisher alerting. Publishers registered in the scope map ahead of going live would keep the aggregate check CRITICAL, so list them with --ignore-publisher — /publishers skips them (reporting them under ignored) while /publishers/{publisher} still probes them on request. A server's scope map may describe every endpoint at its site; pass --scope-endpoint with the URL the scope map uses for this server so /publishers covers only its own publishers. Surrogate resolution is cached per publisher (--metadata-ttl, default 600 s), and check results are cached briefly (--probe-cache-ttl, default 60 s) so aggregate and per-publisher checks polling around the same time share one probe — a shared result carries its age in a cached_age_s field.

Options:

{
  "status": "ok",
  "message": "Publisher 'H1_lldd' live (latency 3.2s)",
  "publisher": "H1_lldd",
  "channel": "H1:AOS-ETMX_BAFFLEPD_1_DC_POWER",
  "replay_id": null,
  "max_latency_s": 12.0,
  "warn_latency_s": 11.0,
  "latency_s": 3.2,
  "transport_latency_s": 0.05,
  "block_time_gps": 1400000000.0
}

NRPE Integration

Each Arrakis host runs the sidecar locally, and NRPE executes check_http against the sidecar's port. The -w and -c flags set warning and critical thresholds on the HTTP response time (in seconds). When the Arrakis hosts are not reachable from the monitoring server, the commands can live on an NRPE gateway host on the same network instead, targeting each host by name:

# health check (warn at 2s, critical at 5s)
command[check_arrakis]=/usr/lib64/nagios/plugins/check_http -H <host> -p 8080 -u /health -w 2 -c 5

# scope map validation (info servers only)
command[check_arrakis_scope_map]=/usr/lib64/nagios/plugins/check_http -H <host> -p 8080 -u /scope-map -w 2 -c 5

# publisher liveliness/latency, every publisher at once (online servers
# with --check-publishers); probes wait for live data, so allow more
# response time than the other checks
command[check_arrakis_publishers]=/usr/lib64/nagios/plugins/check_http -H <host> -p 8080 -u /publishers -w 10 -c 30 -t 40

# per-publisher liveliness/latency, one command per publisher
command[check_arrakis_publisher_<publisher>]=/usr/lib64/nagios/plugins/check_http -H <host> -p 8080 -u /publishers/<publisher> -w 10 -c 30 -t 40

Publisher probes can wait up to --probe-timeout (30 s) for live data, so the NRPE client needs a matching timeout in place of its 10 s default (check_nrpe -t 45, or vars.nrpe_timeout = 45 on the Icinga service).

Resources

• Documentation
• Source Code
• Issue Tracker