arrakis-monitor

HTTP health sidecar for Arrakis Flight servers.

Overview

arrakis-monitor runs alongside each Arrakis server and exposes its health status over HTTP, allowing standard monitoring tools (Nagios/Icinga via NRPE + check_http) to check gRPC Flight servers without needing a gRPC-native plugin.

Installation

pip install arrakis-monitor

Usage

# basic health sidecar
arrakis-monitor --server grpc://localhost:31206

# with scope map domain validation
arrakis-monitor --server grpc://localhost:31206 --expect-domains H1,L1,V1

# with publisher liveliness/latency checks (online servers)
arrakis-monitor --server grpc://localhost:31206 --check-publishers

# custom bind address
arrakis-monitor --server grpc://localhost:31206 --host 127.0.0.1 --port 9090

Endpoints

Health and scope-map endpoints return JSON with an HTTP status code:

• 200 — OK or WARNING
• 503 — CRITICAL (server unreachable or scope map empty)
• 500 — UNKNOWN (unexpected error)

Response format

Both check endpoints include a latency_s field measuring the gRPC call round-trip time in seconds:

{
  "status": "ok",
  "message": "Server 0.12.0 (kafka)",
  "version": "0.12.0",
  "backend": "kafka",
  "domains": ["H1", "L1"],
  "capabilities": ["find", "count", "describe", "stream"],
  "latency_s": 0.042
}

Publisher checks

On online (Kafka-backed) servers, --check-publishers enables per-publisher liveliness and latency monitoring. For each publisher tagged in the server's scope map, the sidecar consumes the publisher's metadata topic from the local broker to select a surrogate channel, then probes the data topic for that channel's partition, waiting for a fresh message. The block-time latency of the freshest record batch is compared against the max_latency advertised in the channel metadata:

• OK — fresh data arrived within the latency budget
• WARNING — latency within --publisher-warn-margin (default 1 s) of max_latency
• CRITICAL — latency at/past max_latency, or no message within max_latency plus one block stride (reported with the age of the last retained message)

/publishers probes every publisher concurrently and reports the worst status; /publishers/{publisher} probes one, for fine-grained per-publisher alerting. Surrogate resolution is cached per publisher (--metadata-ttl, default 600 s), and check results are cached briefly (--probe-cache-ttl, default 60 s) so aggregate and per-publisher checks polling around the same time share one probe — a shared result carries its age in a cached_age_s field.

Options:

{
  "status": "ok",
  "message": "Publisher 'H1_lldd' live (latency 3.2s)",
  "publisher": "H1_lldd",
  "channel": "H1:AOS-ETMX_BAFFLEPD_1_DC_POWER",
  "replay_id": null,
  "max_latency_s": 12.0,
  "warn_latency_s": 11.0,
  "latency_s": 3.2,
  "transport_latency_s": 0.05,
  "block_time_gps": 1400000000.0
}

NRPE Integration

Each Arrakis host runs the sidecar locally, and NRPE executes check_http against localhost:8080. The -w and -c flags set warning and critical thresholds on the HTTP response time (in seconds):

# health check (warn at 2s, critical at 5s)
command[check_arrakis]=/usr/lib/nagios/plugins/check_http -H localhost -p 8080 -u /health -w 2 -c 5

# scope map validation (info servers only)
command[check_arrakis_scope_map]=/usr/lib/nagios/plugins/check_http -H localhost -p 8080 -u /scope-map -w 2 -c 5

# publisher liveliness/latency (online servers with --check-publishers);
# probes wait for live data, so allow more response time than the other checks
command[check_arrakis_publishers]=/usr/lib/nagios/plugins/check_http -H localhost -p 8080 -u /publishers -w 10 -c 30 -t 40

For per-publisher alerting, point check_http at /publishers/<publisher-id> with one command per publisher.

See examples/nrpe/ for per-host configuration examples for all sites.

Resources

• Documentation
• Source Code
• Issue Tracker