prometheus-salt-exporter 0.1.4

Prometheus Exporter for Salt highstate metrics run from the Salt master.

Prometheus Exporter for Salt highstate metrics run from the Salt master. The exporter must have permissions to execute commands locally.

Note: Python 2 is not supported. Instead use Python 3.8 or higher.

Inspired by BonnierNews/saltstack_exporter

Prerequisites

• Install salt

• Be on the salt master node

Installation

pip install prometheus-salt-exporter

Configuration and Usage

usage: prometheus_salt_exporter [-h] [--listen-addr LISTEN_ADDR] [--listen-port LISTEN_PORT] [--highstate-interval HIGHSTATE_INTERVAL] [--wait-on-error-interval WAIT_ON_ERROR_INTERVAL]
               [--batch-size BATCH_SIZE] [--salt-target SALT_TARGET] [--log-level LOG_LEVEL]

options:
-h, --help            show this help message and exit
--listen-addr LISTEN_ADDR
                        Address to bind to. IPv4 and IPv6 addresses can be specified. (default: ::)
--listen-port LISTEN_PORT
                        Port to bind to (default: 9175)
--highstate-interval HIGHSTATE_INTERVAL
                        Seconds between each highstate test run (default: 300)
--wait-on-error-interval WAIT_ON_ERROR_INTERVAL
                        Seconds to wait when an error occurs (e.g. salt-master not responding in time) (default: 300)
--batch-size BATCH_SIZE
                        Batch size to use in salt (default: 10)
--batch-wait BATCH_WAIT
                        Seconds to wait after a minion returns, before sending the command to a new minion (default: 10)
--salt-target SALT_TARGET
                        Salt target to be used (default: *)
--log-level LOG_LEVEL
                        log level (default: 30)

Metrics

Currently, the exporter exposes metrics for highstate conformity only:

Metric	Description
saltstack_states_total	Number of states which apply to the minion in highstate
saltstack_nonhigh_states	Number of states which would change on state.highstate
saltstack_error_states	Number of states which returns an error on highstate dry-run
saltstack_highstate_error	Error in trying to apply highstate
saltstack_last_highstate	Timestamp of the last highstate test run

Output

# HELP saltstack_last_highstate_total Timestamp of the last highstate test run
# TYPE saltstack_last_highstate_total counter
saltstack_last_highstate_total 1.674730426e+09
# HELP saltstack_states_total Number of states which apply to the minion in highstate
# TYPE saltstack_states_total gauge
saltstack_states_total{minion="1.minion"} 271.0
# HELP saltstack_nonhigh_states Number of states which would change on state.highstate
# TYPE saltstack_nonhigh_states gauge
saltstack_nonhigh_states{minion="1.minion"} 13.0
# HELP saltstack_error_states Number of states which returns an error on highstate dry-run
# TYPE saltstack_error_states gauge
saltstack_error_states{minion="1.minion"} 0.0
# HELP saltstack_highstate_error Error in trying to apply highstate
# TYPE saltstack_highstate_error gauge
saltstack_highstate_error{minion="1.minion"} 0.0
# HELP saltstack_states_total Number of states which apply to the minion in highstate
# TYPE saltstack_states_total gauge
saltstack_states_total{minion="2.minion"} 197.0
# HELP saltstack_nonhigh_states Number of states which would change on state.highstate
# TYPE saltstack_nonhigh_states gauge
saltstack_nonhigh_states{minion="2.minion"} 6.0
# HELP saltstack_error_states Number of states which returns an error on highstate dry-run
# TYPE saltstack_error_states gauge
saltstack_error_states{minion="2.minion"} 0.0
# HELP saltstack_highstate_error Error in trying to apply highstate
# TYPE saltstack_highstate_error gauge
saltstack_highstate_error{minion="2.minion"} 0.0

Troubleshooting

Resource issues

If you encounter resource issues (e.g. your salt-master cannot keep up with providing the highstate), please consider using a combination of --batch-wait and --batch-size by decreasing the batch size and increasing the waiting time between requests.