jool-exporter 21.6.15

Export `jool stats display` for prometheus

jool-exporter

jool SIIT-DC + NAT64 stats prometheus exporter

What is this?

jool-exporter is a prometheus exporter HTTP service that wraps jool stats display and reads the statistics at the time of the request into promerthues compatible format.

• jool-exporter changes JSTAT prefix to jool_ prefix for key names
• We attach the explanation of each stat to the guages

Install

From PyPI:

• pip install jool-exporter

From GitHub:

• pip install git+git://github.com/cooperlees/jool-exporter

Running

jool CLI needs CAP_NET_ADMIN capability in order to pull the statistics. Due to this, so does the jool-exporter process. It also need to ability to pass the capability to child processes.

• From version 4.1.5 jool will no longer require this capability to query stats

The process can also just run was root, but running things listening externally as root is a bad security process.

SystemD

We have a Systemd Service unit file commited to the repo that runs as nobody and passes the capability to all children process.

• This is the recommended way to run the service

SystemD install

• cp jool-exporter.service /etc/systemd/system
• sudo systemctl daemon-reload
• sudo systemctl enable jool-exporter
• sudo systemctl start jool-exporter

Logs will by default go to journald

• journalctl -u jool-exporter [-f]

Example Return

• curl http://localhost:6971/metrics

cooper@home1:~$ curl http://localhost:6971/metrics
# HELP python_gc_objects_collected_total Objects collected during gc
# TYPE python_gc_objects_collected_total counter
python_gc_objects_collected_total{generation="0"} 373.0
python_gc_objects_collected_total{generation="1"} 0.0
python_gc_objects_collected_total{generation="2"} 0.0
# HELP python_gc_objects_uncollectable_total Uncollectable object found during GC
# TYPE python_gc_objects_uncollectable_total counter
python_gc_objects_uncollectable_total{generation="0"} 0.0
python_gc_objects_uncollectable_total{generation="1"} 0.0
python_gc_objects_uncollectable_total{generation="2"} 0.0
# HELP python_gc_collections_total Number of times this generation was collected
# TYPE python_gc_collections_total counter
python_gc_collections_total{generation="0"} 40.0
python_gc_collections_total{generation="1"} 3.0
python_gc_collections_total{generation="2"} 0.0
# HELP python_info Python platform information
# TYPE python_info gauge
python_info{implementation="CPython",major="3",minor="8",patchlevel="5",version="3.8.5"} 1.0
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.81788672e+08
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 2.0611072e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.60830649484e+09
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 15.11
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 6.0
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1024.0
# HELP jool_received6 jool metric
# TYPE jool_received6 gauge
jool_received6{hostname="home1.cooperlees.com"} 7.9109475e+07
# HELP jool_received4 jool metric
# TYPE jool_received4 gauge
jool_received4{hostname="home1.cooperlees.com"} 1.3273675e+08
# HELP jool_success Successful translations. (Note: 'Successful translation' does not imply that the packet was actually delivered.)
# TYPE jool_success gauge
jool_success{hostname="home1.cooperlees.com"} 4.501274e+06
# HELP jool_bib_entries Number of BIB entries currently held in the BIB.
# TYPE jool_bib_entries gauge
jool_bib_entries{hostname="home1.cooperlees.com"} 245.0
# HELP jool_sessions Number of session entries currently held in the BIB.
# TYPE jool_sessions gauge
jool_sessions{hostname="home1.cooperlees.com"} 245.0
# HELP jool_unknown_l4_proto Translations cancelled: Packet carried an unknown transport protocol. (Untranslatable by NAT64.)
# TYPE jool_unknown_l4_proto gauge
jool_unknown_l4_proto{hostname="home1.cooperlees.com"} 6783.0
# HELP jool_unknown_icmp6_type Translations cancelled: ICMPv6 header's type value has no ICMPv4 counterpart.
# TYPE jool_unknown_icmp6_type gauge
jool_unknown_icmp6_type{hostname="home1.cooperlees.com"} 279293.0
# HELP jool_pool6_mismatch Translations cancelled: IPv6 packet's destination address did not match pool6. (ie. Packet was not meant to be translated.)
# TYPE jool_pool6_mismatch gauge
jool_pool6_mismatch{hostname="home1.cooperlees.com"} 7.7009472e+07
# HELP jool_pool4_mismatch Translations cancelled: IPv4 packet's destination address and transport protocol did not match pool4. (ie. Packet was not meant to be translated.)\nIf the instance is a Netfilter translator, this counter increases randomly from normal operation, and is harmless.\nIf the instance is an iptables translator, this counter being positive suggests a mismatch between the IPv4 iptables rule(s) and the instance's configuration.
# TYPE jool_pool4_mismatch gauge
jool_pool4_mismatch{hostname="home1.cooperlees.com"} 1.30048136e+08
# HELP jool_bib4_not_found Translations cancelled: IPv4 packet did not match a BIB entry from the database.
# TYPE jool_bib4_not_found gauge
jool_bib4_not_found{hostname="home1.cooperlees.com"} 238.0
# HELP jool_syn6_expected Translations cancelled: Incoming IPv6 packet was the first of a TCP connection, but its SYN flag was disabled.
# TYPE jool_syn6_expected gauge
jool_syn6_expected{hostname="home1.cooperlees.com"} 268.0
# HELP jool_syn4_expected Translations cancelled: Incoming IPv4 packet was the first of a TCP connection, but its SYN flag was disabled.
# TYPE jool_syn4_expected gauge
jool_syn4_expected{hostname="home1.cooperlees.com"} 461.0
# HELP jool_type1pkt Total number of Type 1 packets stored. (See https://github.com/NICMx/Jool/blob/584a846d09e891a0cd6342426b7a25c6478c90d6/src/mod/nat64/bib/pkt_queue.h#L77) (This counter is not decremented when a packet leaves the queue.)
# TYPE jool_type1pkt gauge
jool_type1pkt{hostname="home1.cooperlees.com"} 299.0
# HELP jool_so_exists Translations cancelled: Packet was a Simultaneous Open retry. (Client was trying to punch a hole, and was being unnecessarily greedy.)
# TYPE jool_so_exists gauge
jool_so_exists{hostname="home1.cooperlees.com"} 1.0

Grafana Dashbaord Example

Development

We use Facebook's ptr for testing.

• pip install ptr
• cd . # This repo
• ptr [-k]

It is driven by config in setup.py.