prometheus-dcache-exporter 0.2.0

A Prometheus exporter for dCache.

This Prometheus exporter collects metrics from dCache and provides them via a HTTP server, writes them to standard output or writes them atomically to a file.

Building And Installation

Dependencies

• Python 3.13

• The packages/modules certifi, httpx, humanfriendly, jsonpath, module_utils, paramiko and prometheus_client.

• Optionally, the packages/modules logging_extras, rich and rich_argparse.

Getting The Source Code

Clone the prometheus-dcache-exporter git repository via either of:

$ git clone ssh://[e-mail]/calestyo/prometheus-dcache-exporter.git

$ git clone https://gitlab.com/calestyo/prometheus-dcache-exporter.git

Building A Debian Package

The Debian packaging is maintained in the debian-branch of the prometheus-dcache-exporter git repository.

Preparations:

$ cd prometheus-dcache-exporter
$ git switch -c temporary-build-branch master
$ temporary_version="$(python3 -m setuptools_scm --format plain)"
$ [ -z "$( git tag --list "v${temporary_version}" )" ]  &&  git tag "v${temporary_version}"
$ git merge --allow-unrelated-histories -m "Merge branch 'debian' into temporary-build-branch" debian
$ debian_changelog_version="$(dpkg-parsechangelog --show-field Version)"
$ [ "${temporary_version}"   !=   "$( printf '%s\n' "${debian_changelog_version}"  |  sed -E 's/^[0-9]+://; s/-[^-]+$//;' )" ]    &&    debchange --release-heuristic log --newversion "$( printf '%s\n' "${debian_changelog_version}"  |  sed -E -n 's/^([0-9]+:).*/\1/p' )${temporary_version}-1" 'Local version.'
$ git deborig

Any changes since the tip of the master-branch must be committed right after switching to the temporary-build-branch-branch.

Build via either of:

$ dpkg-buildpackage

$ debuild

optionally adding the --no-sign-option if signing is not desired.

Beware that the above creates files in the parent directory.

git-deborig is part of the git-debpush-package, dpkg-buildpackage and dpkg-parsechangelog are part of the dpkg-dev-package and debuild is part of the devscripts-package.

Installation From The Python Package Index (PyPI)

$ pip install prometheus-dcache-exporter

Usage

There are two main operation modes:

• Executing prometheus-dcache-exporter in the http-export-mode via a systemd service, et cetera, with the former providing the metrics via an internal HTTP server.

  The metrics are (freshly) collected on every scrape. The scraping is to be done directly (by Prometheus).
• Executing prometheus-dcache-exporter in the file-export-mode or in the stdout-export-mode together with a program like sponge (in both cases) via a systemd timer, cron job, et cetera, which writes the metrics atomically (via moving a newly created temporary file) to a .prom-file for Prometheus node exporter’s textfile collector.

  The metrics are (periodically) collected (once) on every execution. The scraping is to be done indirectly via the Prometheus node exporter (by Prometheus).

Invoke prometheus-dcache-exporter with the --help- or -h-option for information about the command arguments.

It’s not necessary that dCache respectively the REST interface of its frontend-service and/or the SSH interface of its admin-service are executed on the same host as that on which prometheus-dcache-exporter is executed.

For security reasons and to allow for alleviation of frontend’s rate limiting without affecting other users, it’s suggested to run dedicated instances of (at least) frontend and (possibly also) admin that are only accessible by trusted users (see the example below, which assumes however that all users on the host are trusted).

While sensible defaults have been chosen for all options, in particular the connection and authentication options for the REST and SSH interfaces may need to be adapted.

Amongst others, these are part of the --dcache-frontend-rest.*- respectively the --dcache-admin-ssh.*-family of options.

For example, assuming frontend and admin being executed with their default configurations except for:

frontend.net.listen=localhost
frontend.authn.protocol=http
frontend.authz.readonly=true
frontend.authz.anonymous-operations=READONLY
frontend.authz.unlimited-operation-visibility=true
frontend.limits.rate.overall=10000
frontend.limits.rate.per-client.fractions=100
admin.net.listen=localhost

and login to the SSH interface being authorised for the user admin with the SSH private key at ~/.ssh/id_rsa, prometheus-dcache-exporter might be invoked like:

$ prometheus-dcache-exporter --dcache-frontend-rest.base-url http://localhost:3880/api/ --dcache-admin-ssh.user admin --dcache-admin-ssh.private-key ~/.ssh/id_rsa

When executed via either of prometheus-dcache-exporter.service- or prometheus-node-exporter-dcache.service, the above options are usually set in the ARGS-variable of their respective “configuration file” (see the EnvironmentFile=-option of the respective systemd service).

It shall be noted, that files read by prometheus-dcache-exporter (like ~/.ssh/id_rsa in the example above) need to be accessible by the user executing prometheus-dcache-exporter, which is usually _prometheus-dcache-exporter.

Metrics

Collection

prometheus-dcache-exporter uses the REST interface of dCache’s frontend-service and the SSH interface of dCache’s admin-service for collecting the metrics. The latter is only used for raw data that isn’t available via the former, yet.

As mentioned previously, the metrics are (freshly) collected on every execution respectively scrape. Likewise, dCache (freshly) retrieves the raw data on every request to the REST interface respectively command execution via the SSH interface. This should be considered when choosing Prometheus’ scrape interval.

Connections to the REST and SSH interfaces are tried to be kept open and re-used, as configured via the --dcache-frontend-rest.connection-pool-*-family of options respectively the --dcache-admin-ssh.keepalive-interval-option.

Per default, the requests to the REST interface respectively the command executions via the SSH interface are performed concurrently (using multithreading), as configured via the --dcache-frontend-rest.max-concurrent-requests- and --dcache-admin-ssh.max-concurrent-commands-options. This is done, amongst other reasons, so that the raw data is as closely synchronised as possible.

Errors

The metrics of the <metrics-name-prefix>__errors_*-family indicate errors during metric collection, with the <metrics-name-prefix>__errors_occurred-metric being a boolean that indicates whether any (metric collection) errors occurred or not.

Extra Labels

In general, labels describe the entity to which a time series of a metric corresponds. For example, in:

# HELP dcache_space_reservation_space_total_bytes The total space of a space reservation in bytes.
# TYPE dcache_space_reservation_space_total_bytes gauge
dcache_space_reservation_space_total_bytes{space_reservation_id="110000"} 4748869032235006
dcache_space_reservation_space_total_bytes{space_reservation_id="380987"} 292706065309706

the label space_reservation_id uniquely identifies the entity (here, a space reservation), thereby effectively being a “primary key”.

There may also be labels that are not necessary for unique identification, which is particularly (but not only) the case for “info metrics” (whose names typically end in _info). For example, in:

# HELP dcache_space_reservation_info Information about a space reservation.
# TYPE dcache_space_reservation_info gauge
dcache_space_reservation_info{description="ATLASDATADISK",space_reservation_id="110000"} 1.0
dcache_space_reservation_info{description="ATLASLOCALGROUPDISK",space_reservation_id="380987"} 1.0

only the label space_reservation_id identifies the corresponding space reservation uniquely, while the label description – albeit often uniquely used by dCache operators – is strictly speaking not ensured to be unique and is not necessary for identifying the corresponding space reservation.

Per default and except for info metrics, prometheus-dcache-exporter generally includes only the minimal set of labels that are necessary to uniquely identify the corresponding entity.

Sometimes it may however be convenient to have additional labels included. For example, so that the above metric looks like:

# HELP dcache_space_reservation_space_total_bytes The total space of a space reservation in bytes.
# TYPE dcache_space_reservation_space_total_bytes gauge
dcache_space_reservation_space_total_bytes{description="ATLASDATADISK",space_reservation_id="110000"} 4748869032235006
dcache_space_reservation_space_total_bytes{description="ATLASLOCALGROUPDISK",space_reservation_id="380987"} 292706065309706

This can be achieved – for certain metrics and labels where this seemed reasonable – using the --metrics.extra-label-option.

In Prometheus, the change of a label’s value (as well as the addition/removal of a label) results in a new/different time series. It’s therefore suggested to choose which extra labels to include, if any, already before starting to use prometheus-dcache-exporter and to avoid changing this later.

Issues

Concurrent Requests To The REST Interface Respectively Command Executions Via The SSH Interface And Connection Re-Use

Currently, quite a number of concurrent requests respectively command executions are made, including one request per dCache pool and one command execution per dCache domain.

These particular two cases can easily cause high numbers of concurrent requests respectively command executions and while they should eventually be reduced to only two requests (see dCache issues #8126 and #8127), this may cause various failures, which are unfortunately not yet understood (in particular, whether they’re caused by prometheus-dcache-exporter itself, the libraries it uses or dCache respectively the libraries it uses in the frontend- and/or admin-services).

Because of the concurrency, such failures usually cause log message flooding. Further, because threads cannot really be forcibly terminated in Python, the already running (but all failing) threads continue their execution despite exiting signals like SIGINT or SIGTERM.

Tests seemed to have indicated that it even makes a difference when the connections to the REST and/or SSH interfaces are tunnelled via OpenSSH’s port forwarding (using ssh’s -L-option).

Troubleshooting

REST Interface: HTTP Status Code 429 (Too Many Requests)

In case of log messages like:

ERROR: prometheus_dcache_exporter._dcache.frontend: 1: Failed to request the resource `/pools/<pool-name>/usage` from the REST interface of the dCache `frontend` service: Client error '429 Too Many Requests' for url 'http://localhost:3880/api/v1/pools/<pool-name>/usage'
ERROR: prometheus_dcache_exporter._dcache.frontend: 2: For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429

prometheus-dcache-exporter likely exceeded the thresholds of the frontend-service’s rate limiting.

This can be configured via the frontend.limits.rate.overall- and frontend.limits.rate.per-client.fractions-options of the frontend`-service (see the example in the chapter “Usage” above).

See also Concurrent Requests To The REST Interface Respectively Command Executions Via The SSH Interface And Connection Re-Use above.

SSH Interface: Timeout opening channel., SSHException And Other Paramiko Errors

In case of log messages like:

ERROR: prometheus_dcache_exporter._dcache.admin: Failed to execute the command `\s <cell-name>@<domain-name> get hostname` via the SSH interface of the dCache `admin` service (try 2 of 2): Timeout opening channel.

despite the Timeout opening channel., the timeout that usually causes this is that configured via the --dcache-admin-ssh.command-channel-timeout-option. Increasing that or decreasing the concurrency via the --dcache-admin-ssh.max-concurrent-commands-option might help.

Ironically and inexplicably, tests have shown that, instead, it might help best to disable connection re-use of the REST interface of dCache’s frontend-service. Similarly, decreasing the concurrency of that via the --dcache-frontend-rest.max-concurrent-requests-options might help.

See also Concurrent Requests To The REST Interface Respectively Command Executions Via The SSH Interface And Connection Re-Use above.

References

• Presentation at the 20th International dCache Workshop

Licence