prometheus-dcache-exporter 0.4.0

Dependencies

• Python 3.13

• The packages/modules certifi, httpx, humanfriendly, jsonpath, module_utils, packaging, paramiko and prometheus_client.

• Optionally, the packages/modules logging_extras, rich and rich_argparse.

Getting The Source Code

Clone the prometheus-dcache-exporter git repository via either of:

$ git clone ssh://git@gitlab.com/calestyo/prometheus-dcache-exporter.git

$ git clone https://gitlab.com/calestyo/prometheus-dcache-exporter.git

Building A Debian Package

The Debian packaging is maintained in the debian-branch of the prometheus-dcache-exporter git repository.

Preparations:

$ cd prometheus-dcache-exporter
$ git switch -c temporary-build-branch master
$ temporary_version="$(python3 -m setuptools_scm --format plain)"
$ [ -z "$( git tag --list "v${temporary_version}" )" ]  &&  git tag "v${temporary_version}"
$ git merge --allow-unrelated-histories -m "Merge branch 'debian' into temporary-build-branch" debian
$ debian_changelog_version="$(dpkg-parsechangelog --show-field Version)"
$ [ "${temporary_version}"   !=   "$( printf '%s\n' "${debian_changelog_version}"  |  sed -E 's/^[0-9]+://; s/-[^-]+$//;' )" ]    &&    debchange --release-heuristic log --newversion "$( printf '%s\n' "${debian_changelog_version}"  |  sed -E -n 's/^([0-9]+:).*/\1/p' )${temporary_version}-1" 'Local version.'
$ git deborig

Any changes since the tip of the master-branch must be committed right after switching to the temporary-build-branch-branch.

Build via either of:

$ dpkg-buildpackage

$ debuild

optionally adding the --no-sign-option if signing is not desired.

Beware that the above creates files in the parent directory.

git-deborig is part of the git-debpush-package, dpkg-buildpackage and dpkg-parsechangelog are part of the dpkg-dev-package and debuild is part of the devscripts-package.

Installation From The Python Package Index (PyPI)

$ pip install prometheus-dcache-exporter

Collection

prometheus-dcache-exporter uses the REST interface of dCache’s frontend-service and the SSH interface of dCache’s admin-service for collecting the metrics. The latter is only used for raw data that isn’t available via the former, yet.

Connections to the REST and SSH interfaces are tried to be kept open and re-used, as configured via the --dcache-frontend-rest.connection-pool-*-family of options respectively the --dcache-admin-ssh.keepalive-interval-option.

Per default, the requests to the REST interface respectively the command executions via the SSH interface are performed concurrently (using multithreading), as configured via the --dcache-frontend-rest.max-concurrent-requests- and --dcache-admin-ssh.max-concurrent-commands-options. This is done, amongst other reasons, so that the raw data is as closely synchronised as possible.

Extra Labels

In general, labels describe the entity to which a time series of a metric corresponds. For example, in:

# HELP dcache_space_reservation_space_total_bytes The total space of a space reservation in bytes.
# TYPE dcache_space_reservation_space_total_bytes gauge
dcache_space_reservation_space_total_bytes{space_reservation_id="110000"} 4748869032235006
dcache_space_reservation_space_total_bytes{space_reservation_id="380987"} 292706065309706

the label space_reservation_id uniquely identifies the entity (here, a space reservation), thereby effectively being a “primary key”.

There may also be labels that are not necessary for unique identification, which is particularly (but not only) the case for “info metrics” (whose names typically end in _info). For example, in:

# HELP dcache_space_reservation_info Information about a space reservation.
# TYPE dcache_space_reservation_info gauge
dcache_space_reservation_info{description="ATLASDATADISK",space_reservation_id="110000"} 1.0
dcache_space_reservation_info{description="ATLASLOCALGROUPDISK",space_reservation_id="380987"} 1.0

only the label space_reservation_id identifies the corresponding space reservation uniquely, while the label description – albeit often uniquely used by dCache operators – is strictly speaking not ensured to be unique and is not necessary for identifying the corresponding space reservation.

Per default and except for info metrics, prometheus-dcache-exporter generally includes only the minimal set of labels that are necessary to uniquely identify the corresponding entity.

Sometimes it may however be convenient to have additional labels included. For example, so that the above metric looks like:

# HELP dcache_space_reservation_space_total_bytes The total space of a space reservation in bytes.
# TYPE dcache_space_reservation_space_total_bytes gauge
dcache_space_reservation_space_total_bytes{description="ATLASDATADISK",space_reservation_id="110000"} 4748869032235006
dcache_space_reservation_space_total_bytes{description="ATLASLOCALGROUPDISK",space_reservation_id="380987"} 292706065309706

This can be achieved – for certain metrics and labels where this seemed reasonable – using the --metrics.extra-label-option.

Label Renaming

The --metrics.rename-label-option allows to rename most labels, which may make sense in some special use cases.

It shall further be noted, that renaming labels may impact compatibility with anything (like dashboards) that assumes the built-in names.

REST Interface: HTTP Status Code 429 (Too Many Requests)

In case of log messages like:

ERROR: prometheus_dcache_exporter._dcache.frontend: 1: Failed to request the resource `/pools/<pool-name>/usage` from the REST interface of the dCache `frontend` service: Client error '429 Too Many Requests' for url 'http://localhost:3880/api/v1/pools/<pool-name>/usage'
ERROR: prometheus_dcache_exporter._dcache.frontend: 2: For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429

prometheus-dcache-exporter likely exceeded the thresholds of the frontend-service’s rate limiting.

This can be configured via the frontend.limits.rate.overall- and frontend.limits.rate.per-client.fractions-options of the frontend-service (see the example in the chapter “Usage” above).

See also Concurrent Requests To The REST Interface Respectively Command Executions Via The SSH Interface And Connection Re-Use above.

SSH Interface: Timeout opening channel., SSHException And Other Paramiko Errors

In case of log messages like:

ERROR: prometheus_dcache_exporter._dcache.admin: Failed to execute the command `\s <cell-name>@<domain-name> get hostname` via the SSH interface of the dCache `admin` service (try 2 of 2): Timeout opening channel.

despite the Timeout opening channel., the timeout that usually causes this is that configured via the --dcache-admin-ssh.command-channel-timeout-option. Increasing that or decreasing the concurrency via the --dcache-admin-ssh.max-concurrent-commands-option might help.

Ironically and inexplicably, tests have shown that, instead, it might help best to disable connection re-use of the REST interface of dCache’s frontend-service. Similarly, decreasing the concurrency of that via the --dcache-frontend-rest.max-concurrent-requests-options might help.

See also Concurrent Requests To The REST Interface Respectively Command Executions Via The SSH Interface And Connection Re-Use above.

High Memory Usage / Seeming Memory Leak

If prometheus-dcache-exporter has an unexpected high memory usage, which may even seem to grow and indicate a memory leak, the reason might be bad interplay between Python’s threading and the GNU C Library’s memory allocation/de-allocation. In that case, there’s actually no memory leak and the memory usage should eventually stabilise.

If the high memory usage is a problem, either of the following workarounds might help:

• Setting the M_ARENA_MAX-environment-variable to the value 1.

• Replacing the GNU C Library’s memory allocation/de-allocation with another, for example via setting the LD_PRELOAD-environment-variable to the ELF shared object that contains the replacement (like libjemalloc.so.2).

  Alternatives include jemalloc, mimalloc, tbbmalloc[1], and tcmalloc[2], but the results may depend on their respective version and/or that of the Python interpreter as well as on other factors.

• Not using the http-export-mode, which may however have other disadvantages.

Prometheus “Ignores” Samples/Metrics

If Prometheus’ scrapes fail while “manual” ones succeed, the exporter’s output might be larger than a limit set via Prometheus’ body_size_limit-option.