dcgm-exporter-lite

A pip-installable, unprivileged Python exporter that serves NVIDIA GPU metrics with DCGM Exporter 4.9.0 metric names, types, units, and labels.

It reads NVML through nvidia-ml-py and serves Prometheus text through prometheus-client. It needs no root, no DCGM install, no container, and no CDI hook. The intended use is a bridge: show the operational value of GPU metrics now, then swap in the real DCGM Exporter without changing scrape configs, dashboards, or alerts.

Requirements

• NVIDIA driver 580 or newer (older drivers run with a warning; some metrics are missing).
• Python 3.12–3.14.
• Linux with NVML available to the running user. /dev/nvidia* is normally world-readable.

Install

From PyPI, into a virtual environment as an ordinary user:

python3 -m venv .venv
.venv/bin/pip install dcgm-exporter-lite

# or with uv
uv tool install dcgm-exporter-lite

From a checkout:

python3 -m venv .venv
.venv/bin/pip install .

Quick start

.venv/bin/dcgm-exporter-lite -a :9400 -c 30000
curl -s http://localhost:9400/health          # OK once the first collection succeeds
curl -s http://localhost:9400/metrics

/metrics serializes the latest immutable snapshot; a scrape never calls NVML directly. /health returns 503 before the first successful collection, 200 afterwards, and 503 again if no fresh collection has completed within three collect intervals (minimum 10 s), e.g. when an NVML call hangs.

Flags

--devices accepts f (all GPUs) or g:0,2. MIG selectors (i:) are rejected. Flags beat environment variables; environment variables beat defaults. Other DCGM Exporter flags (--kubernetes, --config-file, --web-config-file, --remote-hostengine-info, …) exit with a message instead of being silently ignored.

Metric coverage

Verified on an NVIDIA GeForce RTX 5060 Ti (driver 615.71.10, non-root). Datacenter-only metrics are implemented and unit-tested against the fake NVML; they are unverified on hardware until the checklist in docs/testing/datacenter-verification.md is run on an A100/H100/B200 host.

• ✅ verified on hardware
• ◐ present on hardware with in-range idle values; not yet observed under a real event or load
• 🖥 datacenter-only, unverified on hardware
• ❌ omitted

The packaged default is the 4.9.0 default-counters.csv (27 rows). A custom -f file may add other rows; names this exporter does not implement are logged once and ignored.

GPM and the PROF_* metrics

GPM is NVIDIA’s NVML interface to the same hardware activity counters. It exists on Hopper and newer (including Blackwell GeForce). The exporter takes two samples and computes metrics over the interval, so the first collection cycle has no PROF_* values. This differs from DCGM’s DCP sampling, and is the one place where values can differ from the real exporter. On Volta and Ampere these metrics are absent.

Labels

Every GPU metric carries, in order:

gpu, UUID, pci_bus_id, device, modelName, hostname, DCGM_FI_SYSTEM_DRIVER_VERSION

XID samples append err_code and err_msg. --no-hostname omits hostname. --replace-blanks-in-model-name collapses whitespace and replaces spaces with -, matching 4.9.0.

MIG

For a GPU with MIG enabled, only GPU-wide metrics are exported once per physical GPU, with no MIG labels: clocks, temperatures, power, energy, PCIe replay, framebuffer, row remap, and XID. Utilization ratios and all PROF_* metrics are omitted for that GPU, with one INFO log line. MIG instance labels and i: device selectors are out of scope.

Prometheus

scrape_configs:
  - job_name: dcgm-exporter-lite
    scrape_interval: 15s
    static_configs:
      - targets: ["gpu-host:9400"]

Metric and label names match DCGM Exporter 4.9.0. /metrics does not expose Python or process metrics.

Grafana

Import grafana/dcgm-exporter-lite-dashboard.json in Grafana (Dashboards → New → Import), then pick your Prometheus data source. The dashboard uses only 4.9.0 metric names and the instance/gpu labels, so it also works against the real DCGM Exporter. Datacenter-only panels show “Not reported by this GPU” on cards that lack the source.

For comparison, NVIDIA’s 4.9.0 dashboard is at https://github.com/NVIDIA/dcgm-exporter/tree/4.9.0/grafana.

systemd user service

# The unit expects the entry point at ~/.local/bin. If you installed into a
# venv, link it there first:
mkdir -p ~/.local/bin
ln -sf /path/to/.venv/bin/dcgm-exporter-lite ~/.local/bin/dcgm-exporter-lite

mkdir -p ~/.config/systemd/user
cp contrib/systemd/dcgm-exporter-lite.service ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now dcgm-exporter-lite
loginctl enable-linger "$USER"

The unit runs %h/.local/bin/dcgm-exporter-lite and restarts on failure. Logs go to the user journal (journalctl --user -u dcgm-exporter-lite).

Swapping to DCGM Exporter

1. Keep the same listen port and /metrics path.
2. Stop the lite service: systemctl --user disable --now dcgm-exporter-lite
3. Start DCGM Exporter 4.9.0 on the same address (container or package, with its usual privileges).
4. Confirm Prometheus reports the target UP and the job still has its historical series.
5. Confirm the dashboard and alert rules need no edits. They should not: the metric names, types, units, and labels are the same.
6. Verify the driver-version label, pci_bus_id, and the XID err_code/err_msg labels. The lite exporter emits them in 4.9.0 form.
7. Datacenter-only series (memory temperature, NVLink, row remap) start appearing after the swap; the dashboard panels already render them.
8. If you use MIG, expect DCGM to add GPU_I_ID/GPU_I_PROFILE labels. The lite dashboard does not select on them, so it keeps working.

Differences from DCGM Exporter

• No MIG instance metrics or labels. MIG-enabled GPUs export only GPU-wide metrics.
• No Kubernetes, HPC-job, NVSwitch, CPU, or per-link NVLink metrics/labels.
• PROF_* uses GPM, not DCP. Hopper+ only; two-sample rates; first cycle has no values.
• No DCGM health, policy, clock-event, or DCGM_EXP_* metrics.
• No YAML config, ConfigMap source, TLS/web config, systemd socket activation, or container image.
• Minimal --devices grammar (f or g:0,2); MIG selectors are rejected.
• Unsupported DCGM flags exit non-zero with a README pointer instead of being ignored.
• /health returns 503 before the first successful collection, and when collection goes stale. 4.9.0 returns 200 earlier.
• Logging content/format differs, though --log-format json produces one JSON object per line.

Development

uv sync
uv run ruff check
uv run ruff format --check
uv run pytest

Hardware tests are marked hardware and excluded by default:

scripts/smoketest-remote.sh ryzen.local

Datacenter-only metrics need the manual checklist in docs/testing/datacenter-verification.md. The verified 4.9.0 wire contract is recorded in docs/notes/dcgm-4.9.0-contract.md.

Releasing

The package version comes from git tags (hatch-vcs); untagged builds get a development version such as 0.1.dev17+g1a2b3c4. To release, tag a commit on main and push the tag:

git tag v0.1.0
git push origin v0.1.0

.github/workflows/publish.yml then runs the test suite, refuses tags whose commit is not on main, builds, checks that the built version matches the tag and that the counters CSV and both license files are packaged, and publishes to PyPI by trusted publishing from the pypi environment.

License

This project is MIT licensed (see LICENSE).

Two pieces are copied from NVIDIA DCGM Exporter 4.9.0 and remain under the Apache License 2.0 (see LICENSES/Apache-2.0.txt):

• src/dcgm_exporter_lite/data/default-counters.csv, from etc/default-counters.csv
• the XID_MESSAGES table in src/dcgm_exporter_lite/xid.py, from internal/pkg/collector/xid_errors.go

The distribution's license expression is therefore MIT AND Apache-2.0.