netbox-plugin-prometheus-sd

Provide Prometheus http_sd compatible API Endpoint with data from Netbox.

HTTP SD is a feature since Prometheus 2.28.0 that allows hosts to be found via a URL instead of just files. This plugin implements API endpoints in Netbox to make devices, services, IPs and virtual machines available to Prometheus.

Compatibility

We aim to support the latest major versions of Netbox.

Check the .github/workflows/ci.yml pipeline for the current tested builds. Other versions may work, but we do not test them explicitly. All relevant target versions are tested in CI.

Plugin 2.0 also fixed a set of N+1 queries that made the endpoints very slow on larger installations (#265). If service discovery is putting noticeable load on your Netbox database, upgrading is worthwhile.

Installation

The plugin is available as a Python package in pypi and can be installed with pip

pip install netbox-plugin-prometheus-sd

Enable the plugin in /opt/netbox/netbox/netbox/configuration.py:

    PLUGINS = ['netbox_prometheus_sd']

The plugin has not further plugin configuration.

Usage

The plugin only provides a new API endpoint on the Netbox API. There is no further action required after installation.

API

The plugin reuses Netbox API view sets with new serializers for Prometheus. This means that all filters that can be used on the Netbox API can also be used to filter Prometheus targets. Paging is disabled because Prometheus does not support paged results.

The plugin also reuses the Netbox authentication and permission model. Depending on the Netbox configuration, a token with valid object permissions must be passed to Netbox.

GET        /api/plugins/prometheus-sd/devices/              Get a list of devices in a prometheus compatible format
GET        /api/plugins/prometheus-sd/virtual-machines/     Get a list of vms in a prometheus compatible format
GET        /api/plugins/prometheus-sd/services/             Get a list of services in a prometheus compatible format
GET        /api/plugins/prometheus-sd/ip-addresses/         Get a list of ip in a prometheus compatible format

Extended services filters

Apart from standard Netbox filters, services endpoint also supports tenant=<slug> or tenant_id=<id>, and site=<slug> or site_id=<id> parameters. The lookup is only executed against the tenant/site attribute of the object associated with the service.

Filtering and response size

Because paging is disabled, one request serializes every object the token is allowed to see. Filtering with the standard Netbox query parameters is the supported way to keep responses small, and it is usually what you want anyway — Prometheus should not be told about decommissioned hardware:

/api/plugins/prometheus-sd/devices/?status=active&tag=monitoring&site=dc1

Labels

Every target carries __meta_netbox_* labels describing the Netbox object. Prometheus discards labels starting with __ after service discovery, so they have to be copied into real labels with relabel_configs (see below) if you want them on your metrics.

A label is only present when the underlying field is populated — a device with no tenant has no __meta_netbox_tenant.

¹ Netbox 4.2 replaced the cluster's site with a generic scope. On 4.2+ a VM emits scope/scope_slug for the cluster, and site/site_slug for its own site; below 4.2 the cluster's site is reported as site. ² Only when the device is assigned to a cluster.

Config context can additionally set __metrics_path__ and __scheme__, which Prometheus consumes directly (see Config context).

Relabeling

The __meta_netbox_* labels are dropped unless you map them. A typical device job scraping node_exporter on the primary IP:

scrape_configs:
  - job_name: netbox-devices
    http_sd_configs:
      - url: http://netbox:8080/api/plugins/prometheus-sd/devices/?status=active&tag=monitoring
        refresh_interval: 60s
        authorization:
          type: Token
          credentials: "<your-netbox-api-token>"

    relabel_configs:
      # Skip anything without a primary IPv4, otherwise the address below is empty.
      - source_labels: [__meta_netbox_primary_ip4]
        regex: ^$
        action: drop

      # Scrape the primary IP instead of the device name, which may not resolve.
      - source_labels: [__meta_netbox_primary_ip4]
        target_label: __address__
        replacement: "$1:9100"

      # Keep the Netbox name as the instance label rather than the IP.
      - source_labels: [__meta_netbox_name]
        target_label: instance

      # Promote the dimensions worth alerting and grouping on.
      - source_labels: [__meta_netbox_site_slug]
        target_label: site
      - source_labels: [__meta_netbox_role_slug]
        target_label: role
      - source_labels: [__meta_netbox_tenant_slug]
        target_label: tenant

Two things worth knowing:

• tags and tag_slugs are comma-joined, so match them with .*,?value,?.* rather than =.
• Devices and VMs use the same label names, so one set of relabel_configs can be reused across both jobs.

Config context

The plugin can also discover extra config to inject in the HTTP SD JSON from the config context of the devices/virtual machines. If you have a prometheus-plugin-prometheus-sd entry in your config context with the following schema it will be automatically picked up:

prometheus-plugin-prometheus-sd:
  - metrics_path: /not/metrics
    port: 4242
    scheme: https
  - port: 4243

This allow you to configure those values directly into netbox instead of doing that inside the Prometheus config and filtering each scenario by a specific tag for instance.

If there is only one entry you can also use this form:

prometheus-plugin-prometheus-sd:
  metrics_path: /not/metrics
  port: 4242
  scheme: https

Example

example/prometheus.yml is a complete Prometheus configuration covering all four endpoints, with relabel_configs mapping the __meta_netbox_* labels onto real ones.

It is not a snippet that happens to be in the repository: the test suite runs Prometheus against this exact file and asserts that the expected targets and labels are discovered, so it cannot quietly stop working.

To see it running, with a Netbox that has demo data already loaded:

poetry run invoke build-dev

• Netbox: http://localhost:8000 (admin / admin)
• Prometheus: http://localhost:9090 — discovered targets are under Status → Target health, and expanding one shows the raw __meta_netbox_* labels before relabeling.

The demo data is the unit-test fixtures, so it also exercises config context (the VMs get two targets each, on different ports), services, contacts and tags.

Development

We use Poetry for dependency management and invoke as task runner. To test the plugin in an isolated environment, we use testcontainers which creates "throwaway, lightweight" Netbox Docker containers.

Install the dependencies with poetry install, then run the tasks through poetry run (this works on every Poetry version, whereas poetry shell was moved into a separate plugin in Poetry 2.0):

# Unit tests plus the Prometheus end-to-end check
poetry run invoke test

# Test against a specific Netbox release (default: latest)
NETBOX_VER=v4.6.5 poetry run invoke test

# Either half on its own
poetry run invoke unittest
poetry run invoke test-prometheus

The Netbox image is built from the Dockerfile in the repository root.

Testing has two layers:

• Unit tests — plain Django tests under netbox_prometheus_sd/tests/, executed inside the Netbox container. These cover the serializers and the label output.
• Prometheus end-to-end — Netbox serving real HTTP with seeded data, and a real Prometheus configured from example/prometheus.yml. It asserts that the expected jobs discover targets and that relabeling produced the expected labels.

The second layer exists because "valid JSON with targets and labels" is not the same as "Prometheus accepts this as an http_sd source". A response that Prometheus rejects would pass every unit test in this repository. It also keeps the documented example honest, since it is the file under test.

Features should be covered by a test, but sometimes it is easier to develop against a running system:

# Netbox + Prometheus with demo data, left running until Ctrl+C
poetry run invoke build-dev

Netbox is on http://localhost:8000 (admin / admin) and Prometheus on http://localhost:9090, already scraping it.

API endpoints for testing can be found at http://localhost:8000/api/plugins/prometheus-sd/

Conventional Commits

This repository follows the Conventional Commits specification for versioning and changelog generation. Conventional Commits provide a standardized way of writing commit messages to convey semantic meaning about the changes made. Each commit message follows a defined format that includes a type, an optional scope, and a message. The types typically include features, fixes, documentation, and more. By adhering to this convention, we ensure clear and automated versioning, release notes, and changelog generation.