racklint

Check servers and switches against the inventory before they go into service.

racklint is a commissioning validator for bare-metal and GPU clusters. It reads what each server reports through its BMC over Redfish, and what each switch reports over gNMI with OpenConfig models, and compares that with what NetBox says should be there. Every mismatch is flagged: a wrong GPU count, a wrong serial number, firmware not on the approved list, a missing NIC, a cable that lands on the wrong switch port.

You can use it in two ways: as the racklint command, or as a Python library.

Why

When a rack is brought up, the question is not "what hardware is in it?" but "is the hardware in it what we ordered and planned for?" Existing tools such as netbox-sync and netbox-agent answer the first question: they read the hardware and write it into NetBox, so NetBox ends up describing whatever was found.

racklint works the other way. NetBox holds the planned state and stays the source of truth. racklint checks the hardware against it, changes nothing, and fails loudly on any difference. That makes it useful as a gate: a device only moves to the next stage of bring-up once racklint passes it.

It is read-only. It sends only GET requests to NetBox and to the BMCs, and only gNMI Get requests to the switches. The host OS does not need to be running.

Install

pip install racklint

To check switches as well, install the network extra, which adds the gNMI client (pygnmi):

pip install "racklint[network]"

racklint needs Python 3.12 or later. Without the extra, a plan that contains switches stops with a configuration error that names the extra to install.

Command line

Settings

The command reads its settings from RACKLINT_* environment variables, or from a .env file in the working directory.

racklint builds a reader only for the kinds of device in the plan. A run that checks only switches needs no BMC settings, and a run that checks only servers needs no switch settings.

A token that starts with nbt_ (NetBox 4.5 and later) is sent as a bearer token. Any other token is sent as a legacy Token.

Passwords from files

A password in an environment variable can leak into process listings and CI logs. Instead of RACKLINT_BMC_PASSWORD or RACKLINT_SWITCH_PASSWORD, you can set RACKLINT_BMC_PASSWORD_FILE or RACKLINT_SWITCH_PASSWORD_FILE to the path of a file that holds the password. A final newline is ignored. Setting both the password and its file is an error, and so is a file that cannot be read or is empty. The error names the file, never its content.

racklint reads the file at the start of every run. This suits a secret manager that mounts secrets as files, such as Docker or Kubernetes secrets or Vault Agent: when the manager rotates a password by rewriting the file, racklint uses the new password on its next run, and no secret sits in the environment. One set of credentials still applies to every BMC and one to every switch.

Run a check

racklint check --rack R01
racklint check --device gpu-01 --device gpu-02

Pass exactly one of --rack and --device. A run checks the servers and switches among the devices and prints one report for all of them.

--ca-bundle and --insecure apply to BMCs only. Switches have their own settings, because they usually sit in a different PKI.

The servers are read first and then the switches, each kind with up to --concurrency devices at once.

Firmware rules

The approved firmware versions live in NetBox, in the firmware list of the racklint config context, next to the rest of the plan. Assign the context to the device type, so every device of that type gets the rules:

racklint:
  firmware:
    - match: "BMC"
      approved: ["1.4.0"]
    - match: "GPU_*"
      approved: ["96.00.89.00.01"]

Each match is a case-sensitive glob on the Id of a firmware item the device reports. Versions must be quoted strings. A rule must match at least one item, and every item it matches must run an approved version. A switch reports its OS as the firmware item OS, so the rule match: "OS" covers a switch's OS version.

A device whose type has no rules fails with no firmware baseline for device type <slug>.

--baseline is an override. It names a YAML file with the approved versions for each device type, and when you give it, racklint uses the file's rules for every device and ignores the rules in the config contexts. It replaces the rules and never merges with them. Use it to check against a list kept in a repository, or to keep using a v0.1 baseline file unchanged.

device_types:
  gpu-node-h100:             # NetBox device-type slug
    - match: "BMC"
      approved: ["1.4.0"]
    - match: "GPU_*"
      approved: ["96.00.89.00.01"]

Exit codes

Use the exit code to gate a pipeline.

Output

The table has one row per device and one line per reason, then a summary:

┏━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Node ┃ Kind   ┃ Address    ┃ Status ┃ Reasons                                ┃
┡━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ n01  │ server │ 192.0.2.11 │ PASS   │                                        │
│ n02  │ server │ 192.0.2.12 │ PASS   │                                        │
│ n03  │ server │ 192.0.2.13 │ FAIL   │ gpu: 7 found, 8 planned                │
│ n04  │ server │ 192.0.2.14 │ PASS   │                                        │
│ n05  │ server │ 192.0.2.15 │ FAIL   │ memory: 31 DIMMs (1984 GiB), 32        │
│      │        │            │        │ planned (2048 GiB)                     │
│      │        │            │        │ firmware: BMC 1.3.2 not in approved    │
│      │        │            │        │ [1.4.0]                                │
│ n06  │ server │ 192.0.2.16 │ PASS   │                                        │
└──────┴────────┴────────────┴────────┴────────────────────────────────────────┘
4 passed, 2 failed, 0 errored

The kind is server or switch. The address is where racklint read the device: a server's BMC address, or a switch's management address. A device has status PASS, FAIL or ERROR. ERROR means racklint could not read the device (a connection failure after retries, a timeout, a rejected login or an invalid reply). No checks run for that device, and the reason is the error.

With --format json the output is the report as JSON. This is one failing server:

{
  "schema_version": 2,
  "results": [
    {
      "node": "n03",
      "kind": "server",
      "address": "192.0.2.13",
      "status": "fail",
      "findings": [
        {
          "check": "gpu",
          "message": "7 found, 8 planned"
        }
      ],
      "error": null
    }
  ]
}

status is pass, fail or error. error is set only when the status is error.

Python library

The names exported from racklint (its __all__) are the public API. Anything else may change before 1.0. The package ships type hints (py.typed). The names fall into these groups:

The library is async and does not read RACKLINT_* settings: you pass it explicit objects. NetBoxSource returns both kinds of device, ExpectedNode for a server and ExpectedSwitch for a switch. You split them, read each kind with its own reader and run its own checks: server_checks() for servers, switch_checks() for switches. This example reads the plan for rack R01 from NetBox, checks the servers and the switches, and prints a verdict per device. It takes its connection details from the environment.

import asyncio
import os
from pathlib import Path

from racklint import (
    BmcAccess,
    ExpectedNode,
    ExpectedSwitch,
    GnmiAccess,
    GnmiReader,
    NetBoxSource,
    RedfishReader,
    check_nodes,
    netbox_client,
    server_checks,
    switch_checks,
)


async def main() -> None:
    async with netbox_client(os.environ["NETBOX_URL"], os.environ["NETBOX_TOKEN"]) as client:
        nodes = await NetBoxSource(client).rack("R01")
    servers = [node for node in nodes if isinstance(node, ExpectedNode)]
    switches = [node for node in nodes if isinstance(node, ExpectedSwitch)]

    bmc = BmcAccess(username=os.environ["BMC_USERNAME"], password=os.environ["BMC_PASSWORD"])
    switch = GnmiAccess(
        username=os.environ["SWITCH_USERNAME"],
        password=os.environ["SWITCH_PASSWORD"],
        ca_bundle=Path("switch-ca.pem"),
    )
    server_report = await check_nodes(servers, RedfishReader(bmc), server_checks(), concurrency=16)
    switch_report = await check_nodes(switches, GnmiReader(switch), switch_checks(), concurrency=16)

    results = (*server_report.results, *switch_report.results)
    for result in results:
        print(result.node, result.kind, result.status, [finding.message for finding in result.findings])
    print("all passed" if server_report.passed and switch_report.passed else "not all passed")


asyncio.run(main())

check_nodes returns a Report with the results in the order of the devices you pass. To act on each result as soon as it is ready, for live progress, loop over iter_results instead. It takes the same arguments and yields NodeResults in the order they finish:

import asyncio
import os

from racklint import (
    BmcAccess,
    ExpectedNode,
    NetBoxSource,
    RedfishReader,
    iter_results,
    netbox_client,
    server_checks,
)


async def main() -> None:
    async with netbox_client(os.environ["NETBOX_URL"], os.environ["NETBOX_TOKEN"]) as client:
        nodes = await NetBoxSource(client).devices(["gpu-01", "gpu-02"])
    servers = [node for node in nodes if isinstance(node, ExpectedNode)]

    access = BmcAccess(username=os.environ["BMC_USERNAME"], password=os.environ["BMC_PASSWORD"])
    async for result in iter_results(servers, RedfishReader(access), server_checks(), concurrency=16):
        print(result.node, result.status)


asyncio.run(main())

NetBoxSource raises NetBoxError and FirmwareBaseline.load raises BaselineError; both are subclasses of RacklintError. A device that cannot be read does not raise: its result gets status error. Reading a switch needs the network extra, and without it GnmiReader raises RacklintError; the gNMI names are importable either way.

FirmwareBaseline.applied_to(node) returns the node with the baseline's rules for its device type in place of its planned rules, as --baseline does for every device. A FirmwareBaseline can also be built from a dict with FirmwareBaseline.model_validate, so an application can keep the baseline in its own database.

The NetBox source is optional. A NetBox plugin runs inside NetBox, so it can build ExpectedNodes and ExpectedSwitches from the Django ORM instead of the REST API, then call asyncio.run(check_nodes(...)) from a background job, which is synchronous code.

What it checks

A device passes only if every check passes. Firmware items that match no rule are ignored, and so are reported ports beyond the plan.

If NetBox lacks a value a check needs, the device fails with not planned in NetBox: <item>. A device with no address to read it at fails with that reason alone: BMC address for a server, management address for a switch.

What NetBox must hold

For a server:

Assign the config context to the device type, so every device of that type gets it:

racklint:
  gpu:    {count: 8, model: "NVIDIA H100 80GB HBM3"}
  memory: {dimm_count: 32, total_gib: 2048}
  firmware:
    - {match: "BMC", approved: ["1.4.0"]}

An invalid racklint config context is a configuration error and stops the run with exit code 2.

On the BMC side, racklint starts at /redfish/v1 and follows the links the service returns. It hardcodes no vendor paths. It reads systems, processors, memory, network adapters and the firmware inventory. It uses HTTP Basic authentication on every request and opens no Redfish sessions. It makes up to three attempts at a request that fails with a connection error, a timeout, HTTP 5xx or 429, and uses a connect timeout of 5 seconds and a read timeout of 15 seconds. A refused login is never retried, because another try with a wrong or just-rotated password could lock the account.

Switches

A switch is a device whose type has kind: switch in its config context. The default kind is server. Assign the context to the switch's device type:

racklint:
  kind: switch
  firmware:
    - {match: "OS", approved: ["v26.7.2-519-g94fbd638fa8"]}

For a switch, racklint reads these from NetBox:

It reads these from the switch, over gNMI with OpenConfig models:

The OS version is compared exactly, as firmware is: the approved string is what the switch reports, build suffix included.

racklint connects to each switch on RACKLINT_GNMI_PORT with RACKLINT_SWITCH_USERNAME and RACKLINT_SWITCH_PASSWORD. To verify the switch's TLS certificate it needs a CA bundle in RACKLINT_SWITCH_CA_BUNDLE, because pygnmi cannot use the system trust store. Set RACKLINT_SWITCH_VERIFY_TLS=false to accept any certificate instead. Requests are retried like Redfish requests, up to three attempts, so a switch that is still starting does not show as an error, but a refused login is not retried.

The cabling rule

LinkCheck compares the planned cables of a switch with what the switch sees over LLDP. It checks a planned link when the far end is a switch. Servers send no LLDP while no OS is running, so a link to a device planned as a server is left out. The far end is the interface NetBox traces the cable to, through patch panels; where NetBox has traced no path, it is the cable's own far end when that is an interface. A switch's links into another rack are checked from this switch's side, and the far switch need not be part of the run.

A link matches when the LLDP neighbour on the local port is the planned far end:

• the neighbour's system-name equals the far device's NetBox name, or its short form before the first dot (switches often report an FQDN), and
• its port-id or port-description equals the far interface's NetBox name (some vendors put a MAC address in port-id).

The findings:

link: ethernet-1/1 expected spine1 ethernet-1/1, sees spine1 ethernet-1/2
link: ethernet-1/3 expected spine1 ethernet-1/3, no LLDP neighbour
link: ethernet-1/5 not planned in NetBox, sees spine1 ethernet-1/4
link: ethernet-1/6 cannot be checked: spine2 has no racklint config context
link: ethernet-1/7 cannot be checked: its cable does not reach an interface in NetBox

A planned link is never dropped silently. If the far device has no racklint config context, so racklint does not know its kind, or the cable does not reach an interface in NetBox, the link is reported as one that cannot be checked. A port with no planned cable that sees any neighbour is a finding: the cable is there, but not in the plan. Management ports are never considered, because switches see each other on them.

NetBox interface names must match the switch's own interface names, for example ethernet-1/1.

Try it without hardware

examples/no-hardware/ is a Docker Compose stack with a real NetBox and six mock BMCs, two of them built wrong on purpose. One command checks the rack, and the summary line is 4 passed, 2 failed, 0 errored. With --profile fabric it also starts three SR Linux leaves and a spine with containerlab, and the same command checks nine devices: 5 passed, 4 failed, 0 errored. See its README for the steps and what is planted.

Limitations

• GPUs are found only as Redfish Processors with ProcessorType == "GPU". A BMC that exposes GPUs another way shows zero GPUs.
• One set of credentials applies to every BMC in a run, and one to every switch.
• Switches are checked for serial, OS version and switch-to-switch cabling. Optics, the hardware model, the cabling of BMC management ports and the cabling of servers' data ports are not checked.
• OpenConfig coverage differs by vendor and version. racklint reads the models above and is tested only against Nokia SR Linux. The reader in racklint.gnmi is the one place to extend.
• Read-only by design. racklint reports differences and never fixes them in NetBox or on a device.

Changes in 0.2

New: switches, the LinkCheck, firmware rules in NetBox's config context, and password files. --baseline is now optional, and the BMC credentials are needed only when the plan contains servers.

These changes break v0.1 users; racklint allows breaking changes before 1.0:

• JSON report: schema_version is 2. Each result's bmc_address is now address, and each result gains kind. The table gains the Kind and Address columns.
• Firmware rules: they come from NetBox first. To keep a v0.1 baseline file working, pass it with --baseline; it overrides the rules in NetBox.
• Library: default_checks(baseline) is replaced by server_checks() and switch_checks(). FirmwareCheck takes no argument and reads the rules from the node. NetBoxSource.rack() and devices() return servers and switches. NodeResult changes like the JSON.
• Messages: the serial check says device reports instead of BMC reports, because it serves switches too.

Related work

racklint is smaller than any of these and does one thing: it compares the hardware with NetBox's plan and reports the differences.

License

MIT