racklint

Check servers and switches against the inventory before they go into service.

racklint is a commissioning validator for bare-metal and GPU clusters. It reads what each server reports through its BMC over Redfish, and what each switch reports over gNMI with OpenConfig models, and compares that with what NetBox says should be there. Every mismatch is flagged: a wrong GPU count, a wrong serial number, firmware not on the approved list, a missing NIC, a cable that lands on the wrong switch port.

You can use it in two ways: as the racklint command, or as a Python library.

Why

When a rack is brought up, the question is not "what hardware is in it?" but "is the hardware in it what we ordered and planned for?" Existing tools such as netbox-sync and netbox-agent answer the first question: they read the hardware and write it into NetBox, so NetBox ends up describing whatever was found.

racklint works the other way. NetBox holds the planned state and stays the source of truth. racklint checks the hardware against it, changes nothing, and fails loudly on any difference. That makes it useful as a gate: a device only moves to the next stage of bring-up once racklint passes it.

It is read-only. It sends only GET requests to NetBox and to the BMCs, and only gNMI Get requests to the switches. The one exception is logging in: a BMC session is opened with a POST and deleted when the read ends, and an AppRole login to Vault is a POST. The host OS does not need to be running.

Install

pip install racklint

To check switches as well, install the network extra, which adds the gNMI client (pygnmi):

pip install "racklint[network]"

racklint needs Python 3.12 or later. Without the extra, a plan that contains switches stops with a configuration error that names the extra to install.

Command line

Settings

The command reads its settings from RACKLINT_* environment variables, or from a .env file in the working directory.

racklint builds a reader only for the kinds of device in the plan. A run that checks only switches needs no BMC settings, and a run that checks only servers needs no switch settings. The BMC and switch settings are one of three ways to give racklint its credentials, and they set a single login for all BMCs and one for all switches. Credentials describes the other two.

A token that starts with nbt_ (NetBox 4.5 and later) is sent as a bearer token. Any other token is sent as a legacy Token.

Passwords from files

A password in an environment variable can leak into process listings and CI logs. Instead of RACKLINT_BMC_PASSWORD or RACKLINT_SWITCH_PASSWORD, you can set RACKLINT_BMC_PASSWORD_FILE or RACKLINT_SWITCH_PASSWORD_FILE to the path of a file that holds the password. A final newline is ignored. Setting both the password and its file is an error, and so is a file that cannot be read or is empty. The error names the file, never its content.

racklint reads the file at the start of every run. This suits a secret manager that mounts secrets as files, such as Docker or Kubernetes secrets or Vault Agent: when the manager rotates a password by rewriting the file, racklint uses the new password on its next run, and no secret sits in the environment.

Run a check

racklint check --rack R01
racklint check --device gpu-01 --device gpu-02

Pass exactly one of --rack and --device. A run checks the servers and switches among the devices and prints one report for all of them.

--ca-bundle and --insecure apply to BMCs only. Switches have their own settings, because they usually sit in a different PKI.

The servers are read first and then the switches, each kind with up to --concurrency devices at once.

Firmware rules

The approved firmware versions live in NetBox, in the firmware list of the racklint config context, next to the rest of the plan. Assign the context to the device type, so every device of that type gets the rules:

racklint:
  firmware:
    - match: "BMC"
      approved: ["1.4.0"]
    - match: "GPU_*"
      approved: ["96.00.89.00.01"]

Each match is a case-sensitive glob on the Id of a firmware item the device reports. Versions must be quoted strings. A rule must match at least one item, and every item it matches must run an approved version. A switch reports its OS as the firmware item OS, so the rule match: "OS" covers a switch's OS version.

A device whose type has no rules fails with no firmware baseline for device type <slug>.

--baseline is an override. It names a YAML file with the approved versions for each device type, and when you give it, racklint uses the file's rules for every device and ignores the rules in the config contexts. It replaces the rules and never merges with them. Use it to check against a list kept in a repository, or to keep using a v0.1 baseline file unchanged.

device_types:
  gpu-node-h100:             # NetBox device-type slug
    - match: "BMC"
      approved: ["1.4.0"]
    - match: "GPU_*"
      approved: ["96.00.89.00.01"]

Exit codes

Use the exit code to gate a pipeline.

Output

The table has one row per device and one line per reason, then a summary:

┏━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Node ┃ Kind   ┃ Address    ┃ Status ┃ Reasons                                ┃
┡━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ n01  │ server │ 192.0.2.11 │ PASS   │                                        │
│ n02  │ server │ 192.0.2.12 │ PASS   │                                        │
│ n03  │ server │ 192.0.2.13 │ FAIL   │ gpu: 7 found, 8 planned                │
│ n04  │ server │ 192.0.2.14 │ ERROR  │ no credentials for n04: set bmc-r02    │
│      │        │            │        │ not found in Vault                     │
│ n05  │ server │ 192.0.2.15 │ FAIL   │ memory: 31 DIMMs (1984 GiB), 32        │
│      │        │            │        │ planned (2048 GiB)                     │
│      │        │            │        │ firmware: BMC 1.3.2 not in approved    │
│      │        │            │        │ [1.4.0]                                │
│ n06  │ server │ 192.0.2.16 │ PASS   │                                        │
└──────┴────────┴────────────┴────────┴────────────────────────────────────────┘
3 passed, 2 failed, 1 errored

The kind is server or switch. The address is where racklint read the device: a server's BMC address, or a switch's management address. A device has status PASS, FAIL or ERROR. ERROR means racklint could not read the device (a connection failure after retries, a timeout, a rejected login, no credentials for the device or an invalid reply). No checks run for that device, and the reason is the error.

With --format json the output is the report as JSON. This is one failing server:

{
  "schema_version": 2,
  "results": [
    {
      "node": "n03",
      "kind": "server",
      "address": "192.0.2.13",
      "status": "fail",
      "findings": [
        {
          "check": "gpu",
          "message": "7 found, 8 planned"
        }
      ],
      "error": null
    }
  ]
}

status is pass, fail or error. error is set only when the status is error.

Credentials

Each device is read with its own credentials. racklint takes the first of these that exists, and uses only that one:

1. A factory secret for the device's serial number. A new BMC often still has the password it left the factory with, one per unit. Store it under the device's serial and racklint logs in with it.

2. The credential set the device's config context names. In NetBox, put the name in the racklint key of a config context, which the site, rack, role or device type can hold like any config context, or in the device's own local context:

   racklint:
     credentials: bmc-dc1
   

   The name is made of letters, digits, ., _ and -, and starts with a letter or digit. NetBox holds the name of a set, never a password.

3. The default set for the device's kind: default-server for a server, default-switch for a switch.

A device is logged in to once. If the credentials found are refused, racklint reports the device as an error, names where they came from, and does not try another set: a BMC or switch that counts failed logins could lock the account. A refused factory secret adds that it should be removed once the device's password is rotated. The error for a device whose set does not exist reads no credentials for n04: set bmc-r02 not found in Vault. The other devices are read as usual.

Where the credentials come from

A run uses exactly one source:

Giving more than one source is an error, and so is giving the BMC or switch settings with Vault or a credentials file, because they would not be used. Vault settings without RACKLINT_VAULT_ADDR are an error too. racklint checks this before it reads NetBox, and exits with code 2. A setup that uses only the BMC and switch settings works as in 0.2: it has no factory secrets and no named sets, so every server and switch is read with the one login, and a device whose config context names a set is an error, because that set does not exist.

Each set and each serial is looked up once per run, and kept in memory for that run only, as a secret that prints as **********.

The credentials file

RACKLINT_CREDENTIALS_FILE names a YAML file:

sets:
  default-server: {username: admin, password_file: /run/secrets/bmc}
  bmc-dc1:        {username: admin, password_env: BMC_DC1_PASSWORD}
  default-switch: {username: admin, password_file: /run/secrets/switch}
factory:                       # by serial
  SN-0004: {username: root, password_file: /run/secrets/factory/SN-0004}

Each entry takes its password from password_file or password_env, exactly one of them. A password written into the file is an error, so the file holds no secrets and can live in git. Password files are read as described under Passwords from files, at the start of the run. An unknown key, an unset variable, an empty password file and a set name that is not a safe path segment are errors, and none of the messages shows a password.

Vault

racklint reads secrets from Vault's KV version 2 engine over its HTTP API, with no extra dependency. Each secret holds the keys username and password:

The mount is secret and the prefix racklint unless RACKLINT_VAULT_MOUNT and RACKLINT_VAULT_PREFIX say otherwise. Write one with, for example, vault kv put secret/racklint/sets/bmc-dc1 username=admin password=.... A missing secret is not an error for a factory secret: racklint moves on to the device's set. A missing set, and any other answer from Vault, is that device's error; the message names the path and the status, never a token.

racklint logs in to Vault in one of two ways, and exactly one must be given:

• a token, in RACKLINT_VAULT_TOKEN or in the file RACKLINT_VAULT_TOKEN_FILE names; or
• an AppRole, with RACKLINT_VAULT_ROLE_ID and the file RACKLINT_VAULT_SECRET_ID_FILE names. racklint logs in once per run.

Files are read at the start of the run. Prefer a token file, filled by Vault Agent, to a token in the environment, which can leak into process listings and CI logs. Give racklint a policy that can read only <mount>/data/<prefix>/*.

Redfish sessions

A BMC is logged in to once per read, not on every request. racklint reads the service root without credentials, as Redfish allows. If the root links a session collection, racklint opens one session, sends its token with every request, and deletes the session when the read ends, also when the read fails. If a delete fails, the BMC's session timeout removes the session. A BMC without a session service is read with Basic authentication, as in 0.2. One that refuses the anonymous service root is asked again with Basic authentication. A refused login (HTTP 401, or 403, which racklint reports as access denied (HTTP 403)) is final. Connection errors, timeouts, 429 and 5xx on a login are retried like any request.

Python library

The names exported from racklint (its __all__) are the public API. Anything else may change before 1.0. The package ships type hints (py.typed). The names fall into these groups:

The library is async and does not read RACKLINT_* settings: you pass it explicit objects. NetBoxSource returns both kinds of device, ExpectedNode for a server and ExpectedSwitch for a switch. You split them, read each kind with its own reader and run its own checks: server_checks() for servers, switch_checks() for switches. A CredentialSource gives each device its credentials; CredentialLookup is the one that applies the lookup order to a CredentialStore: a StaticStore that you fill, a VaultStore, or your own class. This example reads the plan for rack R01 from NetBox, checks the servers and the switches, and prints a verdict per device. It takes its connection details from the environment.

import asyncio
import os
from pathlib import Path

from pydantic import SecretStr

from racklint import (
    BmcAccess,
    CredentialLookup,
    Credentials,
    ExpectedNode,
    ExpectedSwitch,
    GnmiAccess,
    GnmiReader,
    NetBoxSource,
    RedfishReader,
    StaticStore,
    check_nodes,
    netbox_client,
    server_checks,
    switch_checks,
)


async def main() -> None:
    async with netbox_client(os.environ["NETBOX_URL"], os.environ["NETBOX_TOKEN"]) as client:
        nodes = await NetBoxSource(client).rack("R01")
    servers = [node for node in nodes if isinstance(node, ExpectedNode)]
    switches = [node for node in nodes if isinstance(node, ExpectedSwitch)]

    credentials = CredentialLookup(
        StaticStore(
            {
                "default-server": Credentials(
                    username=os.environ["BMC_USERNAME"],
                    password=SecretStr(os.environ["BMC_PASSWORD"]),
                    origin="set default-server",
                ),
                "default-switch": Credentials(
                    username=os.environ["SWITCH_USERNAME"],
                    password=SecretStr(os.environ["SWITCH_PASSWORD"]),
                    origin="set default-switch",
                ),
            }
        )
    )
    bmc = BmcAccess()
    switch = GnmiAccess(ca_bundle=Path("switch-ca.pem"))
    server_report = await check_nodes(
        servers, RedfishReader(bmc), server_checks(), credentials, concurrency=16
    )
    switch_report = await check_nodes(
        switches, GnmiReader(switch), switch_checks(), credentials, concurrency=16
    )

    results = (*server_report.results, *switch_report.results)
    for result in results:
        print(result.node, result.kind, result.status, [finding.message for finding in result.findings])
    print("all passed" if server_report.passed and switch_report.passed else "not all passed")


asyncio.run(main())

check_nodes returns a Report with the results in the order of the devices you pass. To act on each result as soon as it is ready, for live progress, loop over iter_results instead. It takes the same arguments and yields NodeResults in the order they finish:

import asyncio
import os

from pydantic import SecretStr

from racklint import (
    BmcAccess,
    CredentialLookup,
    Credentials,
    ExpectedNode,
    NetBoxSource,
    RedfishReader,
    StaticStore,
    iter_results,
    netbox_client,
    server_checks,
)


async def main() -> None:
    async with netbox_client(os.environ["NETBOX_URL"], os.environ["NETBOX_TOKEN"]) as client:
        nodes = await NetBoxSource(client).devices(["gpu-01", "gpu-02"])
    servers = [node for node in nodes if isinstance(node, ExpectedNode)]

    login = Credentials(
        username=os.environ["BMC_USERNAME"],
        password=SecretStr(os.environ["BMC_PASSWORD"]),
        origin="set default-server",
    )
    credentials = CredentialLookup(StaticStore({"default-server": login}))
    async for result in iter_results(
        servers, RedfishReader(BmcAccess()), server_checks(), credentials, concurrency=16
    ):
        print(result.node, result.status)


asyncio.run(main())

NetBoxSource raises NetBoxError and FirmwareBaseline.load raises BaselineError; both are subclasses of RacklintError. A device that cannot be read, or has no credentials, does not raise: its result gets status error; a refused login is an AuthenticationError inside the reader and is reported the same way. Reading a switch needs the network extra, and without it GnmiReader raises RacklintError; the gNMI names are importable either way.

FirmwareBaseline.applied_to(node) returns the node with the baseline's rules for its device type in place of its planned rules, as --baseline does for every device. A FirmwareBaseline can also be built from a dict with FirmwareBaseline.model_validate, so an application can keep the baseline in its own database.

GpuCheck finds a server's GPUs with GpuRules: GpuCheck(rules) takes any sequence of objects with a find(actual, planned) method that returns the Gpus found in an ActualNode, and server_checks() uses GPU_RULES. To teach racklint another way a BMC exposes GPUs, add a rule after the built-in ones:

from racklint import GPU_RULES, ActualNode, Gpu, GpuCheck, GpuSpec


class VendorGpus:
    """A BMC that reports its GPUs as processors of type OEM with the planned model."""

    def find(self, actual: ActualNode, planned: GpuSpec) -> tuple[Gpu, ...]:
        return tuple(
            Gpu(id=processor.id, model=processor.model)
            for processor in actual.processors
            if processor.type == "OEM" and processor.model == planned.model
        )


check = GpuCheck([*GPU_RULES, VendorGpus()])

The NetBox source is optional. A NetBox plugin runs inside NetBox, so it can build ExpectedNodes and ExpectedSwitches from the Django ORM instead of the REST API, then call asyncio.run(check_nodes(...)) from a background job, which is synchronous code.

What it checks

A device passes only if every check passes. Firmware items that match no rule are ignored, and so are reported ports beyond the plan.

If NetBox lacks a value a check needs, the device fails with not planned in NetBox: <item>. A device with no address to read it at fails with that reason alone: BMC address for a server, management address for a switch.

What NetBox must hold

For a server:

Assign the config context to the device type, so every device of that type gets it:

racklint:
  gpu:    {count: 8, model: "NVIDIA H100 80GB HBM3"}
  memory: {dimm_count: 32, total_gib: 2048}
  firmware:
    - {match: "BMC", approved: ["1.4.0"]}

An invalid racklint config context is a configuration error and stops the run with exit code 2.

On the BMC side, racklint starts at /redfish/v1 and follows the links the service returns. It hardcodes no vendor paths. It reads systems, processors, memory, network adapters and the firmware inventory. GPUs are found by rules, tried in order: Processors of type GPU (the DMTF standard), then Processors of type Accelerator with the planned GPU model (upstream OpenBMC). The first rule that finds any decides. It logs in once per BMC with a Redfish session, sends the session's X-Auth-Token with every request and logs out when the read ends; a BMC without a session service is read with HTTP Basic authentication. It makes up to three attempts at a request that fails with a connection error, a timeout, HTTP 5xx or 429, and uses a connect timeout of 5 seconds and a read timeout of 15 seconds. A refused login is never retried, because another try with a wrong or just-rotated password could lock the account.

Switches

A switch is a device whose type has kind: switch in its config context. The default kind is server. Assign the context to the switch's device type:

racklint:
  kind: switch
  firmware:
    - {match: "OS", approved: ["v26.7.2-519-g94fbd638fa8"]}

For a switch, racklint reads these from NetBox:

It reads these from the switch, over gNMI with OpenConfig models:

The OS version is compared exactly, as firmware is: the approved string is what the switch reports, build suffix included.

racklint connects to each switch on RACKLINT_GNMI_PORT with RACKLINT_SWITCH_USERNAME and RACKLINT_SWITCH_PASSWORD. To verify the switch's TLS certificate it needs a CA bundle in RACKLINT_SWITCH_CA_BUNDLE, because pygnmi cannot use the system trust store. Set RACKLINT_SWITCH_VERIFY_TLS=false to accept any certificate instead. Requests are retried like Redfish requests, up to three attempts, so a switch that is still starting does not show as an error, but a refused login is not retried.

The cabling rule

LinkCheck compares the planned cables of a switch with what the switch sees over LLDP. It checks a planned link when the far end is a switch. Servers send no LLDP while no OS is running, so a link to a device planned as a server is left out. The far end is the interface NetBox traces the cable to, through patch panels; where NetBox has traced no path, it is the cable's own far end when that is an interface. A switch's links into another rack are checked from this switch's side, and the far switch need not be part of the run.

A link matches when the LLDP neighbour on the local port is the planned far end:

• the neighbour's system-name equals the far device's NetBox name, or its short form before the first dot (switches often report an FQDN), and
• its port-id or port-description equals the far interface's NetBox name (some vendors put a MAC address in port-id).

The findings:

link: ethernet-1/1 expected spine1 ethernet-1/1, sees spine1 ethernet-1/2
link: ethernet-1/3 expected spine1 ethernet-1/3, no LLDP neighbour
link: ethernet-1/5 not planned in NetBox, sees spine1 ethernet-1/4
link: ethernet-1/6 cannot be checked: spine2 has no racklint config context
link: ethernet-1/7 cannot be checked: its cable does not reach an interface in NetBox

A planned link is never dropped silently. If the far device has no racklint config context, so racklint does not know its kind, or the cable does not reach an interface in NetBox, the link is reported as one that cannot be checked. A port with no planned cable that sees any neighbour is a finding: the cable is there, but not in the plan. Management ports are never considered, because switches see each other on them.

NetBox interface names must match the switch's own interface names, for example ethernet-1/1.

Try it without hardware

examples/no-hardware/ is a Docker Compose stack with a real NetBox and six mock BMCs, two of them built wrong on purpose and one whose credential set is not in the stack's Vault. racklint reads every credential from that Vault. One command checks the rack, and the summary line is 3 passed, 2 failed, 1 errored. With --profile fabric it also starts three SR Linux leaves and a spine with containerlab, and the same command checks nine devices: 4 passed, 4 failed, 1 errored. See its README for the steps and what is planted.

Limitations

• GPUs are found in Redfish Processors only. A BMC that lists GPUs only under PCIeDevices shows zero GPUs; the finding then names the processors the BMC did list, and a library user can add a GpuRule.
• Credentials come from one source per run. Vault support is KV version 2 with a token or an AppRole login; for another secret manager, write a CredentialStore. The BMC and switch settings give one login to every BMC and one to every switch.
• Switches are checked for serial, OS version and switch-to-switch cabling. Optics, the hardware model, the cabling of BMC management ports and the cabling of servers' data ports are not checked.
• OpenConfig coverage differs by vendor and version. racklint reads the models above and is tested only against Nokia SR Linux. The reader in racklint.gnmi is the one place to extend.
• Read-only by design. racklint reports differences and never fixes them in NetBox or on a device.

Changes in 0.3

New: credentials per device (a factory secret by serial, a named set, or a default set), read from Vault, a credentials file or the 0.2 settings, and one Redfish session per BMC instead of a password on every request. A setup that uses only the BMC and switch settings works unchanged.

These changes break users of the library; racklint allows breaking changes before 1.0. The JSON report and the command line output do not change. See the changelog for the list.

Changes in 0.2

New: switches, the LinkCheck, firmware rules in NetBox's config context, and password files. --baseline is now optional, and the BMC credentials are needed only when the plan contains servers.

These changes break v0.1 users; racklint allows breaking changes before 1.0:

• JSON report: schema_version is 2. Each result's bmc_address is now address, and each result gains kind. The table gains the Kind and Address columns.
• Firmware rules: they come from NetBox first. To keep a v0.1 baseline file working, pass it with --baseline; it overrides the rules in NetBox.
• Library: default_checks(baseline) is replaced by server_checks() and switch_checks(). FirmwareCheck takes no argument and reads the rules from the node. NetBoxSource.rack() and devices() return servers and switches. NodeResult changes like the JSON.
• Messages: the serial check says device reports instead of BMC reports, because it serves switches too.

Related work

racklint is smaller than any of these and does one thing: it compares the hardware with NetBox's plan and reports the differences.

License

MIT