racklint

Check servers against the inventory before they go into service.

racklint is a commissioning validator for bare-metal and GPU clusters. It reads what each server reports through its BMC over Redfish and compares that with what NetBox says should be there. Every mismatch is flagged: a wrong GPU count, a wrong serial number, firmware not on the approved baseline, a missing NIC.

You can use it in two ways: as the racklint command, or as a Python library.

Why

When a rack is brought up, the question is not "what hardware is in it?" but "is the hardware in it what we ordered and planned for?" Existing tools such as netbox-sync and netbox-agent answer the first question: they read the hardware and write it into NetBox, so NetBox ends up describing whatever was found.

racklint works the other way. NetBox holds the planned state and stays the source of truth. racklint checks the hardware against it, changes nothing, and fails loudly on any difference. That makes it useful as a gate: a node only moves to the next stage of bring-up once racklint passes it.

It is read-only. It sends only GET requests to NetBox and to the BMCs. The host OS does not need to be running.

Install

pip install racklint

racklint needs Python 3.12 or later.

Command line

Settings

The command reads its settings from RACKLINT_* environment variables, or from a .env file in the working directory.

A token that starts with nbt_ (NetBox 4.5 and later) is sent as a bearer token. Any other token is sent as a legacy Token.

Run a check

racklint check --rack R01 --baseline baseline.yaml
racklint check --device gpu-01 --device gpu-02 --baseline baseline.yaml

Pass exactly one of --rack and --device.

The baseline is a YAML file with the approved firmware versions for each NetBox device type. Each match is a case-sensitive glob on the Id of a member of the BMC's firmware inventory. Versions must be quoted strings.

device_types:
  gpu-node-h100:             # NetBox device-type slug
    - match: "BMC"
      approved: ["1.4.0"]
    - match: "GPU_*"
      approved: ["96.00.89.00.01"]

Exit codes

Use the exit code to gate a pipeline.

Output

The table has one row per node and one line per reason, then a summary:

┏━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Node ┃ BMC        ┃ Status ┃ Reasons                                         ┃
┡━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ n01  │ 192.0.2.11 │ PASS   │                                                 │
│ n02  │ 192.0.2.12 │ PASS   │                                                 │
│ n03  │ 192.0.2.13 │ FAIL   │ gpu: 7 found, 8 planned                         │
│ n04  │ 192.0.2.14 │ PASS   │                                                 │
│ n05  │ 192.0.2.15 │ FAIL   │ memory: 31 DIMMs (1984 GiB), 32 planned (2048   │
│      │            │        │ GiB)                                            │
│      │            │        │ firmware: BMC 1.3.2 not in approved [1.4.0]     │
│ n06  │ 192.0.2.16 │ PASS   │                                                 │
└──────┴────────────┴────────┴─────────────────────────────────────────────────┘
4 passed, 2 failed, 0 errored

A node has status PASS, FAIL or ERROR. ERROR means racklint could not read the BMC (a connection failure after retries, a timeout, a rejected login or an invalid reply). No checks run for that node, and the reason is the error.

With --format json the output is the report as JSON. This is one failing node:

{
  "schema_version": 1,
  "results": [
    {
      "node": "n03",
      "bmc_address": "192.0.2.13",
      "status": "fail",
      "findings": [
        {
          "check": "gpu",
          "message": "7 found, 8 planned"
        }
      ],
      "error": null
    }
  ]
}

status is pass, fail or error. error is set only when the status is error.

Python library

The names exported from racklint (its __all__) are the public API. Anything else may change before 1.0. The package ships type hints (py.typed). The names fall into these groups:

The library is async and does not read RACKLINT_* settings: you pass it explicit objects. This example reads the plan for rack R01 from NetBox, reads each BMC and prints a verdict per node. It takes its connection details from the environment.

import asyncio
import os
from pathlib import Path

from racklint import (
    BmcAccess,
    FirmwareBaseline,
    NetBoxSource,
    RedfishReader,
    check_nodes,
    default_checks,
    netbox_client,
)


async def main() -> None:
    async with netbox_client(os.environ["NETBOX_URL"], os.environ["NETBOX_TOKEN"]) as client:
        nodes = await NetBoxSource(client).rack("R01")

    access = BmcAccess(username=os.environ["BMC_USERNAME"], password=os.environ["BMC_PASSWORD"])
    baseline = FirmwareBaseline.load(Path("baseline.yaml"))
    report = await check_nodes(nodes, RedfishReader(access), default_checks(baseline), concurrency=16)

    for result in report.results:
        print(result.node, result.status, [finding.message for finding in result.findings])
    print("all passed" if report.passed else "not all passed")


asyncio.run(main())

check_nodes returns a Report with the results in the order of nodes. To act on each result as soon as it is ready, for live progress, loop over iter_results instead. It takes the same arguments and yields NodeResults in the order they finish:

import asyncio
import os
from pathlib import Path

from racklint import (
    BmcAccess,
    FirmwareBaseline,
    NetBoxSource,
    RedfishReader,
    default_checks,
    iter_results,
    netbox_client,
)


async def main() -> None:
    async with netbox_client(os.environ["NETBOX_URL"], os.environ["NETBOX_TOKEN"]) as client:
        nodes = await NetBoxSource(client).devices(["gpu-01", "gpu-02"])

    access = BmcAccess(username=os.environ["BMC_USERNAME"], password=os.environ["BMC_PASSWORD"])
    checks = default_checks(FirmwareBaseline.load(Path("baseline.yaml")))
    async for result in iter_results(nodes, RedfishReader(access), checks, concurrency=16):
        print(result.node, result.status)


asyncio.run(main())

NetBoxSource raises NetBoxError and FirmwareBaseline.load raises BaselineError; both are subclasses of RacklintError. A BMC that cannot be read does not raise: its node gets status error.

A FirmwareBaseline can also be built from a dict with FirmwareBaseline.model_validate, so an application can keep the baseline in its own database.

The NetBox source is optional. A NetBox plugin runs inside NetBox, so it can build ExpectedNodes from the Django ORM instead of the REST API, then call asyncio.run(check_nodes(...)) from a background job, which is synchronous code.

What it checks

A node passes only if every check passes. Firmware items that match no baseline entry are ignored, and so are reported ports beyond the plan.

If NetBox lacks a value a check needs, the node fails with not planned in NetBox: <item>. A device with no BMC address fails with that reason alone.

What NetBox must hold

Assign the config context to the device type, so every device of that type gets it:

racklint:
  gpu:    {count: 8, model: "NVIDIA H100 80GB HBM3"}
  memory: {dimm_count: 32, total_gib: 2048}

An invalid racklint config context is a configuration error and stops the run with exit code 2.

On the BMC side, racklint starts at /redfish/v1 and follows the links the service returns. It hardcodes no vendor paths. It reads systems, processors, memory, network adapters and the firmware inventory. It uses HTTP Basic authentication on every request and opens no Redfish sessions. It makes up to three attempts at a request that fails with a connection error, a timeout, HTTP 5xx or 429, and uses a connect timeout of 5 seconds and a read timeout of 15 seconds.

Try it without hardware

examples/no-hardware/ is a Docker Compose stack with a real NetBox and six mock BMCs, two of them built wrong on purpose. One command checks the rack, and the summary line is 4 passed, 2 failed, 0 errored. See its README for the steps and what is planted.

Limitations

• GPUs are found only as Redfish Processors with ProcessorType == "GPU". A BMC that exposes GPUs another way shows zero GPUs.
• One set of BMC credentials applies to every node in a run.
• Read-only by design. racklint reports differences and never fixes them in NetBox or on a BMC.

Related work

racklint is smaller than any of these and does one thing: it compares the hardware with NetBox's plan and reports the differences.

License

MIT