vsphere-triage

A read-only vSphere MCP server that triages VMs which are stuck, did not come back after a reboot, or keep losing VMware Tools — across a fleet of vCenters where VMs are short-lived and their names are reused. It sweeps the fleet, groups the abnormal VMs by what they share, weighs the causes, and ends in a verdict. It discovers each vCenter's layout rather than assuming it, so it is not tied to any one platform; it plugs into whatever orchestrator creates your VMs through configuration and a skill (see Using it with an orchestrator).

This repo ships two things: the MCP server (the vsphere_triage package, installed as the vsphere-triage command) and a diagnosis skill under skills/vm-incident/ — the step-by-step incident procedure for a Claude Code agent driving the server. The server is delivered through PyPI; the skill through your Claude Code plugin / skills channel.

vROps (Aria / VCF Operations) is an optional integration: configure VSPHERE_VROPS_URL and it adds a handful of history tools (longer and finer than vCenter keeps). Everything else — the whole diagnosis — runs on vCenter alone, so the server is fully useful with no vROps at all.

It is read-only with one deliberate exception: run_in_guest executes a command inside a guest OS for a diagnostic a read cannot answer. It is marked not-read-only and routed outside the read-only guard on purpose (see below); nothing else can change a VM.

It answers two kinds of question:

• From a VMware Tools error your monitoring raised (which names the vCenter, the machine and the environment id): diagnose_vm(vcenter, env_id, vm_name) runs every check in a fixed order and ends in the likely cause.
• About the fleet ("anything stuck on these vCenters?"): scan judges every VM, groups the abnormal ones by what they share (host, datastore, linked-clone base disk, environment, ...) against their population, and explain_group says why a group is abnormal.

Read-only, by construction (with one marked exception)

The accounts it may run with can write, so the guarantee is in the code, not the account:

• Every vSphere call goes through readonly.ReadOnlyStub, which refuses any method not on a short ALLOWED_METHODS allowlist before a byte is sent: login and logout, the clock, property reads, container views, QueryEvents, QueryPerf, BrowseDiagnosticLog / QueryDescriptions, QueryDateTime (a host's clock) and SearchDatastore_Task (which only lists files).
• Every HTTP request (vmware.log and serial files over /folder, the console over /screen) goes through ensure_read_request: GET only.
• The console is read through /screen, never CreateScreenshot_Task, which writes a PNG into the VM's folder.
• A test drives every read-only tool against simulated vCenters and asserts that the methods sent are a subset of the allowlist; another tries power, reset, answer, reconfigure, destroy and screenshot-to-datastore through a live session and expects each to be refused.

The one exception is run_in_guest, enabled deliberately: it runs a command inside a guest OS through VMware Tools, so it cannot be read-only. It does not go through ReadOnlyStub — it uses the session's raw stub on purpose (VCenterSession.run_in_guest) and is the only path that can change a VM. It needs VMware Tools running in the guest and guest-OS credentials (per call, or VSPHERE_GUEST_USERNAME / VSPHERE_GUEST_PASSWORD), captures output via a temp file in the guest that it downloads and deletes, and a test proves that invoking it never adds a method to the read-only (guarded) surface. Everything else is read-only: nothing is created, changed, answered, powered or deleted, and the server runs nothing in the background.

Concepts, not constants

Nothing hard-codes a folder path, a log field or a naming pattern:

• An environment is a folder named by its environment id. find_environment looks for it (exact name first, then names containing it) and reports where it found it and under which parent (environments, projects, anything). Several matches are reported, never guessed between. VSPHERE_ENV_PARENT_HINTS only orders the search.
• A VM is identified by (vcenter, moref). Its name is looked up only inside its environment folder, since later environments reuse it.
• A vCenter answers to its name, host, the host's first label and any configured alias, so whatever spelling an upstream log uses resolves.
• Group values carry their vCenter (vc-a:DC0_H0): two vCenters' hosts of the same name are different hosts.
• Every check returns the numbers it judged, beside its verdict; thresholds are defaults in VSPHERE_THRESHOLDS, tuned on real incidents.

Install and configure

uv tool install vsphere-triage               # from PyPI once published
uv tool install .                             # or from a checkout; add [ocr] for console OCR (needs tesseract)
pipx install vsphere-triage                   # or with pipx / pip

Configure it entirely through VSPHERE_-prefixed environment variables — only VSPHERE_VCENTERS plus credentials are required; everything else has a default.

An MCP client config, over stdio:

{
  "mcpServers": {
    "vsphere-triage": {
      "command": "vsphere-triage",
      "env": {
        "VSPHERE_VCENTERS": "[{\"name\": \"vc-01\", \"host\": \"vc01.example\"}]",
        "VSPHERE_USERNAME": "svc-readonly@vsphere.local",
        "VSPHERE_PASSWORD": "...",
        "VSPHERE_CA_BUNDLE": "/etc/pki/internal-ca.pem",
        "VSPHERE_ENV_PARENT_HINTS": "environments,projects"
      }
    }
  }
}

Or shared over HTTP, alongside your other MCP servers: vsphere-triage --transport streamable-http --host 0.0.0.0 --port 8000 --allowed-host vsphere-triage.example (/healthz for probes). Over HTTP every caller shares the configured credential, which is read-only in effect either way.

A dedicated vCenter role is still worth creating where the account allows it, so the guarantee holds twice: read inventory, console screen read (confirm the privilege /screen needs on your version), datastore browse and file read, and diagnostics (host logs).

Tools

States and flags

Every VM gets exactly one state, from the bulk read alone:

Flags ride beside the state. Those that matter alone (counted as abnormal): boot_loop, tools_flapping, heartbeat_intermittent (yellow heartbeat: Tools may be crashing), memory_reclaimed (ballooned, swapped or compressed), host_memory_pressure, vcenter_alarm (a red alarm, by name), host_not_connected, long_running (older than long_running_hours, 24 by default), consolidation_needed. Those that only add detail: host_in_maintenance, tools_outdated, cpu_limited (a CPU limit caps it below full speed — apparent starvation that is configured, not the host) and memory_limited (a memory limit below its RAM forces balloon/swap). scan(states=[...]) selects any state or flag, for example ["no_tools"] or ["long_running"].

The bulk read asks for about 35 properties per VM. A property an older vCenter does not have is dropped and the read retried, rather than failing the scan, and the result names what was dropped.

Results lead with finding / findings, keep the raw values, say what was not read (not_checked, available: false with a reason), and are bounded to fit a context window.

What the diagnoses rest on

• CPU starvation: the VM's cpu.ready.summation as a percent per vCPU, and the host's cpu.demand.average against cores x MHz. Demand over 100% is overcommit; usage alone is reported as such, since it cannot exceed 100%.
• Storage: the hosts' datastore.total{Read,Write}Latency for the VM's datastores, disk.{device,kernel,queue}Latency, aborts and resets per device. High device latency points at the storage array (e.g. PowerFlex), high queue latency at the host's own queue.
• The environment: whether the healthy VMs beside a stuck one share its host, datastore and base disk (which argues against them) or not.
• The boot: vmware.log is the hypervisor's log, not the guest's. It shows how far a boot got (power-on or reset, firmware, first disk I/O, drivers, Tools), never the guest kernel's own words.
• The guest's words, when present: a serial port writing to a file (Windows EMS, a Linux kernel console) and the console screen. A VM without them is normal and the result says so.

Developing and testing

The project is managed with uv; uv.lock is committed, so uv sync reproduces the exact dev environment (the dev dependency group — pytest, ruff, ty — installs by default). CI runs uv sync --frozen.

uv sync                                        # set up the dev environment from the lock
uv run python -m pytest -m "not integration"   # the fast unit suite (what CI runs)
uv run python -m pytest                         # everything, including the vcsim integration tests
uv run ruff check . && uv run ruff format --check .
uv run ty check vsphere_triage                  # type check (Astral's ty)

Lint is ruff with select = ["ALL"] (a short, documented ignore list in pyproject.toml); types are checked with ty, scoped to the package — the tests use deliberately loose fixture typing. Python 3.12+.

CI (.github/workflows/ci.yml) runs lint, format, type check and the unit suite on every push, Linux only. The Docker-backed integration suite is not run per-commit — dispatch it by hand (Actions → CI → Run workflow → integration = true) or run it locally.

The integration tests build two simulated vCenters in Docker with vmware/vcsim:v0.52.0 and vmware/govc:v0.52.0. One uses the environments layout and one the projects layout. They hold linked clones, stuck Windows and Linux VMs, a pending question, a boot loop, an EMS stop code, an environment id present on both vCenters and machine names reused across environments. For an air-gapped machine, docker save both images and docker load them there.

What the simulator cannot do, and how the tests deal with it:

• Its performance counters are random, so CPU and storage numbers come from a controlled fake of VCenterSession.perf. Everything else (inventory, folders, disk chains, serial ports, events, files) is read from vcsim.
• It has no /screen endpoint. The 404 path is tested against it; the image analysis is tested with generated screenshots.
• It has no host logs. That path is tested as "unavailable, said so".

To confirm on a real vCenter

• The privilege /screen needs, and what it returns for a black console.
• The statistics level: what QueryPerf keeps past the realtime hour.
• The datastore counter instances (volume id from ds:///vmfs/volumes/<id>/). The datastore name and the last part of the URL are tried after it.
• The vmware.log lines a real stuck boot, a stun and an APD leave. The patterns in vmlog.py are heuristics, and every match is returned with its line.
• What EMS writes to the serial port on the Windows versions in the fleet.
• The vROps stat keys on your version. vrops_list_stat_keys reads them.
• Snapshot sizes: the simulator reports them as 0, so the delta/memory-state accounting from layoutEx is unit-tested only; confirm the GB figures and the Sessions.View privilege for sessions against a real vCenter.
• Host clock skew: QueryDateTime per host and the config.dateTimeInfo/ config.service fields check_host_time reads — the simulator does not model NTP state or a drifting clock, so the classification is unit-tested only. Confirm the exact NTP service key (ntpd vs chronyd) and the skew at which your AD auth breaks.
• Datastore over-provisioning and APD/PDL: summary.uncommitted for thin space, and a host mount that is mounted but accessible: false with an inaccessibleReason. The simulator does not model either, so check_datastores is unit-tested only; confirm the exact inaccessibleReason values your arrays raise.
• Resource limits: a VM's config.cpuAllocation.limit / memoryAllocation.limit are read on the simulator, but confirm that a linked clone made from a template with a limit actually inherits it (the clone-inheritance note in find_resource_limits rests on that).
• run_in_guest: the simulator does not implement guest operations, so the guest-command path is unit-tested only (command shaping, the *-host transfer URL). Confirm against a real vCenter: the guest-operation privileges the account needs, that output capture via a redirected temp file and InitiateFileTransferFromGuest works for your Windows and Linux guests, and that the credentials behave as expected. run_in_guest is the one tool that changes a VM; keep its credentials scoped accordingly.
• That a guest reboot in your fleet logs a reset or guest-reboot event (so uptime is measured from the reboot, not power-on); if some reboots are silent to vCenter, pass rebooted_at from your orchestrator. The simulator cannot exercise this — the logic is covered by unit tests only.
• The dcPath for a datacenter that sits inside a folder, and a datastore name shared across two datacenters. The simulator has one flat datacenter, so these paths (full datacenter inventory path, datastore identity by moref) are correct by construction but unverified live.
• Whether /folder honours a suffix Range request; when it does not, a log larger than VSPHERE_MAX_DOWNLOAD_BYTES is reported as un-tailable rather than pulled whole.

Releasing to PyPI

.github/workflows/release.yml publishes with PyPI Trusted Publishing — no token lives in the repo. One-time setup at pypi.org/manage/account/publishing: add a publisher for project vsphere-triage, owner Heknon, repository vsphere-triage, workflow release.yml, environment pypi (and the same on test.pypi.org if you use the testpypi dispatch). Reserve the name first by publishing 0.0.0, since PyPI names are first-come.

Then releasing is a version bump:

1. raise version in pyproject.toml, commit, merge to main;
2. the workflow sees the new version has no vX.Y.Z tag, builds the sdist + wheel, verifies the wheel installs and imports, publishes to PyPI, then tags the commit and makes the GitHub release.

A merge that does not change the version stops at the gate. Tagging by hand (git tag v0.2.0 && git push origin v0.2.0) still works, and workflow_dispatch can publish the current branch to TestPyPI first.

Both workflows are Linux-only and single-job by design — a pure-Python package needs no cross-platform matrix, which keeps the Actions bill small. The Docker-backed integration suite is never on the per-commit path; dispatch it when you want it.