Inference AIops (preview)

Disclaimer: Community-maintained open-source project. Not affiliated with, endorsed by, or sponsored by the vLLM or Ray projects or any inference-serving vendor. Product and trademark names belong to their owners. MIT licensed.

Governed AI-ops for GPU inference clusters — vLLM (OpenAI API + Prometheus /metrics) and Ray Serve / Ray Jobs (Ray dashboard), plus the single-process serving engines SGLang and TGI (Text Generation Inference) — with a built-in governance harness: unified audit log, policy engine, token/runaway budget guard, undo-token recording, and graduated-autonomy risk tiers. It parses each engine's Prometheus /metrics directly (no Prometheus server required) and probes the Ray dashboard independently. A bearer token is optional (many stacks run open). Preview — mock-validated only, not yet verified against a live cluster.

Serving engines. vLLM is the flagship (full Ray Serve control plane: scale, drain, autoscale, LoRA, hot-swap). SGLang and TGI are supported for engine-agnostic observability — health, running-model identity, request-latency metrics, queue depth, and latency RCA — read from each engine's own endpoints and metric names. Being single-process servers, they have no Ray-shaped scale/drain API: those writes return a teaching error pointing you at a real horizontal-scale layer (Ray Serve / Kubernetes / a load balancer).

What it does

The flagship value is root-cause analysis, wrapped in guarded reads and writes:

• diagnose_latency_spike (flagship RCA) — when TTFT/TPOT/e2e latency climbs, it correlates queue depth (running vs waiting), KV-cache pressure / preemptions, and prefix-cache locality into a ranked cause plus the specific knob to turn (add replicas, raise max-num-seqs, fix routing, enlarge KV cache). Every flag is a number, not a black-box verdict.
• diagnose_low_utilization — the inverse: idle GPUs, over-provisioned replicas, or routing that strands a cache-warm replica → what to scale down.
• Prometheus-native — reads vLLM's /metrics endpoint directly; no Prometheus/Grafana deployment needed.
• Governance-grade — the first governance-grade entrant in this niche: audit + budget + risk-tier approval + undo-token + prompt-injection sanitize, with dry-run + double-confirm on the fragile prod ops (scale-down, scale-to-zero, drain, redeploy, hot-swap) the community reports as dangerous.
• Laptop self-test — ~80% of the tool self-tests free: vLLM on a single GPU or CPU-mock + Ray in one local container (ray start --head).

Capability matrix (35 MCP tools)

The engine-agnostic group works against any supported engine (including vLLM); use it for SGLang/TGI targets or a uniform view across a mixed fleet. The Ray Serve / cluster / deploy write groups are vLLM-only (Ray control plane) — they teach-and-refuse on a SGLang/TGI target.

21 read, 14 write. High-risk writes (scale_replicas_down, scale_to_zero, drain_replica, lora_unload, model_hot_swap, replica_restart, model_undeploy, deployment_redeploy) all support dry_run + double-confirm; reversible writes record an undo descriptor.

Install

uv tool install inference-aiops          # or: pipx install inference-aiops

Quick start

inference-aiops init                     # wizard: engine (vllm/sglang/tgi) + host + port + scheme
inference-aiops doctor                   # vLLM: probes Ray + vLLM; SGLang/TGI: engine health + inventory
inference-aiops overview                 # deployments + total replicas + queue backpressure
inference-aiops metrics diagnose         # why is inference slow? ranked RCA + the knob to turn
inference-aiops serve list               # Ray Serve deployments + replica counts

Run as an MCP server (stdio) for the full 35-tool surface:

export INFERENCE_AIOPS_MASTER_PASSWORD=...   # only if a bearer token is stored
inference-aiops mcp

The CLI is a convenience subset (init, overview, serve …, metrics …, secret …, doctor, mcp); the full 35 tools are exposed via the MCP server.

Governance

Every MCP tool passes through the bundled @governed_tool harness:

• Audit — every call (params, result, status, duration, risk tier, approver, rationale) logged to ~/.inference-aiops/audit.db (relocatable via INFERENCE_AIOPS_HOME).
• Budget / runaway guard — token and call budgets trip a circuit breaker on tight poll/retry loops.
• Risk tiers — graduated autonomy; high-risk ops can require a named approver (INFERENCE_AUDIT_APPROVED_BY / INFERENCE_AUDIT_RATIONALE).
• Undo recording — reversible writes (scale, autoscale-config, routing, hot-swap, LoRA load) record an inverse descriptor.

Supported scope + limitations

Preview / mock-only. All behaviour is validated against mocked vLLM /metrics, vLLM OpenAI API, and Ray dashboard responses. ~80% of the tool self-tests on a laptop — vLLM on a single GPU or CPU-mock plus a local one-node Ray head. Not yet verified against a live production cluster.

Unverified against real hardware / topology:

• multi-GPU tensor-parallel / pipeline-parallel deployments,
• real GPU thermal / throttle telemetry (utilisation is best-effort from the Ray dashboard's /api/nodes),
• multi-node drain and node-reboot orchestration.

The fastest live check is inference-aiops doctor.

Missing a capability?

This is the GPU-inference member of the AIops-tools family (governed AI-ops with audit + budget + undo + risk tiers). If a vLLM or Ray capability you need is missing, or your stack speaks a dialect these tools don't yet handle — open an issue or a PR. Contributions welcome.