infermon

Real-time monitoring and live knob control for vLLM 0.31.0. Closed source. Free to monitor, paid to change.

• Per-step engine data, not scraped histograms: exact TTFT / ITL / TPOT / e2e / queue quantiles, throughput, running / waiting / skipped, KV usage, preemptions, prefix cache hit rate, CUDA graph padding waste, spec decode acceptance, batch mix.
• Binding-limit diagnosis for each step: KV_PRESSURE, SEQ_CAP, TOKEN_BUDGET, DEFERRED, UNDERLOADED. SLA attainment and goodput.
• Live control (Pro): max_num_active_seqs and max_num_scheduled_tokens at the next step boundary, no restart. Bounds = the startup values.
• Local web UI, infermon top in the terminal, Prometheus /metrics.

Stock vLLM wheel, no fork. Two probes (a stat-logger plugin and a scheduler mixin) write to a shared-memory ring; one static Rust binary (infermon-agent) reads it. See docs/ARCHITECTURE.md.

Install (2 minutes)

Linux x86_64 or aarch64. vLLM 0.31.0 exactly (other versions run as passthrough, see FAQ).

A. Fresh: Docker image

docker run --gpus all --network host --ipc host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  ghcr.io/bardavid/infermon:0.31.0-cuda \
  Qwen/Qwen2.5-7B-Instruct \
  --scheduler-cls infermon.sched.TunableAsyncScheduler \
  --enable-logging-iteration-details

• The image is vllm/vllm-openai:v0.31.0 plus the infermon wheel. Arguments go to vllm serve.
• The entrypoint starts the agent next to vLLM. UI: http://127.0.0.1:8765/.
• INFERMON_AGENT=0 turns the built-in agent off. INFERMON_LISTEN=0.0.0.0:8765 with INFERMON_TOKEN=<t> exposes it.
• CPU: ghcr.io/bardavid/infermon:0.31.0-cpu (from vllm/vllm-openai-cpu:v0.31.0).
• Kubernetes: deploy/helm/infermon (agent as a sidecar).

B. Attach to an existing vLLM

pip install infermon            # into the same env as vllm==0.31.0; no dependencies
                                # (or: pip install infermon-0.1.0-cp310-abi3-manylinux_2_28_$(uname -m).whl from the GitHub release)
infermon doctor                 # vLLM version, compat canary, flags, agent binary

Monitoring (Free) needs no vLLM flag: the stat-logger plugin loads automatically. Restart vllm serve once. For live control (Pro), add the scheduler class:

vllm serve <model> --scheduler-cls infermon.sched.TunableAsyncScheduler \
  --enable-logging-iteration-details      # batch mix + step time panels
infermon up                     # starts the bundled agent; prints the UI URL

infermon serve -- <vllm serve args> adds --scheduler-cls for you. Use TunableScheduler if you run with --no-async-scheduling.

Optional flags for more panels: --cudagraph-metrics (CUDA graph waste), --enable-mfu-metrics, --kv-cache-metrics. Do not pass --disable-log-stats.

Try it with no GPU

pip install "infermon[vllm]"    # pulls vllm==0.31.0
infermon demo                   # real vLLM scheduler + synthetic model + agent; prints the UI URL

The demo runs the real vLLM 0.31.0 Scheduler with our mixin on CPU, with Poisson load and a regime shift. No model download, no network.

License (Pro and up)

infermon license install license.json   # saves to ~/.config/infermon/license.json, reloads the agent

Offline Ed25519 check. No phone-home. Expired license + grace → Free; vLLM never stops.

Share a trace, get 30 days of Pro

infermon trace record --hours 24            # all probes in INFERMON_DIR -> infermon-trace-<time>.trace.gz
infermon trace info infermon-trace-*.trace.gz
infermon trace share infermon-trace-*.trace.gz --email you@example.com   # prints a Stripe Checkout URL

• A trace holds ring records only: counts and timings. No prompt or output text exists in them. Ring/socket paths are dropped, the hostname is replaced by its sha256, a local model path keeps only its last component.
• The reward is a 30-day Pro trial for the GPUs in the trace. A card is required. It auto-renews into paid Pro unless you cancel. One reward per customer.
• Only trace share sends data, and only when you run it. Format: docs/SPEC.md section 14.

Tiers

* v0.1 keeps history in memory only (Pro capped at 7 d). The disk store comes later.

Security and privacy

• Prompt and output content never leave the vLLM process. The probes record counts and times only.
• The agent makes no outbound connections. No telemetry. infermon trace share uploads a trace only when you run it. Licenses are verified offline.
• Rings, manifests and control sockets are files with mode 0600 in a 0700 directory (INFERMON_DIR, default /dev/shm/infermon-<uid>). The control socket checks the peer uid (SO_PEERCRED).
• The agent API binds 127.0.0.1:8765. A non-loopback --listen requires --token; mutating requests then need Authorization: Bearer <token>.
• Knob changes go through a governor: schema bounds (never above startup), ≤ 1 change per knob per 2 s, audit log.

FAQ

Is it safe in my serving process?

• Probe-F (stat logger) never raises into vLLM. On an error it logs one WARNING and turns itself off; after 10 errors in a row in record() it disables itself.
• Probe-E (scheduler mixin) runs only if the version check and the AST canary over the vLLM scheduler source pass. Otherwise it is a passthrough to the stock Scheduler: no ring, no socket, vLLM serves.
• Knob writes apply only at a step boundary, from a lock-protected swap. The values are bounded by the startup values, so no buffer or CUDA graph is resized.
• If the agent dies, the knobs keep their last value and vLLM keeps serving. INFERMON_DISABLE=1 turns every probe off.
• Lowering max_num_active_seqs gates admission; it does not preempt running requests. They drain to the new cap.

What does it cost in latency? Measured on CPU (tests/py/bench_probe.py, Python 3.13, vLLM 0.31.0):

GPU step times are 10–50 ms, so the engine-side cost is far below 0.1 % of TPOT. GPU numbers come from the owner GPU run (docs/gpu-test.md). The optional vLLM flags (--enable-logging-iteration-details, --cudagraph-metrics, ...) have their own cost inside vLLM; measure them with the same script.

Which vLLM versions? 0.31.0. On another version, Probe-E is a passthrough and logs an ERROR; Probe-F runs if its canary passes. INFERMON_ALLOW_UNTESTED=1 overrides the version check (not the canary). A nightly job tests each new vLLM release.

Does it work with Grafana? Yes. Scrape http://127.0.0.1:8765/metrics (infermon_<metric>{engine,window}, latencies in ms).

Multi-node, PD disaggregation? Not in v0.1. Multiple engines and DP ranks on one host: yes.

Build from source (maintainers)

scripts/build_wheel.sh                     # static musl agent + manylinux_2_28 wheel in dist/ (off a manylinux host: add --zig)
cargo test --workspace && pytest tests/py -q

CI: .github/workflows/ — ci (T0–T2), e2e-cpu (T3: real vllm serve on CPU), release (wheels + images on v* tags), nightly-gate (new vLLM releases).