sgtop

A btop-style terminal dashboard for a live SGLang deployment. Not affiliated with the SGLang project — just a small, read-only tool that never touches your inference traffic.

nvtop will tell you a GPU is at 98% utilization. It won't tell you whether your sglang server can actually take more concurrent requests, whether one data-parallel replica is sitting idle while another is drowning, or what your real time-to-first-token looks like. That's what this is for: concurrency vs. the capacity sglang actually computed at startup, KV-cache headroom per replica, and (with --enable-metrics) rolling TTFT/E2E/queue-time percentiles.

Install

pipx install sgtop

Pure standard library, no dependencies, Python ≥ 3.9. pip install --user sgtop works too if you don't have pipx.

Quick start

Point it at a running sglang instance — that's it, nothing else to run:

sgtop --host <sglang-host> --port 30000

Leave out --port entirely and it'll probe a handful of common ports for you. For the LATENCY panel (TTFT / end-to-end / queue-time / cache-hit-rate), start sglang with --enable-metrics; without it you still get concurrency, KV-cache usage, and decode throughput straight off sglang's own API. Prometheus histograms are cumulative counters, so sgtop diffs consecutive scrapes itself to give you an actual rolling 5-minute p50/p90/p99, not a number that just drifts toward whatever your first few requests looked like.

Run sgtop right on the sglang box (or ssh in and run it there) and the GPU panel fills in for free — it shells out to the local nvidia-smi, i.e. whatever's on the machine sgtop itself is running on. Watching a remote instance from somewhere without GPUs of its own just leaves that panel empty, correctly.

The optional sidecar

The one thing sglang's own API genuinely can't give you is a recent-errors line — that needs its log file, which isn't exposed over HTTP at all. For that there's sgtop-server, a small sidecar you can run on the sglang box that also tails its log:

python -m sglang.launch_server --model-path ... --enable-metrics > server.log 2>&1 &
sgtop-server --log server.log --service-port 30000 --port 30001 &
sgtop --port 30001    # or --mode proxy to skip auto-detection

It also serves a plain HTML version of the same data at http://<host>:30001/. Everything else — concurrency, KV-cache, latency — works exactly the same with or without it.

q quits. That's the whole interface.

Watching a fleet of servers

If you've got several machines on the same LAN each running their own sglang, --fleet gives you one screen with a card per server instead of running sgtop four times in four terminals:

sgtop --fleet 'A=10.0.0.1:30001,B=10.0.0.2:30001,C=10.0.0.3:30001,D=10.0.0.4:30001'

--fleet also takes a file path instead of an inline string, one name=host:port per line, # comments allowed — handy once you've got more than two or three boxes.

Each entry is auto-probed the same way single-host mode is, so a plain sglang instance (direct mode) and a sgtop-server sidecar (proxy mode) can be mixed freely in the same fleet. The catch is the same as single-host mode: direct mode only ever shows local GPU stats, i.e. whichever machine sgtop itself is running on, so a genuinely remote box in direct mode shows GPU n/a in its card. To get a real GPU panel and an error count per remote server, run sgtop-server there too and point the fleet entry at its port instead of sglang's own — everything else (concurrency, decode throughput, cache-hit) works either way.

License

MIT