LLM Router Monitor

English | 简体中文

Monitor LLM Routers and inference workers from one dashboard.

• Automatic Router detection and worker grouping by model.
• 15 per-model trend charts, rank details, and health reasons.
• English/Chinese, light/dark themes, and a self-contained HTML interface.

Install and run

Python 3.10+:

python -m pip install llm-router-monitor
rmon http://127.0.0.1:4000

Open localhost:8899. llm-router-monitor is equivalent to rmon; all options are listed in rmon --help.

The dashboard binds to 0.0.0.0. Set ROUTER_API_KEY for Router authentication and WORKER_API_KEY for worker authentication. The positional URL takes precedence over --router-url and ROUTER_URL.

Supported Routers

Enable the Router's metrics endpoint or exporter before connecting.

Envoy and bare NGINX Plus addresses are discovered as HTTP endpoints. Use --worker for HTTPS or custom paths.

Envoy with a separate admin listener:

rmon http://127.0.0.1:8080 \
  --admin-url http://127.0.0.1:9901 \
  --metrics-url http://127.0.0.1:9901/stats/prometheus

Exporter with explicit workers:

rmon http://127.0.0.1:8080 \
  --metrics-url http://127.0.0.1:9113/metrics \
  --worker http://worker-a:8000 --worker http://worker-b:8000

Metrics

• TTFT/ITL use count-weighted counter deltas; cache hit uses a 60-second token-weighted window.
• Missing values remain missing. TTFT/ITL connect valid observations; other charts bridge gaps up to 30 seconds with dashed lines. Collection outages and unavailable models stay disconnected.
• Axes adapt to the data: nonnegative values, percentages within 0–100%, and integer count ticks.
• Settings saves the window, refresh interval, language, and theme. Enter minutes for the window and seconds for refresh; decimals are supported.
• The default window is 24 hours, starting no earlier than the first sample. History accumulates in memory until the monitor restarts.

SGLang maintenance

These policies apply to SGLang-managed registrations only.

Recovery, re-registration, or lost registry visibility resets the cleanup timer. Keepalive backs off after failures. Worker processes continue running after deregistration.

To launch SGLang Router, install a compatible sglang_router runtime exposing RouterArgs, then run bash scripts/router.sh. Monitor it with rmon --host 127.0.0.1 (API 2730, metrics 29000).

Development

Keep runtime modules flat in router_monitor/:

Shell entries live in scripts/; behavioral tests live in tests/. Frontend tests require Node.js.

python -m pip install -e '.[dev]'
python -m unittest discover -s tests
ruff check .
ruff format --check router_monitor tests
python -m build

Build outputs are written to dist/. Install a wheel with python -m pip install dist/llm_router_monitor-0.0.1-py3-none-any.whl.