vllmtop

Production-grade TUI dashboard for monitoring vLLM inference servers. Real-time metrics, persistent storage, and Grafana-style visualizations in your terminal.

Features

• Live Dashboard - Real-time KPI cards, gauge bars, and sparklines with 60s rolling history
• Historical Explorer - Time-range analysis with interactive charts
• Request Breakdown - Request completion outcomes and statistics
• Cache & System Analytics - KV cache usage, prefix cache hit rates, and system metrics
• Configurable Alerts - Threshold-based alert rules with persistent alert history
• Multiple Graph Styles - Line, braille, and block charts (press g to cycle)
• Multi-Server Monitoring - Monitor multiple vLLM instances from a single dashboard
• Persistent Storage - All metrics saved to SQLite with automatic retention management

Installation

Requires Python 3.11+.

pip install .

For development:

pip install -e ".[dev]"

Quick Start

# Connect to a local vLLM server (default: http://localhost:8000)
vllmtop

# Connect to a remote server
vllmtop --url http://gpu-server:8000

# Use a config file
vllmtop --config config.yaml

# Custom poll interval and retention
vllmtop --url http://localhost:8000 --interval 2 --retention 60

Configuration

Copy the example config and customize:

cp config.example.yaml config.yaml

targets:
  - url: http://localhost:8000
    name: "GPU Server 1"

graph_style: "line"       # line, braille, or block
poll_interval: 1.0        # seconds
db_path: "./vllm_metrics.db"
retention_days: 30

alert_rules:
  - name: "KV Cache Critical"
    metric: "vllm:kv_cache_usage_perc"
    operator: ">"
    threshold: 90.0
    enabled: true

See config.example.yaml for the full configuration reference.

CLI Options

Keyboard Shortcuts

Metrics Tracked

• Requests - Running, waiting, swapped, queue time
• Cache - KV cache usage, GPU cache usage, prefix cache hit rate
• Tokens - Prompt tokens, generation tokens, totals
• Latency - Time-to-first-token (TTFT), time-per-output-token (TPOT), end-to-end latency
• Throughput - Prompt and generation throughput (tok/s)

License

MIT