llamastat

nvtop for llama.cpp: a zero-infrastructure terminal dashboard for llama-server inference. Point it at a running server and watch slots, throughput, and context usage live.

Why llamastat?

llama-server tells you what it is doing, but only in raw JSON. llamastat turns that into a live picture:

• Per-slot activity: which slots are processing prompts, which are generating, and how full each context window is.
• Throughput without --metrics: llamastat diffs slot token counters between polls, so you get generation and prompt tokens/s even on servers started without the Prometheus endpoint.
• Zero infrastructure: no Prometheus, no Grafana, no agents. One pip install, one command, works over any HTTP reachable server (including an SSH port-forward to a production box).
• Degrades gracefully: if the server flaps, is still loading the model, or has /slots disabled, llamastat shows what it can instead of crashing.

Sibling project: vllmstat does the same for vLLM.

Install

pip install llamastat

Usage

llamastat                          # local server on the default port
llamastat -u http://host:8080      # remote server
llamastat -u http://host:8080 -i 0.5
llamastat --once --json            # one snapshot as JSON, for scripting
llamastat --mock                   # synthetic data, no server needed

Key bindings

Flags

What it shows

• Header: model alias, quantization, parameter count, file size, context per slot, training context, server build, health.
• Slots: one row per slot with phase (prompt/gen/idle), task id, prompt progress, generated tokens, per-slot tokens/s, and a context-usage bar.
• Throughput: EWMA generation and prompt tokens/s with session peaks and totals, plus a braille history plot.
• Requests: running/peak concurrency, completed tasks, aggregate context in use.
• Host sensors (Linux, when llamastat runs on the inference box itself): aggregate CPU utilization bar with load average, RAM usage, and CPU, RAM, motherboard, NVMe, and GPU temperatures from hwmon, grouped per chip, plus power draw from hwmon sensors and RAPL package counters. The CPU bar reflects the whole machine, so standalone tools like llama-bench show up here even though they never touch llama-server. RAPL wattage is derived by diffing the energy counters between polls. On hardened kernels energy_uj is root-only; run sudo chmod a+r /sys/class/powercap/intel-rapl*/energy_uj (or run llamastat as root) to expose package power, and llamastat will say so in the panel. Disable everything with --no-host.
• Server metrics: when the server was started with --metrics, llamastat also shows the server-side Prometheus counters (total tokens, deferred requests, context high-water mark, busy slots per decode). The panel hides itself otherwise.
• Sampling defaults: temperature, top_p, top_k, min_p, max_tokens from /props.

llamastat only issues read-only GET requests (/health, /slots, /props, /v1/models, and optionally /metrics), so it is safe to point at a production server.

Requirements

• Python 3.10+
• A llama.cpp llama-server new enough to expose /slots (enabled by default; some deployments disable it, in which case llamastat falls back to header and metrics panels)

Development

git clone https://github.com/bryanvine/llamastat
cd llamastat
pip install -e ".[dev]"
pytest -q

See CONTRIBUTING.md.

License

Apache-2.0. See LICENSE.