win-llmtop

A live terminal panel for llama.cpp's llama-server on Windows. It shows where the memory goes while a model runs: dedicated and shared GPU memory per process against the OS budget, system RAM, commit charge and the page file, the running request's progress and speed, the history of finished requests, and the model's memory map. It is read-only: it reads the server's API and logs and Windows' performance counters, and never changes the server.

Requirements

• Windows 10 version 1709 or later (the per-process GPU counters appear in that release);
• 64-bit Python 3.11 or later;
• no other packages (the standard library only);
• a llama.cpp llama-server — in router mode or serving a single model — on the same machine.

Install

pipx install win-llmtop

(recommended: an isolated install with win-llmtop on PATH), or uv tool install win-llmtop, or py -m pip install win-llmtop.

Getting 64-bit Python on Windows: python.org's Windows installer, or winget install Python.Python.3.13.

From source: py -m pip install git+https://github.com/insane-kane/win-llmtop, or clone the repository and run py llmtop.py.

A signed standalone win-llmtop.exe and a winget package are planned for 0.1.1.

First run

Run win-llmtop --check. It reports on the prerequisites and the sources it can reach, one line each, and exits 0 when the server is reachable and the GPU telemetry is present, 1 otherwise:

Getting the most from it

llmtop shows whatever it can reach. A few flags on the llama-server, and options on llmtop, unlock more:

• --log-file (one per model; in router mode, the preset's log-file) and --log-timestamps: these give llmtop the per-model log it reads for the request history — per-request timings, draft acceptance, prefix reuse. Without a log file llmtop can find (give --log-file, or --log-dir so it can build <log-dir>\<preset>.log), the request history is empty and llmtop prints a notice. Without --log-timestamps the log lines carry no timestamps, so the history keeps its relative timings but has no wall-clock times, and llmtop prints a notice suggesting the flag.
• --slots enabled: the running request's live progress — its phase, how far prefill or decode has got, the current t/s and the context fill — is read from the server's /slots endpoint. Without it the endpoint is off, so llmtop shows the memory and system figures but not the in-flight request, and raises a warning telling you to start the server with --slots.
• For the router's own events — model loads, queueing, evictions — point --router-log at the router's log file, or --log-dir at the folder it writes its llama-server-<date>-<time>.log files into. Without either, llmtop cannot find the router log and those events are not shown.

Usage

Run win-llmtop for the live view. win-llmtop --once prints one snapshot and exits; --once --page all prints every page. win-llmtop --plan [PRESET] projects each preset's VRAM against the OS budget without loading anything.

Keys: 1–6, Tab, or ← → switch pages; [ ] change the history window; g sets the graph height; s switches the graph style; ↑ ↓ PgUp PgDn Home End scroll every page except Live; p or Space pauses the display (collection continues); q quits. On the Memory and Tensors pages m cycles models; on Requests m filters by preset; on Events f filters by level; on Tensors t switches between layers and every tensor and d filters by device.

Options, with their environment variables and defaults:

Where it keeps its files

By default, %LOCALAPPDATA%\win-llmtop\ (override with --state-dir or LLMTOP_STATE_DIR). It holds:

• samples-<date>.jsonl — one sample per 5 s plus alert events, 14 days kept, so the sparklines and the event list span restarts;
• requests.jsonl — finished requests, kept because llama-server rewrites a model's log each time that model loads;
• calibration.json — the VRAM overhead learned per configuration.

--no-persist turns the writing off; a read-only run still replays whatever is already there.

Limits

• Which allocation or tensor is in shared GPU memory cannot be told from the counters; only how much.
• The OS budget comes from DXGI for llmtop's own process; the model's budget may differ.
• GPU temperature and power are not exposed by Windows' counters and are not shown.
• The address-space walk shows what kind of memory the process holds, not which allocation is the prompt cache or a checkpoint; the log carries no cache sizes at its default verbosity, so only the configured limits are shown.
• Prefill progress advances once per batch (--batch-size), so the live prefill rate is the rate over the last completed batch.
• KV and state figures are derived; they were checked against llama-fit-params on a hybrid model and a sliding-window mixture-of-experts model, and on any architecture the Memory page reports whether they agree.
• Fixed assumptions: llama.cpp's tensor naming, its offload rules, the GGUF type sizes, and the log line formats of build b11669. The alert thresholds, the colour thresholds for t/s and the size buckets of the Requests page are display or policy choices, not measurements.

Licence

GNU General Public License, version 3 or (at your option) any later version (GPL-3.0-or-later); see LICENSE.