llama-doctor
A command-line tool that reads a running llama.cpp or Unsloth Studio server's metrics and tells you what is misconfigured.
The live dashboard on an RTX 5080 rig running Unsloth Studio, serving Qwen3.8-Flash-Next-GGUF.
What it does
- Reads
/metrics,/propsand/v1/modelsover HTTP. It never attaches to, restarts or reconfigures the server process. - Runs 10 analyzers that emit 43 distinct findings across connectivity, prefill and decode throughput, memory, cache hit ratio, context sizing, request queueing, speculative decoding and expert offload, and quantization where exported.
- Ranks recommendations by impact and names the server flag to change.
- Returns an exit code per severity, so you can gate CI jobs or cron checks on it.
- Optionally serves a local dashboard with live charts and findings.
It works with llama.cpp's server and forks that expose the same llamacpp: metrics namespace. It has been tested against Unsloth Studio's llama.cpp builds.
Install
Requires Python 3.10 or newer. There are no runtime dependencies; it uses the standard library only.
The recommended way is pipx, which installs the command in its own environment:
llama-doctor can be installed from PyPI starting with release v0.2.1; until that release is available, install it from source.
pipx install llama-doctor
Or with pip, ideally inside a virtual environment:
pip install llama-doctor
Check the install with llama-doctor --version.
From source
For development, or to run unreleased changes:
git clone https://github.com/msalsas/llama-doctor.git
cd llama-doctor
pip install -e .
To run it without installing anything, from the repository root (the PYTHONPATH=src commands only resolve the package from there):
PYTHONPATH=src python3 -m llama_doctor diagnose http://127.0.0.1:8080
Quick start
Point it at a running server. A plain llama.cpp server needs only the URL (for example http://127.0.0.1:8080); wrappers that require an API key, such as Unsloth Studio, take it from an environment variable via --api-key-env:
llama-doctor diagnose http://laptop-msi:8888 --api-key-env UNSLOTH_API_KEY
llama-doctor 0.2.1 http://laptop-msi:8888
model Qwen3.8-Flash-Next-GGUF quant UD-Q3_K_XL ctx 80128 slots 1 build unsloth-studio/v0.1.908-beta
rates (lifetime averages)
prefill 135.2 tok/s
decode 24.6 tok/s
prefix cache 96.7 %
drafts len 2.00 acceptance 64.6 %
findings
WARN context.near_limit Context is near its limit
The observed peak of tokens in context is close to the configured context size.
OK connectivity.ok Endpoints available
OK decode.ok Decode throughput is healthy
OK prefix_cache.ok Prefix cache is effective
OK queue.ok No requests queued
OK spec.ok Speculative decoding is healthy
OK props.ok Properties raise no issue
N/A prefill.insufficient_data Prefill rate cannot be judged
Prefill measures 135 tok/s, under 300 tok/s, but 97 % of prompt tokens come from the prefix cache, so the lifetime rate mixes short uncached tails with any large uncached prompt and cannot tell them apart. Run --probe for a cache-free measurement, or watch it under serve, whose window shows the uncached tokens per request.
N/A kv.insufficient_data KV cache data unavailable
The KV cache usage metric is not exported.
N/A memory.insufficient_data Memory estimate unavailable
Provide --vram-gib and optionally --ram-gib.
recommendations (by impact)
1. medium Raise the context size or shorten prompts. [-c]
overall: WARN (exit 1)
Findings are sorted by severity, then by the impact of their recommendations. OK findings print without a detail line. Prefill here runs at 135 tok/s, under the 300 tok/s threshold, yet is not flagged: 96.7 % of prompt tokens come from the prefix cache, so the lifetime rate mixes short uncached tails with any large uncached prompt and prefill.insufficient_data (N/A) is reported instead. prefill.slow (WARN) fires only when the prefix-cache hit is under 50 % or unknown.
Recommendations that name the same flag stay separate when their actions differ, so one finding asking to shrink -c and another asking to raise it both show up.
Exit codes: 0 = OK, 1 = WARN, 2 = CRIT (including server unreachable), 3 = invalid usage or arguments.
Dashboard
llama-doctor serve http://127.0.0.1:8080
serve runs a local HTTP server on port 8765; open http://127.0.0.1:8765 in a browser. It polls the llama.cpp server every 2 s and keeps the last 300 points in memory. Chart rates are computed over a sliding 30 s window (--rate-window), because llama.cpp counters only move when a request completes.
Tiles, left to right: Decode tok/s, Prefill tok/s, Prefix-cache hit, Context, Queue.
- Decode, Prefill and Prefix-cache hit show the latest windowed value, with the lifetime average below it (or only the lifetime average while the server is idle).
- Current llama-server builds and Unsloth Studio do not export a KV usage gauge, so the dashboard has no KV tile. Older builds that still export
kv_cache_usage_ratiokeep the KV findings (kv.saturatedwarns at 90 % or more of the real gauge;kv.estimatedappears when only/slotsis readable). - Context shows the peak tokens held in context as a share of
n_ctx(n_tokens_max / n_ctx, for example98.0 %and78.5k / 80.1k). The peak is the highest point since the server started, not live use; when/slotsis readable the tile readslive X % - peak Y %, where live is the tokens held by the slots right now (prompt plus decoded) over their summedn_ctx.context.near_limitwarns at 90 % or more. - Queue shows requests being processed and, below it, requests deferred while waiting for a free slot (
requests_processing,requests_deferred).
Charts:
- Decode throughput and Prefill throughput, in tokens per second, averaged over the rate window.
- Prefix cache and context, in percent: Prefix-cache hit, and Context used as a dashed line (a peak, not a live gauge).
- Speculative decoding: Draft length in tokens and Draft acceptance in percent, averaged over the rate window.
Every tile, chart and the quant, context and slots fields in the header have a help tooltip that says what the value measures and how to read it, independent of model or hardware.
The findings and recommendations panels re-run every analyzer on each poll, so a change to your server flags shows up within one interval. The dashboard binds to 127.0.0.1 by default and loads no external resources: no CDNs, fonts or analytics.
CLI reference
llama-doctor diagnose URL
| Flag | Effect |
|---|---|
--json |
Print findings as JSON instead of text. |
--probe |
Send a warm-up and one synthetic completion request to measure real latency. |
--probe-budget SEC |
Total time allowed for --probe (default 300). The warm-up divides it evenly across the completion routes tried and the measured prompt gets the rest; slow servers get a smaller prompt that is still reliable. |
--vram-gib N |
GPU memory available, used by the memory estimate. |
--ram-gib N |
System RAM available for offloaded weights. |
--weights-gib N |
Model weight size in GiB, used only when the server does not report it; the memory finding then shows weights_source: hint. |
--api-key-env VAR |
Read the server's API key from environment variable VAR. |
--model NAME |
For a llama-server in router mode (several models behind one URL): read /metrics for model NAME by sending GET /metrics?model=NAME. /props, /v1/models and /slots are read as usual. Cannot be combined with --probe (exit 3). |
--timeout SEC |
HTTP timeout per request. --probe requests use --probe-budget instead. |
--no-color |
Disable ANSI colour in text output. |
--no-slots |
Do not read GET /slots (live context and the KV estimate in findings). Without this flag it is read with the same auth as /metrics; any failure just leaves those values out. Only token counts are used; prompt text and ids are never kept. |
--probe is off by default. It is the only option that sends a write request to the server; everything else is a read-only GET.
llama-doctor serve URL
| Flag | Effect |
|---|---|
--host |
Address to bind (default 127.0.0.1). |
--port |
Port to listen on (default 8765). |
--interval |
Seconds between polls (default 2.0). |
--history |
Number of points kept for the charts (default 300). |
--rate-window SEC |
Seconds over which chart rates and the queue check are computed (default 30.0). |
--vram-gib N |
GPU memory available, same as in diagnose; without it the memory check stays N/A on the dashboard. |
--ram-gib N |
System RAM available for offloaded weights, same as in diagnose. |
--weights-gib N |
Model weight size in GiB, same as in diagnose; the size reported by the server wins when present. |
--model NAME |
Router mode model whose /metrics the dashboard polls, same as in diagnose. |
--no-slots |
Do not poll GET /slots, same as in diagnose; the context tile then shows only the peak. |
serve also accepts --api-key-env and --timeout.
Checks
The analyzers cover connectivity, server properties, prefill throughput, decode throughput, prefix cache, KV cache usage, context window (near the limit or oversized), request queue, speculative decoding, and memory (exact weights sizing and expert offload). The modules in src/llama_doctor/analysis/ are the source of truth for each finding and its threshold.
Two findings that the server above does not trigger but other setups do:
prefix_cache.low_hit_ratio(WARN below 20 %): for example, only 10.3 % of prompt tokens came from the cache. The fix is to keep the prompt prefix byte-identical across requests and enable--cache-reuse.context.oversized(WARN when the peak is under 25 % of a context of 16384 tokens or more): for example, the server reserves a 32768-token context but the peak is 5.5k tokens. The suggested size is twice the peak rounded up to a multiple of 4096, so-c 12288frees KV cache memory.
Development
python3 -m venv .venv
. .venv/bin/activate
pip install -e . ruff
python3 -m unittest discover -v
ruff check src tests
To check the package the way CI does:
pip install "build>=1.2" "twine>=6.1"
python3 -m build
python3 -m twine check --strict dist/*
python3 -I scripts/check_dist.py dist
Note: scripts/check_dist.py is POSIX-only (it uses venv/bin paths).
Pull requests are welcome.
License
MIT. See LICENSE.
Metadata
Release files for llama-doctor 0.2.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| llama_doctor-0.2.1.tar.gz | 167.9 kB | With provenance | Details for llama_doctor-0.2.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| llama_doctor-0.2.1-py3-none-any.whl | Python 3 | none | any | Details for llama_doctor-0.2.1-py3-none-any.whl |
Total release size: 297.8 kB
Release files / llama_doctor-0.2.1.tar.gz
| Download URL | llama_doctor-0.2.1.tar.gz |
|---|---|
| Size | 167.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
d85b6b5090d6b2cca12d35f490cc50869c474a74a5dc93e7700966b2b26870ad
|
|
BLAKE2b-256 checksum How to use checksums |
aa4f046aeac43796c734fb6aec1a88c8c66eb026638084cb1f1196b2a5d7cae9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.
Transparency logRelease files / llama_doctor-0.2.1-py3-none-any.whl
| Download URL | llama_doctor-0.2.1-py3-none-any.whl |
|---|---|
| Size | 129.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
ef1fcba03bb18e7f8db5900876d07bb91600909e38d1a172aa2ba1921288c325
|
|
BLAKE2b-256 checksum How to use checksums |
ccf1a14e7b52180d0f548bd1e5a6151281dca391f5c17de86fcd7047909d4d4a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.
Transparency logSecurity for llama-doctor 0.2.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for llama_doctor-0.2.1.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.