Skip to main content

llama-doctor logo

llama-doctor

A command-line tool that reads a running llama.cpp or Unsloth Studio server's metrics and tells you what is misconfigured.

CI Python 3.10+ License: MIT

Dashboard

The live dashboard on an RTX 5080 rig running Unsloth Studio, serving Qwen3.8-Flash-Next-GGUF.

What it does

  • Reads /metrics, /props and /v1/models over HTTP. It never attaches to, restarts or reconfigures the server process.
  • Runs 10 analyzers that emit 43 distinct findings across connectivity, prefill and decode throughput, memory, cache hit ratio, context sizing, request queueing, speculative decoding and expert offload, and quantization where exported.
  • Ranks recommendations by impact and names the server flag to change.
  • Returns an exit code per severity, so you can gate CI jobs or cron checks on it.
  • Optionally serves a local dashboard with live charts and findings.

It works with llama.cpp's server and forks that expose the same llamacpp: metrics namespace. It has been tested against Unsloth Studio's llama.cpp builds.

Install

Requires Python 3.10 or newer. There are no runtime dependencies; it uses the standard library only.

The recommended way is pipx, which installs the command in its own environment:

llama-doctor can be installed from PyPI starting with release v0.2.1; until that release is available, install it from source.

pipx install llama-doctor

Or with pip, ideally inside a virtual environment:

pip install llama-doctor

Check the install with llama-doctor --version.

From source

For development, or to run unreleased changes:

git clone https://github.com/msalsas/llama-doctor.git
cd llama-doctor
pip install -e .

To run it without installing anything, from the repository root (the PYTHONPATH=src commands only resolve the package from there):

PYTHONPATH=src python3 -m llama_doctor diagnose http://127.0.0.1:8080

Quick start

Point it at a running server. A plain llama.cpp server needs only the URL (for example http://127.0.0.1:8080); wrappers that require an API key, such as Unsloth Studio, take it from an environment variable via --api-key-env:

llama-doctor diagnose http://laptop-msi:8888 --api-key-env UNSLOTH_API_KEY
llama-doctor 0.2.1  http://laptop-msi:8888
model  Qwen3.8-Flash-Next-GGUF  quant UD-Q3_K_XL  ctx 80128  slots 1  build unsloth-studio/v0.1.908-beta

rates (lifetime averages)
  prefill       135.2 tok/s
  decode        24.6 tok/s
  prefix cache  96.7 %
  drafts        len 2.00  acceptance 64.6 %

findings
  WARN context.near_limit  Context is near its limit
        The observed peak of tokens in context is close to the configured context size.
  OK   connectivity.ok  Endpoints available
  OK   decode.ok  Decode throughput is healthy
  OK   prefix_cache.ok  Prefix cache is effective
  OK   queue.ok  No requests queued
  OK   spec.ok  Speculative decoding is healthy
  OK   props.ok  Properties raise no issue
  N/A  prefill.insufficient_data  Prefill rate cannot be judged
        Prefill measures 135 tok/s, under 300 tok/s, but 97 % of prompt tokens come from the prefix cache, so the lifetime rate mixes short uncached tails with any large uncached prompt and cannot tell them apart. Run --probe for a cache-free measurement, or watch it under serve, whose window shows the uncached tokens per request.
  N/A  kv.insufficient_data  KV cache data unavailable
        The KV cache usage metric is not exported.
  N/A  memory.insufficient_data  Memory estimate unavailable
        Provide --vram-gib and optionally --ram-gib.
recommendations (by impact)
  1. medium Raise the context size or shorten prompts.  [-c]
overall: WARN (exit 1)

Findings are sorted by severity, then by the impact of their recommendations. OK findings print without a detail line. Prefill here runs at 135 tok/s, under the 300 tok/s threshold, yet is not flagged: 96.7 % of prompt tokens come from the prefix cache, so the lifetime rate mixes short uncached tails with any large uncached prompt and prefill.insufficient_data (N/A) is reported instead. prefill.slow (WARN) fires only when the prefix-cache hit is under 50 % or unknown.

Recommendations that name the same flag stay separate when their actions differ, so one finding asking to shrink -c and another asking to raise it both show up.

Exit codes: 0 = OK, 1 = WARN, 2 = CRIT (including server unreachable), 3 = invalid usage or arguments.

Dashboard

llama-doctor serve http://127.0.0.1:8080

serve runs a local HTTP server on port 8765; open http://127.0.0.1:8765 in a browser. It polls the llama.cpp server every 2 s and keeps the last 300 points in memory. Chart rates are computed over a sliding 30 s window (--rate-window), because llama.cpp counters only move when a request completes.

Tiles, left to right: Decode tok/s, Prefill tok/s, Prefix-cache hit, Context, Queue.

  • Decode, Prefill and Prefix-cache hit show the latest windowed value, with the lifetime average below it (or only the lifetime average while the server is idle).
  • Current llama-server builds and Unsloth Studio do not export a KV usage gauge, so the dashboard has no KV tile. Older builds that still export kv_cache_usage_ratio keep the KV findings (kv.saturated warns at 90 % or more of the real gauge; kv.estimated appears when only /slots is readable).
  • Context shows the peak tokens held in context as a share of n_ctx (n_tokens_max / n_ctx, for example 98.0 % and 78.5k / 80.1k). The peak is the highest point since the server started, not live use; when /slots is readable the tile reads live X % - peak Y %, where live is the tokens held by the slots right now (prompt plus decoded) over their summed n_ctx. context.near_limit warns at 90 % or more.
  • Queue shows requests being processed and, below it, requests deferred while waiting for a free slot (requests_processing, requests_deferred).

Charts:

  • Decode throughput and Prefill throughput, in tokens per second, averaged over the rate window.
  • Prefix cache and context, in percent: Prefix-cache hit, and Context used as a dashed line (a peak, not a live gauge).
  • Speculative decoding: Draft length in tokens and Draft acceptance in percent, averaged over the rate window.

Every tile, chart and the quant, context and slots fields in the header have a help tooltip that says what the value measures and how to read it, independent of model or hardware.

Dashboard findings

The findings and recommendations panels re-run every analyzer on each poll, so a change to your server flags shows up within one interval. The dashboard binds to 127.0.0.1 by default and loads no external resources: no CDNs, fonts or analytics.

CLI reference

llama-doctor diagnose URL

Flag Effect
--json Print findings as JSON instead of text.
--probe Send a warm-up and one synthetic completion request to measure real latency.
--probe-budget SEC Total time allowed for --probe (default 300). The warm-up divides it evenly across the completion routes tried and the measured prompt gets the rest; slow servers get a smaller prompt that is still reliable.
--vram-gib N GPU memory available, used by the memory estimate.
--ram-gib N System RAM available for offloaded weights.
--weights-gib N Model weight size in GiB, used only when the server does not report it; the memory finding then shows weights_source: hint.
--api-key-env VAR Read the server's API key from environment variable VAR.
--model NAME For a llama-server in router mode (several models behind one URL): read /metrics for model NAME by sending GET /metrics?model=NAME. /props, /v1/models and /slots are read as usual. Cannot be combined with --probe (exit 3).
--timeout SEC HTTP timeout per request. --probe requests use --probe-budget instead.
--no-color Disable ANSI colour in text output.
--no-slots Do not read GET /slots (live context and the KV estimate in findings). Without this flag it is read with the same auth as /metrics; any failure just leaves those values out. Only token counts are used; prompt text and ids are never kept.

--probe is off by default. It is the only option that sends a write request to the server; everything else is a read-only GET.

llama-doctor serve URL

Flag Effect
--host Address to bind (default 127.0.0.1).
--port Port to listen on (default 8765).
--interval Seconds between polls (default 2.0).
--history Number of points kept for the charts (default 300).
--rate-window SEC Seconds over which chart rates and the queue check are computed (default 30.0).
--vram-gib N GPU memory available, same as in diagnose; without it the memory check stays N/A on the dashboard.
--ram-gib N System RAM available for offloaded weights, same as in diagnose.
--weights-gib N Model weight size in GiB, same as in diagnose; the size reported by the server wins when present.
--model NAME Router mode model whose /metrics the dashboard polls, same as in diagnose.
--no-slots Do not poll GET /slots, same as in diagnose; the context tile then shows only the peak.

serve also accepts --api-key-env and --timeout.

Checks

The analyzers cover connectivity, server properties, prefill throughput, decode throughput, prefix cache, KV cache usage, context window (near the limit or oversized), request queue, speculative decoding, and memory (exact weights sizing and expert offload). The modules in src/llama_doctor/analysis/ are the source of truth for each finding and its threshold.

Findings panel

Two findings that the server above does not trigger but other setups do:

  • prefix_cache.low_hit_ratio (WARN below 20 %): for example, only 10.3 % of prompt tokens came from the cache. The fix is to keep the prompt prefix byte-identical across requests and enable --cache-reuse.
  • context.oversized (WARN when the peak is under 25 % of a context of 16384 tokens or more): for example, the server reserves a 32768-token context but the peak is 5.5k tokens. The suggested size is twice the peak rounded up to a multiple of 4096, so -c 12288 frees KV cache memory.

Development

python3 -m venv .venv
. .venv/bin/activate
pip install -e . ruff
python3 -m unittest discover -v
ruff check src tests

To check the package the way CI does:

pip install "build>=1.2" "twine>=6.1"
python3 -m build
python3 -m twine check --strict dist/*
python3 -I scripts/check_dist.py dist

Note: scripts/check_dist.py is POSIX-only (it uses venv/bin paths).

Pull requests are welcome.

License

MIT. See LICENSE.

Metadata

Release files for llama-doctor 0.2.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for llama-doctor 0.2.1
File Size Uploaded Provenance Actions
llama_doctor-0.2.1.tar.gz 167.9 kB With provenance Details for llama_doctor-0.2.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for llama-doctor 0.2.1
File Interpreter ABI Platform Actions
llama_doctor-0.2.1-py3-none-any.whl Python 3 none any Details for llama_doctor-0.2.1-py3-none-any.whl

Total release size: 297.8 kB

Release files / llama_doctor-0.2.1.tar.gz

Download URL llama_doctor-0.2.1.tar.gz
Size 167.9 kB
Tags Source
SHA-256 checksum
How to use checksums
d85b6b5090d6b2cca12d35f490cc50869c474a74a5dc93e7700966b2b26870ad
BLAKE2b-256 checksum
How to use checksums
aa4f046aeac43796c734fb6aec1a88c8c66eb026638084cb1f1196b2a5d7cae9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.

Transparency log

Release files / llama_doctor-0.2.1-py3-none-any.whl

Download URL llama_doctor-0.2.1-py3-none-any.whl
Size 129.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
ef1fcba03bb18e7f8db5900876d07bb91600909e38d1a172aa2ba1921288c325
BLAKE2b-256 checksum
How to use checksums
ccf1a14e7b52180d0f548bd1e5a6151281dca391f5c17de86fcd7047909d4d4a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.

Transparency log

Security for llama-doctor 0.2.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for llama_doctor-0.2.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.2.1 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page