litellm-doctor

CLI llm for OpenAI-compatible gateways (LiteLLM and similar): list models, ping, local token estimates, needle context test, stepped context bench.

Setup

git clone git@github.com:hewimetall/litellm-doctor.git
cd litellm-doctor
cp .env.example .env   # set OPENAI_API_KEY (+ optional OPENAI_BASE_URL)
uv sync

.env:

OPENAI_BASE_URL=http://localhost:4000/v1
OPENAI_API_KEY=sk-...

Default base URL if unset: http://localhost:4000/v1.
Auth: Authorization: Bearer $OPENAI_API_KEY.

Vendored tokenizer files ship in the repo under tiktoken_cache/ (no download required for --mode openai).

Commands

Examples

uv run llm list
uv run llm info
uv run llm info --json          # includes input_types + supports_* flags
uv run llm ping gpt-4o-mini
uv run llm pings -j 8 --timeout 5

uv run llm tokens "hello world"
uv run llm tokens - --mode openai < prompt.txt

uv run llm context my-model -n 32000
uv run llm context my-model -n 32000 --mode openai --no-random

uv run llm bench my-model
uv run llm bench my-model --start 1000 --stop 64000 --step 8000
uv run llm bench my-model --no-random --json

Common flags: --timeout, --max-tokens, --json (where supported).

Tokenizer (--mode)

Model → encoding for --mode openai: see MODEL_ENCODING in src/helpers/tokenize.py
(default cl100k_base; o200k_base for gpt-4o / o3 / …). Unknown model names fall back to cl100k_base.

Vendored encodings

Committed in-repo (safe to git push):

• tiktoken_cache/cl100k_base.tiktoken
• tiktoken_cache/o200k_base.tiktoken

Source of the pin script (optional refresh): GitHub niieani/gpt-tokenizer raw data — not openaipublic.blob.core.windows.net.

bash scripts/pin_tiktoken.sh   # refresh from GitHub if needed

Runtime loads BPE from disk (load_tiktoken_bpe(path)). tiktoken.get_encoding() is not used, so OpenAI CDN is never contacted.

context / bench: prompt shape

Single user message:

[bust=<hex>]          # only when --random (default)
SECRET=NEEDLE-42
<padding>
Find SECRET= in this message. Reply with the SECRET value only.

Step success: HTTP 200 and reply contains NEEDLE-42.

bench defaults

• sizes: 1000 … 128000, step 8000
• max_tokens=16, timeout=30
• columns: target, est, usage (prompt_tokens), ms, status, recall
• summary: max_ok_target, first_fail_target, meta max_input_tokens

Releases

Push a tag v* (e.g. v0.1.0) to run .github/workflows/release.yml:

Manual workflow_dispatch builds artifacts / validates the image without publishing a GitHub Release, pushing to GHCR, or uploading to PyPI.

PyPI uses Trusted Publishing (OIDC) from this workflow — no API token in repo secrets. Register the publisher on PyPI with workflow release.yml and Environment name left empty (Any).

docker pull ghcr.io/hewimetall/litellm-doctor:latest
docker run --rm -e OPENAI_API_KEY -e OPENAI_BASE_URL ghcr.io/hewimetall/litellm-doctor:latest list

Layout

src/helpers/llm.py          # CLI
src/helpers/tokenize.py     # token estimates + needle_prompt
src/helpers/paths.py        # data/.env roots (dev, wheel, frozen)
scripts/pin_tiktoken.sh     # optional refresh from GitHub
tiktoken_cache/*.tiktoken   # vendored encodings (in git)
.github/workflows/release.yml
Dockerfile                  # GHCR runtime image
.env                        # secrets — gitignored
.env.example                # template