What it does

litmus-lab runs your model through HuggingFace (FP16, INT8, NF4, FP4, NF4+double-quant, HQQ, Quanto INT8/INT4, plus AWQ/GPTQ against a pre-quantized checkpoint) and vLLM (FP16, BitsAndBytes, FP8, AWQ, GPTQ) in isolated passes — measuring VRAM, throughput, latency and perplexity on your actual hardware — then outputs a single deployment recommendation.

┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Mode            ┃ VRAM (MB) ┃ TPS      ┃ TTFT     ┃ Perplexity ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━┩
│ HF · FP16       │ 7297.83   │ 32.69    │ 0.030s   │ 5.64       │
│ HF · INT8       │ 3846.47   │ 14.26    │ 0.083s   │ 5.81       │
│ HF · NF4        │ 2334.96   │ 25.98    │ 0.069s   │ 7.34       │
│ vLLM · FP16     │ 12687.31  │ 111.68   │ 0.448s   │ 5.65       │
└─────────────────┴───────────┴──────────┴──────────┴────────────┘

  Recommendation  Deploy vLLM · FP16
                  3.4× faster than HF · PPL delta 0.01
                  Memory-constrained? HF · NF4 saves 4963 MB (PPL +1.70)

Installation

# HF Transformers backend — FP16, INT8, NF4, FP4, NF4+double-quant, HQQ, Quanto INT8/INT4
pip install litmus-lab

# + vLLM backend (FP16, BitsAndBytes, FP8)
pip install "litmus-lab[vllm]"

# + AI recommendations via Groq
pip install "litmus-lab[ai]"

# + AWQ / GPTQ pre-quantized checkpoint support (HF + vLLM)
# Note: gptqmodel builds from source and can fail depending on your CUDA toolchain —
# kept separate from `all` on purpose so a failed build here never breaks the base CLI.
pip install "litmus-lab[awq,gptq]"

# vLLM + AI recommendations (guaranteed to install cleanly — no source builds)
pip install "litmus-lab[all]"

Note: vLLM requires Linux or WSL2. It does not run on Windows natively.

Quick start

litmus-lab \
  --model microsoft/Phi-3-mini-4k-instruct \
  --prompt "Explain the transformer architecture" \
  --backend all

Usage

Flags

Backend modes

Any mode that hits a missing optional dependency, an unsupported GPU (e.g. FP8 on pre-Hopper hardware), or doesn't fit in VRAM prints a yellow skip line and continues — it won't abort the whole run. Failed modes also show up as their own row in the results table (Status column) with a short reason, instead of silently disappearing.

vLLM configuration

litmus-lab currently runs every vLLM mode with a fixed configuration — not yet exposed as CLI flags:

If a mode's weights + KV cache genuinely can't fit in your VRAM even with this config (e.g. FP16 on a 7B model on a 16GB card, where weights alone can take ~85% of it), that's a real finding, not a bug — it shows up as a failed row in the results table rather than crashing the run, and tells you quantization isn't optional for that model on that GPU.

🚧 Planned: flags to override these (--max-model-len, --gpu-memory-utilization, etc.) so you can tune vLLM's config to your own hardware instead of relying on the fixed defaults above.

Examples

# Compare quantization formats
litmus-lab \
  --model Qwen/Qwen2.5-7B-Instruct \
  --prompt "Explain quantum gravity" \
  --backend hf

# vLLM only
litmus-lab \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --prompt "Write a Linux shell script" \
  --backend vllm

# Full benchmark — HF + vLLM side by side
litmus-lab \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --token hf_xxxxxxx \
  --prompt "Explain TCP congestion control" \
  --backend all

# Include AWQ / GPTQ rows against pre-quantized checkpoints
litmus-lab \
  --model teknium/OpenHermes-2.5-Mistral-7B \
  --prompt "Explain the theory of relativity in simple terms" \
  --backend all \
  --awq-model TheBloke/OpenHermes-2.5-Mistral-7B-AWQ \
  --gptq-model TheBloke/OpenHermes-2.5-Mistral-7B-GPTQ

AI Recommendations

Set a Groq API key (free tier) to get an AI-powered verdict from llama-3.3-70b-versatile instead of the built-in heuristic:

export GROQ_API_KEY=gsk_...
litmus-lab --model Qwen/Qwen2.5-7B --prompt "Hello" --backend all

No extra flags needed. Falls back to the offline heuristic automatically if the key is missing, rate limit is hit, or there is no internet.

Metrics explained

Supported models

✅ HF backend

Any HuggingFace causal language model loadable via AutoModelForCausalLM:

Gated models (Llama, Gemma) require --token.

Quantization modes run against the base repo on the fly — no separate checkpoint needed — except AWQ/GPTQ:

✅ vLLM backend

Same model coverage as HF. VRAM is generally higher than HF at the same precision due to the KV cache pool, but throughput is significantly better under concurrent load.

❌ Not supported

Requirements

• Python 3.10+
• NVIDIA GPU with CUDA support (CPU works but is very slow, and quantization gives smaller wins there)
• NVIDIA driver + CUDA 11.8+ / 12.x for most GPUs. Exception: Blackwell-generation GPUs (RTX 50-series, compute capability 12.x) need CUDA ≥ 12.9 and driver ≥ 575.51 — the vllm backend's FlashInfer kernels fail to initialize below that (RuntimeError: FlashInfer requires GPUs with sm75 or higher / SM 12.x requires CUDA >= 12.9). The hf backend is unaffected and works fine on any CUDA version.
• Disk space for whatever model you benchmark — 7B models are roughly 15-30GB on disk depending on dtype — plus the small WikiText-2 dataset used for perplexity
• --token required for gated models (Llama, Gemma, etc.)
• vLLM backend (--backend vllm / all): Linux or WSL2 only, install with pip install "litmus-lab[vllm]" — does not run natively on Windows
• AWQ/GPTQ pre-quantized modes (--awq-model / --gptq-model): need pip install "litmus-lab[awq,gptq]" and a separately pre-quantized checkpoint repo (e.g. TheBloke/*-AWQ) — these don't run against the base model repo

Troubleshooting

vLLM fails to initialize on Blackwell GPUs (RTX 50-series)

Symptoms — one or more of these during --backend vllm / all, often across every vLLM mode at once:

ImportError: vLLM import failed (libcudart.so.13: cannot open shared object file...)
RuntimeError: ... cudaHostGetDevicePointer failed: CUDA driver version is insufficient for CUDA runtime version
Failed to get device capability: SM 12.x requires CUDA >= 12.9.
RuntimeError: FlashInfer requires GPUs with sm75 or higher
RuntimeError: The NVIDIA driver on your system is too old (found version 12080)

Root cause: these are all symptoms of the same mismatch — Blackwell-generation GPUs (compute capability 12.x) need CUDA ≥ 12.9 and driver ≥ 575.51 for vLLM's FlashInfer kernels. Which exact error you see depends on which model/quantization mode happens to hit the mismatch first.

Diagnose:

nvidia-smi   # top-right "CUDA Version" is the max your *driver* supports
python -c "import torch; print(torch.version.cuda, torch.cuda.is_available())"

If nvidia-smi reports less than 12.9, that's confirmed.

Fix, if you control the host (your own machine, not a rented container):

sudo apt update && sudo apt install -y nvidia-driver-575   # or newer — check nvidia.com
sudo reboot
pip uninstall torch torchvision torchaudio vllm -y
pip install torch --index-url https://download.pytorch.org/whl/cu129
pip install vllm

If you're on a rented cloud pod (RunPod/Vast.ai/Lambda/etc.) — the driver belongs to the host, not your container, so nothing you pip/apt install inside it can change it. Options:

• Look for a pod template/image explicitly advertising a newer driver or "Blackwell-ready"/CUDA 12.9 support
• Ask the provider's support directly whether such nodes exist
• Provision a fresh pod — driver versions vary node to node

Careful: upgrading torch alone to a cu129 build without a matching driver makes things worse, not better — torch's own driver check then fails outright (RuntimeError: The NVIDIA driver on your system is too old), and torch.cuda.is_available() silently returns False, so litmus-lab falls back to CPU for everything, including the HF backend that was working fine before. If that happens, revert:

pip uninstall torch torchvision torchaudio -y
pip install torch --index-url https://download.pytorch.org/whl/cu128

In the meantime: --backend hf is unaffected by any of this and works fine on Blackwell GPUs, since it never touches vLLM/FlashInfer.

vLLM: "No available memory for the cache blocks" / negative KV cache memory

Symptom:

(EngineCore pid=...) INFO ... [gpu_worker.py:508] Available KV cache memory: -0.14 GiB
ValueError: No available memory for the cache blocks. Try increasing `gpu_memory_utilization`...

This usually isn't a bug — it means the model's weights alone (plus vLLM's torch.compile/CUDA graph overhead) already consume nearly all the VRAM budget, leaving no room for a KV cache at all. It shows up most often running FP16 on a 7B-class model on a 16GB GPU, where weights alone take ~13-14 GiB. Quantized modes (BitsAndBytes/FP8/AWQ/GPTQ) use roughly a quarter of that for weights and typically have plenty of headroom left over on the same card.

That mode will show up as a failed row in the results table (see the Status column) with the short reason, rather than crashing the whole run. If FP16 fails while your quantized modes succeed, that's the actual answer: quantization isn't optional for that model on that GPU. See vLLM configuration above for exactly what memory budget litmus-lab computes today, and note that per-run tuning flags for this are planned (see roadmap below).

What's being built 🚧

Want to shape the roadmap? Open an issue or join the waitlist for the web platform.

License

MIT