gpu-fit
Will this LLM fit on this GPU, at this context length, for this many users?
gpu-fit answers that before you rent the GPU. It reads the model's config and weight sizes
from Hugging Face and works out vLLM's memory budget: weights, runtime overhead, KV cache, how
many requests fit at full context, and the longest single request. Then it prints the
vllm serve command.
$ gpu-fit Qwen/Qwen3-32B --context 32k --concurrency 4
GPU smallest setup requests at full ctx KV tokens
L4 8× L4 10.6 347,888
L40S 4× L40S 11.6 380,656
A100-80 2× A100-80 9.9 323,484
H100 2× H100 9.8 320,885
H200 1× H200 7.9 259,428
...
$ gpu-fit deepseek-ai/DeepSeek-V3 --gpu H200 --tp 8 --context 128k
vLLM budget per GPU 126.4 GiB
weights 80.1 GiB
runtime overhead (assumed) 2.0 GiB
left for KV cache 44.2 GiB
KV cache capacity 675,671 tokens
Requests at full context 5.2
Longest single request 163,840 tokens
✓ Fits. Suggested command:
vllm serve deepseek-ai/DeepSeek-V3 --max-model-len 131072 --tensor-parallel-size 8 --max-num-seqs 5
Install
pip install gpu-fit # no dependencies, Python 3.10+
Why another VRAM calculator?
Most calculators multiply parameters by bytes and add a guess. Modern models break that:
| Architecture | What changes | gpu-fit |
|---|---|---|
| Grouped-query attention | KV per token depends on KV heads, not attention heads | ✓ |
| Hybrid (linear attention + full attention, e.g. Qwen3-Next family) | Only full-attention layers grow with context; the rest hold fixed state per request | ✓ |
| MLA (DeepSeek) | Compact latent KV, but copied to every tensor-parallel GPU | ✓ |
| Sliding window (Gemma, Mistral) | Window layers stop growing past the window | ✓ |
| FP8 / AWQ / GPTQ checkpoints | Exact weight size from the safetensors metadata | ✓ |
| Tensor parallel beyond KV heads | KV heads get replicated, not split | ✓ |
| FP8 KV cache | Doubles capacity: --kv-cache-dtype fp8 |
✓ |
Options
gpu-fit MODEL [--gpu L40S | --gpu-mem-gib 48] [--tp 2] [--context 128k]
[--concurrency 8] [--kv-cache-dtype fp8] [--util 0.9]
[--overhead-gib 2.0] [--json]
MODEL: a Hugging Face id, or a local folder withconfig.jsonand*.safetensors. Gated models needHF_TOKENin the environment.- Leave out
--gputo compare every GPU in the catalog with the smallest tensor-parallel size that works. --jsonfor scripts and CI.
How the maths works
vLLM takes gpu_memory_utilization × GPU memory, loads the weights, profiles a forward pass
(activations, CUDA graphs), and gives the rest to the KV cache. gpu-fit mirrors that:
KV pool = memory × utilization − weights / tp − overhead
KV/token = 2 × full-attention layers × KV heads per GPU × head dim × dtype bytes
requests = KV pool / (context × KV/token + sliding-window KV + recurrent state)
Accuracy, honestly
- Weights and KV per token come straight from the config, so they're exact.
- Runtime overhead (activations, CUDA graphs) is an assumption: 2 GiB per GPU by default.
It grows with batch size and model width. Override it with
--overhead-gib. - Hybrid models: vLLM pads KV pages to hold the recurrent state, so real capacity can be lower than shown.
- Treat the output as a planning estimate. Confirm on the real GPU before you commit money.
Help make it exact: share what vLLM printed on your hardware (see CONTRIBUTING.md). Every measurement improves the defaults for everyone.
Roadmap
- Measured overhead table from community results
- Throughput estimate (tokens/s) from memory bandwidth
- SGLang and TensorRT-LLM memory models
- Web version
- Vision encoder and LoRA memory
License
MIT
Metadata
Release files for gpu-fit 0.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| gpu_fit-0.1.1.tar.gz | 12.5 kB | With provenance | Details for gpu_fit-0.1.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| gpu_fit-0.1.1-py3-none-any.whl | Python 3 | none | any | Details for gpu_fit-0.1.1-py3-none-any.whl |
Total release size: 23.9 kB
Release files / gpu_fit-0.1.1.tar.gz
| Download URL | gpu_fit-0.1.1.tar.gz |
|---|---|
| Size | 12.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
73ab1c61dd684abd53b5f736781b3181ca0b7eb6ad5e5eac03eac847d087448d
|
|
BLAKE2b-256 checksum How to use checksums |
44ad70c846d7338219ca662238ef5c74fa213596a572801323e21641ff8025b0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.
Transparency logRelease files / gpu_fit-0.1.1-py3-none-any.whl
| Download URL | gpu_fit-0.1.1-py3-none-any.whl |
|---|---|
| Size | 11.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
14002bc929fa9a028fd5a771fba01cb8d6bb67af2a56357d628cae5df2eb3ef1
|
|
BLAKE2b-256 checksum How to use checksums |
5b5086e38f871742a328349a1668bc53e85eda8b6e9045976167762d1a7c833e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.
Transparency logSecurity for gpu-fit 0.1.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for gpu_fit-0.1.1.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.