Skip to main content

gpu-fit

Will this LLM fit on this GPU, at this context length, for this many users?

gpu-fit answers that before you rent the GPU. It reads the model's config and weight sizes from Hugging Face and works out vLLM's memory budget: weights, runtime overhead, KV cache, how many requests fit at full context, and the longest single request. Then it prints the vllm serve command.

$ gpu-fit Qwen/Qwen3-32B --context 32k --concurrency 4

  GPU          smallest setup    requests at full ctx     KV tokens
  L4           8× L4                             10.6       347,888
  L40S         4× L40S                           11.6       380,656
  A100-80      2× A100-80                         9.9       323,484
  H100         2× H100                            9.8       320,885
  H200         1× H200                            7.9       259,428
  ...
$ gpu-fit deepseek-ai/DeepSeek-V3 --gpu H200 --tp 8 --context 128k

  vLLM budget per GPU             126.4 GiB
    weights                        80.1 GiB
    runtime overhead (assumed)      2.0 GiB
    left for KV cache              44.2 GiB
  KV cache capacity              675,671 tokens
  Requests at full context       5.2
  Longest single request         163,840 tokens

✓ Fits. Suggested command:
  vllm serve deepseek-ai/DeepSeek-V3 --max-model-len 131072 --tensor-parallel-size 8 --max-num-seqs 5

Install

pip install gpu-fit        # no dependencies, Python 3.10+

Why another VRAM calculator?

Most calculators multiply parameters by bytes and add a guess. Modern models break that:

Architecture What changes gpu-fit
Grouped-query attention KV per token depends on KV heads, not attention heads ✓
Hybrid (linear attention + full attention, e.g. Qwen3-Next family) Only full-attention layers grow with context; the rest hold fixed state per request ✓
MLA (DeepSeek) Compact latent KV, but copied to every tensor-parallel GPU ✓
Sliding window (Gemma, Mistral) Window layers stop growing past the window ✓
FP8 / AWQ / GPTQ checkpoints Exact weight size from the safetensors metadata ✓
Tensor parallel beyond KV heads KV heads get replicated, not split ✓
FP8 KV cache Doubles capacity: --kv-cache-dtype fp8 ✓

Options

gpu-fit MODEL [--gpu L40S | --gpu-mem-gib 48] [--tp 2] [--context 128k]
              [--concurrency 8] [--kv-cache-dtype fp8] [--util 0.9]
              [--overhead-gib 2.0] [--json]
  • MODEL: a Hugging Face id, or a local folder with config.json and *.safetensors. Gated models need HF_TOKEN in the environment.
  • Leave out --gpu to compare every GPU in the catalog with the smallest tensor-parallel size that works.
  • --json for scripts and CI.

How the maths works

vLLM takes gpu_memory_utilization × GPU memory, loads the weights, profiles a forward pass (activations, CUDA graphs), and gives the rest to the KV cache. gpu-fit mirrors that:

KV pool   = memory × utilization − weights / tp − overhead
KV/token  = 2 × full-attention layers × KV heads per GPU × head dim × dtype bytes
requests  = KV pool / (context × KV/token + sliding-window KV + recurrent state)

Accuracy, honestly

  • Weights and KV per token come straight from the config, so they're exact.
  • Runtime overhead (activations, CUDA graphs) is an assumption: 2 GiB per GPU by default. It grows with batch size and model width. Override it with --overhead-gib.
  • Hybrid models: vLLM pads KV pages to hold the recurrent state, so real capacity can be lower than shown.
  • Treat the output as a planning estimate. Confirm on the real GPU before you commit money.

Help make it exact: share what vLLM printed on your hardware (see CONTRIBUTING.md). Every measurement improves the defaults for everyone.

Roadmap

  • Measured overhead table from community results
  • Throughput estimate (tokens/s) from memory bandwidth
  • SGLang and TensorRT-LLM memory models
  • Web version
  • Vision encoder and LoRA memory

License

MIT

Metadata

Release files for gpu-fit 0.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for gpu-fit 0.1.1
File Size Uploaded Provenance Actions
gpu_fit-0.1.1.tar.gz 12.5 kB With provenance Details for gpu_fit-0.1.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for gpu-fit 0.1.1
File Interpreter ABI Platform Actions
gpu_fit-0.1.1-py3-none-any.whl Python 3 none any Details for gpu_fit-0.1.1-py3-none-any.whl

Total release size: 23.9 kB

Release files / gpu_fit-0.1.1.tar.gz

Download URL gpu_fit-0.1.1.tar.gz
Size 12.5 kB
Tags Source
SHA-256 checksum
How to use checksums
73ab1c61dd684abd53b5f736781b3181ca0b7eb6ad5e5eac03eac847d087448d
BLAKE2b-256 checksum
How to use checksums
44ad70c846d7338219ca662238ef5c74fa213596a572801323e21641ff8025b0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.

Transparency log

Release files / gpu_fit-0.1.1-py3-none-any.whl

Download URL gpu_fit-0.1.1-py3-none-any.whl
Size 11.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
14002bc929fa9a028fd5a771fba01cb8d6bb67af2a56357d628cae5df2eb3ef1
BLAKE2b-256 checksum
How to use checksums
5b5086e38f871742a328349a1668bc53e85eda8b6e9045976167762d1a7c833e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.

Transparency log

Security for gpu-fit 0.1.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for gpu_fit-0.1.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.1 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page