three-models-one-gpu

Three encoders share one 20 GB card, and none of the three knobs that decide whether that works does what its name suggests. gpubudget reads a Triton model repository, works out what it will actually hold, and simulates what happens to each model's tail latency once they are competing for the same device.

instance_group { count: 2 } reads like two servers. On one GPU it is two copies of the weights and a second share of the same silicon. max_batch_size: 32 reads like a ceiling; under the TensorRT provider it is a reservation you pay for whether or not the traffic ever produces a batch that big. max_queue_delay_microseconds reads like a batching hint; below a certain arrival rate it is a fixed addition to every single request. None of this fails loudly. The server starts, serves your smoke test, passes review, and shows up six weeks later as a p95 nobody can explain and an out-of-memory error that only happens on restart.

install

pip install gpubudget

No dependencies. Python 3.10 or newer.

the smallest thing that runs

gpubudget budget example

example is the three-model repository this README is about, shipped inside the package, so there is nothing to clone before the first answer. From a checkout, gpubudget budget model_repository/ is the same thing. Either prints:

| model            | inst | batch | weights  | activation    | total    |
|------------------|------|-------|----------|---------------|----------|
| bge_m3           | 1    | 64    | 1.06 GiB | 64 MiB (est)  | 1.17 GiB |
| siglip2_text     | 1    | 64    | 703 MiB  | 96 MiB (est)  | 847 MiB  |
| siglip2_vision   | 2    | 32    | 4.51 GiB | 864 MiB (est) | 5.45 GiB |
| process overhead |      |       |          |               | 300 MiB  |
| allocator slack  |      |       |          |               | 554 MiB  |
| total            |      |       |          |               | 8.28 GiB |

ceiling 18.40 GiB of 20.00 GiB on gpu, headroom 10.12 GiB, load peak 19.93 GiB
verdict: fits at rest, will not survive a cold start

Ten gibibytes of headroom and it still will not start. That gap between 8.28 and 19.93 is the whole point of the tool, and it is the subject of the next section.

gpubudget example --copy . writes those configs somewhere you can edit them, which is the point: change count or a batch ceiling and watch the numbers move.

The same thing from Python:

from gpubudget import Device, budget_table, estimate_budget, load_repository

from gpubudget import EXAMPLE_REPOSITORY

print(budget_table(estimate_budget(load_repository(EXAMPLE_REPOSITORY), Device.from_gib("card", 20))))

what the arithmetic actually is

Four terms, and only the first is obvious.

Weights, once per instance. An instance_group with count: 2 is two ONNX Runtime sessions. Each one copies the initialisers to the device. There is no sharing, and no warning anywhere that this is what you asked for. Here that is 4.51 GiB where the model on disk is 4.34 GiB in FP16.

Activation working set, sized by a batch that may never happen. TensorRT sizes an execution context from the optimisation profile, so trt_profile_max_shapes decides the reservation whatever the traffic does. The shipped vision config pins that at 32 while preferred_batch_size tops out at 8, which reserves 648 MiB for a batch the batcher cannot form. The ONNX Runtime CUDA arena behaves differently again: it grows to the largest batch it has served and never gives it back, so there the number that matters is the largest batch the batcher can form, not the one it usually does.

Process overhead. The CUDA context, plus the cuBLAS and cuDNN kernel images if CUDA_MODULE_LOADING is not LAZY. That last one is most of a gibibyte and it is set by an environment variable most people never look at.

Allocator slack. The arena rounds up and does not compact. The sum of what you asked for is not the sum of what is resident.

And then the load-time peak, which is larger than all of it and is where servers actually die. With a cold TensorRT cache the FP32 ONNX initialisers stay resident while the builder runs, the builder is allowed up to max_workspace_size_bytes on top, and Triton loads several models at once. Three models, two of them TensorRT, and the peak is 19.93 GiB against an 18.40 GiB ceiling — from a steady state that fits four times over. The fix is --model-load-thread-count=1, or building the engines somewhere with more memory and mounting the cache read-only. gpubudget budget exits non-zero when this happens, so it belongs in CI.

instance count is a priority setting, not a capacity setting

This is the part the documentation does not cover, and the reason the vision config in this repository carries a comment claiming count: 2 doubles throughput. It does not.

Instances are not servers. An instance is a slot that lets a batch be prepared concurrently; the kernels still land on one device, and two models that each fill the SMs do not run in parallel, they interleave. So a second instance of the heavy model does not add capacity. What it adds is a second share in how the device splits its time, taken from the models you did not change.

At a load the device can carry, with the text and multilingual encoders at 30 requests a second each and the vision tower at six:

siglip2_vision instances	siglip2_vision req/s	siglip2_text p50 ms	siglip2_text p95 ms	vram
1	6.1	54	89	5.36 GiB
2	6.1	59	109	8.28 GiB
3	6.1	61	126	11.19 GiB
4	6.1	62	140	14.10 GiB

Throughput does not move, because throughput is set by arrivals and the device was never the constraint. The interactive model's p95 rises by 57% and the bill is 8.7 GiB. Push the vision rate to twelve a second, close to what this configuration can serve, and the shape is the same: no extra throughput, text p50 from 63 ms to 120 ms.

The ceiling on what extra instances can ever buy is the fraction of a request that is not device time — deserialising, the copy on, building the response. Divide it out of your service curve before you spend gigabytes finding out.

the batch you configure is not the batch you get

preferred_batch_size is a wish. The batcher can only group requests that are in the queue at the same moment, and at interactive rates on a model that takes hundreds of milliseconds, they are not.

vision arrivals	mean batch	largest batch	req/s served	device busy	steady
2/s	1.0	2	2.1	63%	yes
6/s	1.1	4	6.1	81%	yes
12/s	1.9	8	12.2	97%	yes
20/s	7.8	8	18.2	100%	no
30/s	7.9	8	18.2	100%	no

Batches of eight only appear once the device is already past what it can serve. Every row where the configuration is viable forms batches of one or two. That is the number the memory reservation should be sized from, and it appears nowhere in the configuration file.

The queue delay follows the same logic from the other side:

max_queue_delay	p50 at 2/s	at 20/s	at 100/s	at 300/s
0 ms	27	27	47	100
0.5 ms	28	28	47	99
5 ms	32	32	47	99

At two requests a second a 5 ms delay is 5 ms of pure waiting for company that never arrives. At a hundred it is free, because by then the queue is never empty and the request would have waited anyway. The crossover is a property of utilisation, not of the batch size you asked for. The vision path in this repository started at 10 ms and every uncontended image search paid all of it.

Both tables come from bench/bench_frontier.py and are the scheduler model's output, not measurements of a GPU. Which brings us to whether the model is worth anything.

does the model agree with a real server

A queueing model that has only been compared against itself is a story. bench/bench_scheduler.py runs the same traffic twice: once through simulate(), and once over real HTTP against a server with real threads, a real socket, a real dynamic batcher and a single device held for the modelled service time. It runs on CPU in about four minutes, needs no GPU, and the results are in bench/results/scheduler-validation.json.

Across four configurations and three models, twelve comparisons:

quantity	median error	worst
p50, raw	9.0%	10.3%
p95, raw	6.1%	10.2%
p50, less measured overhead	4.1%	6.5%
p95, less measured overhead	3.7%	8.1%

The model is consistently optimistic, by a median of 7.9%, and the gap is a fixed 2.4 ms of HTTP and wakeup granularity that the model does not pretend to include. It is a bias, not scatter — which matters, because the model is for comparing configurations against each other, and a constant offset cancels.

Finding that number took two runs. The first reported 45 ms of overhead, which was not overhead: the test server wrote its headers and its body as two separate sends, and the second one sat waiting for a delayed ACK. Forty milliseconds, flat, on every idle request over loopback, looking exactly like a slow model. If you write your own harness, set disable_nagle_algorithm before you believe anything it tells you.

measuring your own hardware

Everything above uses profiles/example-20gb.json, which is illustrative and says so in its own notes field. Two of its numbers are anchored to a real 20 GB Ada-generation card; the rest is arithmetic from parameter counts. Do not size a machine from it.

gpubudget bench example --url http://localhost:8000 --out profiles/mine.json
gpubudget simulate model_repository/ --profile profiles/mine.json \
    --traffic siglip2_text=30 --traffic bge_m3=30 --traffic siglip2_vision=6
gpubudget plan model_repository/ --profile profiles/mine.json --protect siglip2_text \
    --traffic siglip2_text=30 --traffic bge_m3=30 --traffic siglip2_vision=6

bench sweeps batch sizes sequentially on an idle device and fits each model to an intercept and a slope. The intercept is what a batch costs before it has any samples in it, the slope is the marginal sample, and the ratio between them tells you whether batching is worth any queue delay at all. plan enumerates instance counts and batch ceilings, drops the ones that will not fit or will not keep up, and ranks what is left by the tail of the model you nominate.

bench/bench_triton.py does all three phases against a live server and writes a measured-against-modelled table. Nothing in this repository ships results from it, because somebody else's GPU is not useful to you.

There is no GPU here either way:

gpubudget serve model_repository/ --profile profiles/example-20gb.json --port 8000
gpubudget soak model_repository/ --url http://localhost:8000 \
    --traffic siglip2_text=30 --traffic siglip2_vision=6 --rates 1,2,3

serve runs a server that speaks the KServe v2 protocol with synthetic timings, which is enough to exercise the harness end to end.

what did not work

A closed-loop harness. The first version ran N threads in a loop, each sending the next request when the last reply came back. It cannot produce a queue: when the server slows down the harness slows down with it, and the tail flattens exactly where the real one takes off. The harness here fixes the send schedule before the run starts and times every request from when it was scheduled, not from when a worker got to it. It reports the difference, and refuses to pretend the numbers are about the server when they are about itself.

JSON tensors. A batch of four 384×384 images is 1.7 million floats, and json.dumps on that takes longer than the inference. A harness on the JSON path measures its own encoder and concludes that batching does not help, which is the opposite of the truth. The client here implements the binary tensor extension, which is about forty lines and the difference between a benchmark and a fiction.

TensorRT for the multilingual encoder. The ONNX export fuses attention into MultiHeadAttention, SkipLayerNormalization and BiasGelu, which are ONNX Runtime contrib operators with no TensorRT plugin. The provider partitions around them, builds an engine per fragment, and ends up slower than the provider it replaced while using more memory. It stays on the CUDA provider, and the tool warns if you try.

CUDA graphs on the vision tower with count: 2. Each capture pins its own memory pool, and capturing concurrently on an engine that size takes the server down during model load rather than under traffic. The saving was single digit milliseconds on a call of over a hundred. It is off, and the budget warns about the combination.

Trusting a GPU-name lookup table for capacity. The same part number ships with different clocks and different memory, and the driver changed its allocator between releases. Measure the card in front of you; that is what bench is for.

failure modes this is looking for

Over the ceiling at steady state does not fail cleanly. The server starts, serves single requests, and returns 500s only on the batches large enough to need the memory that is not there — so the error rate correlates with your traffic and nothing reproduces in staging.

Over the ceiling at load time fails cleanly but only on a cold cache, which means it works in development, works on the first deploy after you warmed the cache by hand, and fails on the node that comes up in the middle of the night.

TensorRT with dynamic input dimensions and no pinned optimisation profile builds a fresh engine for every distinct sequence length it sees and keeps all of them. Resident memory climbs for hours and then plateaus somewhere nobody predicted. The multilingual encoder here has dims: [-1], which is the worst case for this, and it is a second reason it is not on TensorRT.

dynamic_batching with no preferred_batch_size falls back to max_batch_size. One burst forms a batch that big, the arena grows to fit it, and it stays that way until the server restarts.

And the one that is not a failure mode so much as a wasted week: a run where the device is more than about 95% busy. Past that the queues never drain and every percentile you measure is a property of how long you ran the benchmark. Both the simulator and the load harness say so rather than printing a number.

what this does not model

Multi-GPU, MIG and MPS. Sequence models with a KV cache, where the activation estimate is wrong by a lot rather than by a third. Ensembles and the BLS backend, where one request fans out into several. Pipeline parallelism. And the device model is an approximation either way: kernels from different streams neither serialise completely nor overlap freely, so simulate() offers shared and exclusive and you should run both and see which tracks your hardware.

The activation estimator is shape arithmetic and is worth about ±30%. It is marked (est) in every table until you replace it with a measurement, and the tool says so in its warnings rather than letting you find out.

layout

src/gpubudget/     pbtxt parser, budget, scheduler, load harness, cli
src/gpubudget/data/  the same configs and profile, shipped in the wheel as `example`
model_repository/  the three configs, with weights.json so CI can budget them
profiles/          calibration files; the shipped one is illustrative
bench/             the scripts that produced every number above
tests/             144 tests, no network

licence

Apache-2.0.