InferGrid

Tenant-fair LLM inference orchestration on a single GPU. No Kubernetes.

InferGrid is middleware that sits on top of vLLM/SGLang and gives a quiet user predictable TTFT even when a noisy neighbor is hammering the same shared engine. One pip install. No cluster. No YAML pile.

The hero number

Single A100-SXM4 80GB, Llama-3.1-8B, vLLM 0.19.1, two clients sharing one engine: a flooder at 32 RPS, a quiet user at 1 RPS, 300 s sustained.

Ten lines of YAML. No application code change. The quiet user is essentially unaware the flooder exists — within 14% of the no-contention baseline.

p99 figures exclude the first 10 s warmup window (vLLM JIT-compile transient, single-spike outlier; see CORRECTIONS C7). All 29 steady-state windows after t=10 s have quiet p99 between 36 ms and 65 ms.

Full writeup: results/gate2_preprint_v3/ and the original 5-arm experimental arc at results/gate2_fairness_20260419/ (vLLM 0.8.5, 120 s — newer steady-state numbers in v3 are more honest for current vLLM users).

What InferGrid is and isn't

InferGrid is a thin orchestration layer (~3,500 LOC) that sits between your app and vLLM/SGLang. It adds three things engines can't do internally:

1. Per-tenant token-bucket rate limiting at the budget gate. This is the load-bearing mechanism (validated empirically — see Gate 2-FAIRNESS results). Engines have no concept of a tenant; InferGrid does.
2. Multi-model lifecycle management on a single GPU — frequency+recency eviction, hot-swap routing, no-K8s.
3. OpenAI-compatible HTTP API in front of multiple engines, so your app code doesn't change.

It is not a vLLM/SGLang replacement. It runs them as subprocesses and proxies to them. It is not a Kubernetes alternative for datacenter workloads — Dynamo, llm-d, Mammoth do that better. It is not going to magically lower single-tenant TTFT — Gate 1.5 measured a robust DISCONFIRM there; vLLM's continuous batching matches a coarse upstream cap on single-tenant single-model workloads.

Why InferGrid

Every existing solution requires Kubernetes or gives up tenant fairness:

The "multi-tenant on a small shared box without K8s" cell is empty. That's the gap InferGrid fills.

Quickstart

pip install infergrid
infergrid serve --config configs/quickstart_fairness.yaml

# Wait until /health returns 200. The first call returns 503 with a
# {missing_models: [...]} body until the configured engines finish
# loading (typically 30-90s for an 8B-class model on A100).
until curl -fs localhost:8000/health > /dev/null; do sleep 2; done

# Hit it as two tenants on the same model:
curl localhost:8000/v1/completions -H "X-Tenant-ID: noisy" \
  -d '{"model":"llama31-8b","prompt":"...","max_tokens":64,"stream":true}'
curl localhost:8000/v1/completions -H "X-Tenant-ID: quiet" \
  -d '{"model":"llama31-8b","prompt":"...","max_tokens":64,"stream":true}'

# Watch the rate-limit fire and the engine queue stay composed:
curl localhost:8000/metrics | grep -E "tenant_rejected|admission_queue_depth"

The quickstart_fairness.yaml is heavily commented. For a deeper "when to use which lever" treatment, see the Tuning Guide — every recommendation in it traces back to a specific experiment.

Architecture

                    +---------------------+
                    |   WorkloadRouter    |  request profiling, length-bucket scheduling
                    +--------+------------+
                             |
              +--------------+--------------+
              |              |              |
     +--------v--------+ +--v---+ +--------v--------+
     |  CacheManager   | |Shared| | TenantManager   |  per-tenant token bucket + budget
     |  (KV lifecycle) | |State | | rate_limit_burst|
     +-----------------+ +------+ +-----------------+
              |                            |
     +--------v----------------------------v--------+
     |          vLLM / SGLang Engines (1-N)         |
     +----------------------------------------------+

Total: 2,872 LOC src + 2,400+ LOC tests (144 unit tests passing).

Empirical results

Each claim above traces to a specific experiment. Numbers are honest — TTFT measurement was rebuilt mid-project after a shadow review caught the original harness was timing SSE first-frame RTT, not first non-empty token (see results/CORRECTIONS.md C2/C5).

Total compute spend: ~$13. Full raw artifacts and per-window traces in results/.

Tests

pytest tests/unit/        # 144 tests, no GPU required, ~10 s

Roadmap

Research artifacts

• Gate 2-FAIRNESS OUTCOME + SUPPLEMENT (Arm 5b) + SUPPLEMENT (Arm 6)
• Gate 1.5 OUTCOME
• Gate 1 OUTCOME (with Little's Law caveat)
• Tuning Guide — when to use which lever, with empirical evidence
• Inference Orchestration Gap Analysis — competitive landscape (April 2026)
• Corrections / measurement honesty log — every metric we under-counted and the fix

Contributing

See CONTRIBUTING.md. Bug reports especially welcome with a prometheus_dump.txt and server.log — that's worth more to us than a star.

License

MIT