VelocityLLM

An API server that sits in front of an LLM engine and decides which requests to accept, in what order, and how many to run at once, so that answers arrive inside a time promise (an SLA) instead of everything slowing down together.

It ships with:

• static: a fixed number of requests at a time, first come first served. A simple baseline.
• dynamic: watches the waiting line, the time promise and GPU memory, changes how many requests run at once, and says "no" early (with a reason) when an answer cannot arrive in time.
• A mock engine, so you can try everything on a normal computer with no GPU.
• Test-traffic tools and a benchmark service for comparing the policies.

Install

pip install velocityllm

This installs the server, the mock engine and the tools. It does not install vLLM, because vLLM needs an NVIDIA GPU. For real models:

pip install "velocityllm[gpu]"

Requires Python 3.11 or newer.

Quick start (no GPU)

velocityllm serve --policy dynamic --mock --port 8000

In a second terminal:

curl -X POST http://localhost:8000/generate \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Hello", "max_tokens": 20}'

curl http://localhost:8000/stats

The mock engine returns made-up text at a realistic speed. It is for trying the scheduler, not for real answers.

Real model on a GPU

pip install "velocityllm[gpu]"
velocityllm serve --policy dynamic --model-path /path/to/model --sla-ms 8000

You need an NVIDIA GPU and the model files on disk (a folder with config.json and the weights).

Commands

Add --help to any command to see all options.

API

A request the scheduler cannot serve in time is refused with HTTP 429 and a reason.

Good to know

• Results depend on your GPU, model and traffic. Test on your own setup before relying on any policy.
• Only a normal computer is needed for the mock engine; the real-GPU path needs NVIDIA hardware and model weights.
• This is an early release (version 0.1).

Source and issues

https://github.com/kartikpagariya25/VelocityLLM