velocityllm 0.4.4

---

What is VelocityLLM

LLM servers fail in a predictable way under load. Requests pile up in a queue, latency grows, and eventually clients time out after the work has already been spent. A static batch size makes this worse: it wastes GPU capacity when traffic is light and overloads the system when traffic spikes.

VelocityLLM is a scheduler that sits in front of an LLM engine and controls three things: which requests are admitted, the order they run in, and how much concurrent work the system accepts. The goal is to keep latency within a service-level agreement (SLA) during bursts, and to decline work early when it cannot be finished in time, rather than accepting it and failing later.

The scheduler only looks at request metadata such as arrival time, token counts, and priority. It never inspects model internals, so it works with any model the underlying engine supports.

Features

SLA-aware admission control. Each incoming request is checked against a predicted completion time before it enters the queue. If the request cannot meet its deadline, it is rejected immediately with HTTP 429 and a retry hint. The client learns right away instead of waiting for a timeout.

Priority scheduling with anti-starvation. Requests are ordered as high, normal, or low priority. A request's effective priority rises the longer it waits, so low-priority traffic is delayed under load but is never blocked indefinitely.

Adaptive concurrency. An AIMD (additive increase, multiplicative decrease) controller raises the concurrency limit while latency and memory headroom are healthy, and cuts it sharply when they are not. This tracks the workload without manual tuning.

Failure handling. GPU out-of-memory errors trigger an emergency throttle and recovery rather than a crash. Client disconnects, malformed input, and sudden bursts are handled explicitly, and the server keeps serving other requests.

Automatic hardware and model detection. The NVIDIA GPU is detected automatically. When no GPU is present, the simulated engine runs so that the full pipeline can be exercised on any machine. The model is located from a command-line flag, an environment variable, a local models folder, or an interactive prompt.

Built-in load testing and benchmarks. A load generator supports four arrival patterns, and comparison tools measure static and dynamic policies under identical workloads.

Installation

VelocityLLM requires Python 3.11 or newer.

pip install velocityllm

Installing with the GPU extra adds vLLM, the inference engine used for real models on NVIDIA hardware:

pip install "velocityllm[gpu]"

Confirm the installation:

velocityllm version
velocityllm detect

The detect command reports which GPU was found and which model the server will use, so you can check the setup before starting anything.

Quick start

Start the server with the simulated engine. This needs no GPU and no model download, so it works on any machine:

velocityllm chargeVelocity --mock --port 8000

In another terminal, check that it is running:

curl http://localhost:8000/health

Send a request:

curl -X POST http://localhost:8000/generate \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Explain request scheduling in one sentence", "max_tokens": 64}'

The response includes the generated text, the time spent queued and executing, the tokens per second, and whether the SLA was met.

To serve a real model on a GPU machine, point VelocityLLM at the model folder:

velocityllm chargeVelocity --model-path /path/to/model

Or set it once in the environment:

export VELOCITY_MODEL_PATH=/path/to/model
velocityllm chargeVelocity

Model and GPU detection

When you run chargeVelocity or serve, VelocityLLM resolves the model in this order:

1. The --model-path flag, if given.
2. The VELOCITY_MODEL_PATH environment variable, if set.
3. The first folder containing a config.json file, searched in ./models and then ~/velocity-models.
4. If none of the above applies and you are in an interactive terminal, a prompt asks you to type the path.

GPU detection uses nvidia-smi. If it finds an NVIDIA GPU, the server reports the GPU name and memory at startup and uses the real engine. If it finds no GPU, the server prints a notice and switches to the simulated engine automatically, so nothing needs to be configured.

Commands

Command	Purpose
velocityllm chargeVelocity	Start the server with automatic GPU and model detection
velocityllm serve	Run the scheduler server with full control over options
velocityllm detect	Show the detected GPU and model without starting a server
velocityllm loadgen	Send test traffic to a running server
velocityllm version	Show the installed version

Run velocityllm <command> --help for the full option list of any command.

Scheduling policies

The --policy option selects how the server schedules work:

Policy	Behavior
dynamic (default)	Adaptive concurrency with SLA-aware admission and priority scheduling
static	Fixed concurrency. Used as the baseline for comparing scheduler performance
smart	Dynamic scheduling with the extended smart-scheduler settings

Server options

Option	Description
--mock	Use the simulated engine. No GPU required
--model-path PATH	Folder containing the model weights
--host HOST	Address to bind to. Default 0.0.0.0
--port PORT	Port to listen on. Default 8000
--sla-ms MS	Target latency deadline in milliseconds. Default 3000
--max-concurrency N	Upper limit for the adaptive controller. Default 32
--api-key KEY	Require the X-API-Key header on every request except /health
--config FILE	Load settings from a YAML file. Command-line flags take precedence
--structured-logs	Write JSON log lines for production log collection

Configuration file

Long-running deployments are easier to manage with a configuration file:

policy: dynamic
model_path: ""          # empty means auto-detect
port: 8000
sla_ms: 8000
max_concurrency: 32
max_model_len: 4096
api_key: change-me

Start the server with it:

velocityllm serve --config config.yaml

API

Method	Endpoint	Description
GET	/health	Server status, active policy, and backend in use
POST	/generate	Submit a prompt and receive the response with timing data
GET	/stats	Scheduler statistics, including admissions, rejections, and queue state

A successful /generate response has this shape:

{
  "request_id": "req-c7a3b7997d85",
  "response": "...",
  "latency_seconds": 0.196,
  "queue_time_seconds": 0.002,
  "execution_time_seconds": 0.194,
  "tokens_generated": 14,
  "tokens_per_second": 72.1,
  "sla_met": true,
  "priority": "normal"
}

When the system cannot meet a request's SLA, it returns HTTP 429 with a retry hint instead of accepting work it cannot finish.

Load testing

The load generator sends traffic to a running server using one of four arrival patterns:

velocityllm loadgen --url http://localhost:8000 --num-requests 200 --concurrency 16 --pattern poisson

Pattern	Traffic shape
flood	All requests sent at once
constant	Steady request rate
poisson	Random arrivals with a constant average rate
burst	Periodic spikes separated by quiet periods

Use the same pattern against --policy static and --policy dynamic to see how each scheduler handles the same workload.

Requirements

• Python 3.11 or newer
• An NVIDIA GPU with CUDA drivers, for real model inference (optional)
• The [gpu] extra, which installs vLLM (optional)

Without a GPU, every feature except real model inference works, using the simulated engine.

Links

• Source code and full documentation: https://github.com/kartikpagariya25/VelocityLLM
• Issues and feature requests: https://github.com/kartikpagariya25/VelocityLLM/issues
• License: MIT