smol-vllm 0.1.0

From-scratch paged-attention inference engine: paged KV cache, continuous batching, preemption

smol-vllm

From-scratch paged-attention inference engine: paged KV cache, continuous batching, preemption. Pure Python, no external deps.

Install

pip install smol-vllm

Or from source:

pip install .

Usage

from smol_vllm import LLMEngine

engine = LLMEngine(num_gpu_blocks=64, block_size=16, max_batch_size=8)

# Single request (streaming)
for token in engine.generate([1, 2, 3, 4, 5], max_tokens=20):
    print(token, end=" ")

# Batched: add requests and step
engine.add_request([10, 20, 30], max_tokens=10)
engine.add_request([40, 50, 60], max_tokens=10)
while True:
    outputs = engine.step()
    for out in outputs:
        print(out.output_tokens)
    if all(o.finished for o in outputs):
        break

Demo

pip install smol-vllm
smol-vllm-demo

License

MIT