mini-vllm

A minimal implementation of vLLM's core ideas: PagedAttention and continuous batching.

Installation

pip install mini-vllm

Requirements: Python 3.10+, CUDA-capable GPU

Quick Start

from mini_vllm import LLMEngine

# Initialize the engine
engine = LLMEngine(
    model_name="meta-llama/Llama-3.2-1B",
    block_size=16,
    num_gpu_blocks=100
)

# Add a request
req_id = engine.add_request("The meaning of life is")

# Generate tokens
while True:
    outputs = engine.step()
    if not outputs:
        break
    
    # Check if generation is complete
    if req_id in outputs:
        print(outputs[req_id])

Benchmarks

Hardware: NVIDIA A100 (Modal)
Model: meta-llama/Llama-3.2-1B
Max tokens per request: 50
Prompt: "The meaning of life is"

mini-vllm Performance

Comparison with vLLM

References

• vLLM repo
• vLLM paper
• continuous-batching-llm-inference
• paged-attention-minimal