sched-sim ⏱️

A high-fidelity, CPU-first discrete-event simulator (DES) for LLM serving schedulers.

In modern inference clusters (vLLM, TGI, TensorRT-LLM, SGLang, DistServe), the scheduling policy is the single largest determinant of cluster throughput, latency, and hardware cost. While continuous batching and Paged KV-caching provide the mechanism for dynamic execution, the scheduler determines:

1. Which requests enter the next forward iteration.
2. How large prompt prefills are chunked to prevent decode starvation.
3. What happens when the physical KV-cache runs out of memory (swapping to CPU DRAM vs eviction and recomputation).
4. When to disaggregate prefill and decode phases onto specialized GPU worker pools (Splitwise / DistServe architecture).

Testing these policies on real GPU clusters (e.g. 8x H100 nodes) is slow, noisy, and expensive ($20–$50/hr). sched-sim simulates hours of multi-GPU serving traffic in seconds on any standard CPU, using validated analytical performance models calibrated against empirical silicon measurements from servebench and mini-infer.

⚡ Core Architecture

                          ┌────────────────────────────────────────────────────────┐
                          │                   Arrival Process                      │
                          │   (Poisson, Bursty Gamma, LMSYS / ShareGPT Traces)    │
                          └───────────────────────────┬────────────────────────────┘
                                                      │ Request arrives (prompt_len, output_len)
                                                      ▼
                                       ┌─────────────────────────────┐
                                       │       Waiting Queue         │
                                       │  (FCFS, SJF, Priority, SLO) │
                                       └──────────────┬──────────────┘
                                                      │ Iteration Scheduler
                                                      ▼
                       ┌──────────────────────────────────────────────────────────────┐
                       │               Continuous Batching Step (Iteration)           │
                       │  ┌────────────────────────────┐ ┌──────────────────────────┐ │
                       │  │   Prefill Tokens (Chunked) │ │ Decode Tokens (1/req)    │ │
                       │  └────────────────────────────┘ └──────────────────────────┘ │
                       │       ▲                                      ▲               │
                       │       │ Paged KV Allocator                   │ Memory Check  │
                       │       │                                      │               │
                       │       ▼                                      ▼               │
                       │  ┌──────────────────────────┐   ┌──────────────────────────┐ │
                       │  │   Paged KV-Cache Pool    │   │   Preemption Subsystem   │ │
                       │  │ (Physical 16-tok Blocks) │   │ (Recompute vs CPU Swap)  │ │
                       │  └──────────────────────────┘   └──────────────────────────┘ │
                       └──────────────────────────────────────────────────────────────┘

⏱️ Quickstart in 30 Seconds

Install

# Direct from source with uv (fastest)
git clone https://github.com/Prajwal7-2002/sched-sim.git
cd sched-sim
uv sync

# Or with standard pip
pip install -e .

1. Run a Simulation Scenario

Simulate Chunked Prefill serving a long-context RAG workload on an NVIDIA A100 SXM4 (80GB) cluster running Llama-3-8B:

sched-sim run --scheduler chunked --workload rag --hardware a100 --duration 10

Runs 100% on your CPU: You do not need an A100 or any GPU to run sched-sim. It simulates hours of multi-GPU serving traffic in seconds on any standard CPU. The --hardware flag tells the discrete-event engine which cluster architecture to mathematically simulate (a100, h100, t4, rtx3050, or a custom JSON profile).

2. Compare Schedulers Side-by-Side (Identical Trace)

Benchmark all 5 scheduling policies on the exact same Poisson arrival trace:

sched-sim compare --schedulers fcfs,sjf,chunked,slo,disaggregated --workload rag --hardware a100

3. Replay Real-World Conversation Traces

Replay Chatbot Arena conversation traces through the simulator:

sched-sim trace ./traces/lmsys_sample.json --scheduler chunked

4. Sweep Load to Discover Saturation Knees

Discover the exact request rate where each scheduler hits its Goodput cliff:

sched-sim sweep --schedulers fcfs,chunked,slo,disaggregated --workload rag --min-rate 2 --max-rate 18 --step 4

5. Check Silicon Hardware Presets

Inspect calibrated hardware profiles:

sched-sim info

💡 Runnable Python Examples

Explore self-contained runnable scripts under examples/:

• examples/01_radix_prefix_caching.py: SGLang-style Radix tree prefix caching, prompt matching, and zero-copy block sharing.
• examples/02_simulate_schedulers.py: Side-by-side simulation comparing FCFS vs Cache-Aware scheduling on shared RAG contexts (69.5% TTFT reduction).
• examples/03_perfetto_trace_export.py: Programmatic execution and Catapult JSON trace export for ui.perfetto.dev.

python examples/01_radix_prefix_caching.py
python examples/02_simulate_schedulers.py
python examples/03_perfetto_trace_export.py

🔬 The 6 Core Schedulers

🎨 Interactive Perfetto / Chromium Timeline Visualization

Export native Chromium / Catapult Trace Event JSON files and inspect discrete forward-pass iteration slices and per-request lifecycles in ui.perfetto.dev:

sched-sim run --scheduler cache_aware --workload rag --hardware a100 --trace-out trace.json

Drag and drop trace.json directly into Perfetto to visually analyze:

• Prefill vs. Decode vs. Mixed Slices: Microsecond-accurate forward pass durations on the GPU accelerator.
• Request Streams: Exact arrival, TTFT first-token timestamps, and completion times.
• Counter Tracks: Real-time GPU KV-cache memory utilization (%) and concurrent batch stream counts.

🛠️ Hardware Calibration (Analytical Roofline Model)

sched-sim avoids arbitrary delays. It computes exact step latencies from first-principles roofline mechanics calibrated with empirical silicon data:

1. Prefill Latency ($T_{\text{prefill}}$): Compute-bound matrix multiplication: $$T_{\text{prefill}} = \max\left(\frac{2 \cdot P \cdot N}{\text{Effective TFLOPS}}, \frac{\text{Weights}}{\text{Bandwidth}}\right) + T_{\text{launch}}$$

2. Decode Latency ($T_{\text{decode}}$): Memory-bandwidth bound weight and KV-cache gather: $$T_{\text{decode}} = \frac{P \cdot \text{dtype} + L_{\text{total}} \cdot \text{KV_bytes_per_token}}{\text{DRAM Bandwidth (GB/s)}} + T_{\text{launch}}$$

3. Interconnect Transfer ($T_{\text{transfer}}$): Network serialization delay for PD Disaggregation: $$T_{\text{transfer}} = \frac{L_{\text{prompt}} \cdot \text{KV_bytes_per_token}}{\text{Network Bandwidth}} + T_{\text{net_latency}}$$

🔬 Empirical Silicon Validation (AMD Ryzen 5 6600H + NVIDIA RTX 3050 6GB)

sched-sim is validated against ground-truth benchmarks collected by servebench testing Qwen 2.5 0.5B on both C++/CUDA and Python runtimes on physical silicon:

Run the reproducible validation suite yourself:

uv run python benchmarks/validate_flagships.py

Full methodology and analysis available in benchmarks/RESULTS.md.

📊 Telemetry & Reports

Every run outputs comprehensive telemetry:

• Latency Percentiles: TTFT (p50, p90, p99 ms), TPOT (p50, p90, p99 ms/tok), End-to-End latency.
• Goodput Ratio: Exact percentage of requests meeting both TTFT and TPOT Service Level Objectives.
• Preemptions & Recomputations: Total memory thrashing events and discarded tokens.
• Visual Gantt Timeline: Live ASCII/Unicode GPU iteration breakdowns.
• Export: JSON (--output report.json) and Markdown (--output report.md).

🧪 Testing

Run the deterministic test suite:

uv run pytest

🤝 Contributing

Contributions from systems engineers and researchers are welcome! Check out the Contributing Guide for architectural invariants, testing standards, and high-impact areas where we need help (new scheduling policies, modern hardware presets, and interconnect topologies).

📄 License

Apache 2.0. See LICENSE for details.