inferris-lite

Executable mini-forms of LLM serving mechanisms — continuous batching, token budgets, chunked prefill, paged state, preemption, prefix caching, cache-aware scheduling, speculative decoding, the executor boundary, and rank parallelism — as one dependency-free Python state machine that prints what it is thinking, step by step.

Each mechanism is a mechanism note first (the problem, the invariant, the smallest state machine that expresses it) and an implementation second. The code grows mechanism by mechanism: m00_engine_loop through m11_parallelism, each module the complete engine as of its mechanism, each pinned by one test per line of its note. Reading order is the module order; import m11_parallelism is the whole ladder.

Install

pip install inferris-lite

Python >= 3.12, zero runtime dependencies.

Watch it think

inferris-lite demo

Three requests — staggered arrivals, chunked prefill, page contention, a scripted speculative verdict — on a rank-4 executor group:

step   0  scheduled=['r0']  executed={'r0': 6}
step   1  scheduled=['r0', 'r1']  executed={'r0': 3, 'r1': 4}  accepted={'r0': 2}  finished={'r0': 'LENGTH'}
step   2  scheduled=['r1', 'r2']  executed={'r1': 2, 'r2': 6}  accepted={'r1': 1}  finished={'r1': 'LENGTH'}
...
r2: FINISHED (LENGTH)  generated=5  computed=14

inferris-lite mechs lists every mechanism and its key question. Your own workloads go through run:

inferris-lite run --request a:16:8 --request b:16:8:1 --spec-len 3 --world-size 2
inferris-lite run --spec workload.json --json

A JSON spec is an engine object (max_step_tokens, page_size, num_pages, spec_len, world_size) plus a requests list (request_id, prompt_len or explicit tokens, max_new_tokens, arrival_step, eos_after, accept_script). Explicit shared tokens prefixes are how you make the prefix cache light up in the reused column. --json emits one object per step with every trace field, then a terminal summary.

As a library:

import m11_parallelism as m11

requests = [m11.Request("r0", list(range(6)), 4)]
engine = m11.Engine(requests, max_step_tokens=8, page_size=4, num_pages=8, world_size=2)
for trace in engine.run():
    print(trace.step, trace.scheduled, trace.executed, trace.accepted)

Every earlier mechanism is importable the same way — m03_paged_state before pages met prefix reuse, m09_speculative_decode before the executor boundary was cut — so you can diff neighbouring mechanisms to see exactly what one mechanism adds.

What this is, and is not

Tokens are integers. Execution is bookkeeping: the executor answers with verdicts and page writes, not logits. Ranks are logical and run in lockstep. There is no model, no tokenizer, no server — and that is the point: what runs here is the mechanism semantics themselves, isolated from implementation accident and performance optimization, small enough to read in an afternoon and precise enough to break when you change an invariant.

inferris-lite is the specification layer of inferris, a Rust inference engine; the mechanism notes live in the repo under lite/mech/.