realtime_engine — a standalone realtime inference engine

A standalone realtime inference engine, driven entirely through inbox/outbox queues — so any caller matching the contract can run it.

This follows the design's build order: prove the engine on its own first (the queue boundary gives isolation now, and a path to a process/IPC split later without a process boundary today). Full design write-up lives in Notion.

Layout

Standard src/ layout: src/realtime_engine/ (package) + tests/ + pyproject.toml.

Develop & test

cd backend/realtime_engine
pip install -e ".[dev]"
ruff check src tests && black --check src tests && isort --check-only src tests && mypy src/realtime_engine
pytest -q

Tests cover: distinctness (co-resident sessions differ), invariance (a session is identical solo vs batched), admission (over-capacity attach raises AdmissionError), finalize, and output shape — all with the engine running standalone. Lint/type/test also run in CI (.buildkite/pipeline.yml).

Contract recap

• Level 1 (every engine): manifest / attach / step(due) -> [StepResult] / finalize / detach. step is batched (one StepResult per due session).
• Level 2 (the loop): optional serve(inbox, outbox). Omit it → default_serve. Implement it → own loop (e.g. an sglang-style scheduler).
• Conditioning arriving via Action is action-class (already client→action-translated by the caller); the engine stays ignorant of client semantics.

Next (not built yet)

• A real GPU engine impl behind the same contract (Waypoint stages on a ComposedEngine).
• The runtime caller: a ReactorModel that pumps the inbox (state → Action) and routes the outbox into streaming.
• Open validation item (flagged in review): measure whether the outbox queue adds latency vs a direct callback.