Innards

Innards runs open-weight LLMs and shows, for every message, what happens inside: tokens, context growth, KV cache per layer, memory and timing. It predicts KV-cache size from the model config, measures the real cache, and explains what each attention design (MHA, GQA, MLA, sliding window, hybrid) changed from its predecessor. Observation runs in background threads beside generation, so the model generates exactly as it would without Innards.

Install

pip install innards            # core: KV calculator, lineage, schema, console sink
pip install "innards[hf]"      # + torch, transformers, psutil for in-process Hugging Face models

uv add innards                 # inside a uv project
uv add "innards[hf]"
uv pip install "innards[hf]"   # into an existing environment

The core install has two small dependencies (httpx, huggingface-hub) and works on any Python 3.10+. The hf extra adds PyTorch and Transformers.

Quickstart

Chat with a model and get one record per message:

from innards import Session

with Session(
    "Qwen/Qwen3-0.6B", max_new_tokens=48, sinks=["console"], chat_template_kwargs={"enable_thinking": False}
) as s:
    turn = s.chat("Explain paged attention in 3 lines")
    print(turn.output.text)
    print(turn.tokens.context_after, turn.kv.measured_bytes, turn.kv.predicted_bytes)

    turn = s.chat("Now in one line")
    print(turn.tokens.cached, "prompt tokens reused from the KV cache")

Send the same messages to models with different attention designs and compare:

from innards import compare

results = compare(
    models=["gpt2", "Qwen/Qwen3-0.6B"],
    script=["What is a KV cache?", "Why does it grow with context?"],
    max_new_tokens=16,
)
print(results.table())

Predict KV size without loading weights:

from innards.kv import predict

p = predict("Qwen/Qwen3-0.6B", context_tokens=32768, kv_dtype="float16")
print(p.formula)  # 2 × 28 layers × 8 KV heads × 128 head dim × 2 B = 114,688 B/token
print(p.kv_bytes)  # 3758096384

Imports

Functions

Session

Session(model: str, backend: str = "hf", strategy: str = "full_history",
        sinks: Iterable[str | Sink | Callable] | None = None, *, system: str | None = None,
        observe: bool = True, max_new_tokens: int = 256, record_timeout: float = 30.0,
        queue_size: int = 4096, run_id: str | None = None, session_id: str | None = None,
        load: bool = True, **backend_kwargs) -> Session

• model: Hugging Face Hub repo id or local path.
• backend: "hf" (in-process Transformers).
• strategy: "full_history": every turn is kept, and the KV cache from earlier turns is reused.
• sinks: where finished records go, e.g. ["console"]. Records are also returned by chat.
• system: optional system prompt.
• observe: False turns observation off entirely (no events, no worker threads).
• max_new_tokens: generation budget per turn (greedy decoding).
• record_timeout: how long chat waits for the background record after generation has finished.
• **backend_kwargs: for hf: device ("cpu", "cuda", "mps"), dtype, chat_template_kwargs, generation_kwargs, revision, token, trust_remote_code.

Returns a session bound to one model. Use it as a context manager, or call close().

from innards import Session

s = Session("gpt2", device="cpu", max_new_tokens=16)
print(s.backend.spec.architecture)  # mha
s.close()

Session.chat

Session.chat(message: str, *, timeout: float | None = None) -> TurnRecord

• message: the user message.
• timeout: seconds to wait for the background record (default: record_timeout).

Returns the TurnRecord (also available as Turn) with output text, tokens, KV, memory and timing.

from innards import Session

with Session("gpt2", max_new_tokens=16) as s:
    turn = s.chat("The key-value cache")
    print(turn.kv.cache_tokens, turn.kv.measured_bytes == turn.kv.predicted_bytes)  # e.g. 23 True

Session.stream and Session.turn_metrics

Session.stream(message: str) -> Iterator[str]
Session.turn_metrics(timeout: float | None = None) -> TurnRecord

• stream yields text as it is generated; closing the iterator early cancels generation.
• turn_metrics returns the record for the most recent turn, waiting for the background worker if needed.

from innards import Session

with Session("gpt2", max_new_tokens=16) as s:
    for piece in s.stream("Attention is"):
        print(piece, end="", flush=True)
    print()
    print(s.turn_metrics().timing)

AsyncSession

AsyncSession(model: str, backend: str = "hf", **session_kwargs) -> AsyncSession
async AsyncSession.stream(message: str) -> AsyncIterator[str]
await AsyncSession.turn_metrics(timeout: float | None = None) -> TurnRecord
await AsyncSession.chat(message: str) -> TurnRecord

Takes the same arguments as Session. The model loads and generates in worker threads, so the event loop is never blocked.

import asyncio
from innards import AsyncSession


async def main():
    async with AsyncSession("gpt2", max_new_tokens=16) as s:
        async for token in s.stream("Explain paged attention"):
            print(token, end="", flush=True)  # generation never waits on Innards
        turn = await s.turn_metrics()  # arrives from background workers
        print()
        print(turn.kv.measured_bytes)


asyncio.run(main())

compare

compare(models: Sequence[str], script: str | PathLike | Sequence[str], backend: str = "hf",
        **session_kwargs) -> CompareResult

• models: Hub repo ids or local paths, run one at a time.
• script: a list of user messages, one message, or a JSON file with a list of messages (strings or {"content": ...} objects, optionally under "messages").
• **session_kwargs: passed to every Session.

Returns a CompareResult: turns[model] is the list of records; rows(), table() and to_dict() align them by turn.

import json, pathlib, tempfile
from innards import compare

path = pathlib.Path(tempfile.mkdtemp()) / "script.json"
path.write_text(json.dumps({"messages": ["Hello", "What did I just say?"]}))
result = compare(["gpt2"], path, max_new_tokens=8)
print(result.rows()[0]["kv_measured_bytes"])

predict

predict(model_or_config: str | PathLike | dict | PretrainedConfig | ModelSpec, context_tokens: int,
        kv_dtype: str = "float16", *, memory_bytes: int | None = None, batch_size: int = 1,
        revision: str | None = None, token: str | None = None) -> KVPrediction

• model_or_config: Hub repo id (only config.json is downloaded), a config path or directory, a dict, a PretrainedConfig or a ModelSpec.
• context_tokens: tokens resident in the cache.
• kv_dtype: float32, float16, bfloat16 or float8 (aliases fp32, fp16, bf16, fp8).
• memory_bytes: optional KV budget; sets capacity_tokens.

Returns a KVPrediction with bytes_per_token, kv_bytes, recurrent_state_bytes, total_bytes, per_layer_bytes, by_layer_type, capacity_tokens, formula and notes. "Architectural" KV is what the attention design needs; an engine may hold more, and Innards reports the gap instead of hiding it. Sliding-window layers hold window - 1 tokens: the current token attends to itself plus window - 1 cached ones.

from innards.kv import predict

p = predict("deepseek-ai/DeepSeek-V2-Lite", context_tokens=32768, memory_bytes=8 * 2**30)
print(p.architecture, p.bytes_per_token, p.capacity_tokens)  # mla 31104 163840

A config dict works offline:

from innards.kv import predict

gemma_like = {
    "model_type": "gemma3_text",
    "num_hidden_layers": 26,
    "num_attention_heads": 4,
    "num_key_value_heads": 1,
    "head_dim": 256,
    "sliding_window": 512,
    "sliding_window_pattern": 6,
    "max_position_embeddings": 32768,
}
p = predict(gemma_like, context_tokens=8192, kv_dtype="bf16")
print(p.by_layer_type)  # {'global': 33554432, 'sliding': 11511808, 'recurrent': 0}

explain

explain(model_or_architecture: str | PathLike | dict | ModelSpec) -> LineageEntry

• model_or_architecture: a family name or alias ("mha", "mqa", "gqa", "mla", "sliding_window", "hybrid", "ssm", "fp8"), a Hub repo id, a config path or dict, or a ModelSpec.

Returns a LineageEntry with family, predecessor, change, problem_solved, trade_off, kv_formula and reference. For a model, detail describes that model's own numbers.

from innards.lineage import explain

entry = explain("gqa")
print(entry.predecessor, "->", entry.family)  # mha -> gqa
print(entry.problem_solved)

from innards.lineage import explain

print(explain("deepseek-ai/DeepSeek-V2-Lite"))  # MLA: what it changed from GQA and why

TurnRecord

TurnRecord.to_dict() -> dict
TurnRecord.to_json(**json_kwargs) -> str
TurnRecord.from_dict(data: dict) -> TurnRecord
TurnRecord.from_json(text: str) -> TurnRecord

One JSON record per message, schema v1.0: model, runtime, tokens, context, kv, memory, timing, output and observer. Missing measurements are null, never zero. Readers accept any 1.x record and reject other major versions.

from innards.schema import TurnRecord

record = TurnRecord.from_dict(
    {
        "schema_version": "1.0",
        "session_id": "s-1",
        "turn": 1,
        "timestamp": "2026-10-01T10:42:07Z",
        "model": {"id": "Qwen/Qwen3-0.6B", "architecture": "gqa"},
        "runtime": {"backend": "hf"},
        "kv": {"predicted_bytes": 378929152, "measured_bytes": 378929152, "measured_source": "hf_cache"},
    }
)
print(record.kv.measured_bytes, record.memory.peak_bytes)  # 378929152 None

CLI

innards chat --model gpt2 --backend hf                    # chat; per-turn metrics on stderr
innards chat --model Qwen/Qwen3-0.6B --no-think --json    # TurnRecord JSON after each turn
innards predict --model Qwen/Qwen3-0.6B --context 32768 --kv-dtype fp8
innards predict --model deepseek-ai/DeepSeek-V2-Lite --context 32768 --memory 16GiB --json
innards explain sliding_window

innards chat streams the reply to stdout and prints a three-line summary per turn to stderr: prompt, cached and generated tokens; KV measured vs predicted by layer type; TTFT, prefill, decode speed, peak memory and dropped events. Type /reset to start over, /exit to quit. Options: --system, --max-new-tokens, --device, --dtype, --no-think, --json, --no-metrics.

Supported

Architectures (KV calculator and HF measurements):

Legacy config keys (n_layer, n_head, n_embd, ChatGLM, Falcon, MPT) and nested text_config (multimodal checkpoints) are read too. KV dtypes: fp32, fp16, bf16, fp8.

Backends: hf, in-process Hugging Face Transformers 5.x on CPU, CUDA or Apple MPS. Remote engines (vLLM, Ollama, llama.cpp) and managed APIs use the same TurnRecord and plug in through innards.backends.register_backend. They are planned, not shipped yet.

Sinks: console (text or JSON lines). Any object with write(record) and close(), or a plain callable, also works as a sink.

Environments: anywhere Python runs: local terminal, Jupyter and VS Code, Docker, cloud VMs and containers (AWS, GCP, Azure), Kaggle and Colab.

Python: 3.10, 3.11, 3.12, 3.13 and 3.14. Python 3.15 is tested against its release candidates, with support from the final release (1 October 2026). The core install runs on 3.15 today. The hf extra needs PyTorch wheels for your Python version, and PyTorch has none for 3.15 yet.

Non-blocking guarantee

Innards never sits in the generation path:

• Hot path: the only Innards code inside the token loop takes a timestamp and offers a small event to a bounded queue. It makes no network calls, disk writes, tensor copies or device synchronization.
• Bounded queues that never block: when the per-token queue is full, the event is dropped and counted, never waited on. Every record carries observer.dropped_events, so gaps are visible. The two per-turn events (start and end, sent outside the token loop) are never dropped, so every message gets a record.
• Background workers: KV math, cache measurement (tensor shapes and dtypes only, numel × element_size), memory sampling and sink writes all run in background threads.
• Fail-open: if a collector or sink fails, the error is logged and counted, and generation continues.
• Same output: greedy output with Innards is identical to plain model.generate, which the test suite checks.

Overhead budget: throughput within 1% of running without Innards, and no added time to first token. scripts/overhead_bench.py measures it (same model and prompts, Innards on vs off, interleaved runs, medians).

License

Apache-2.0