Membrane

Treat the context window like RAM, not a scrap heap of strings.

Website · PyPI · Issues

Membrane is a prompt compiler for LLM applications. You give it your system rules, the memories your retriever found, the conversation so far and the current user turn. It returns a provider-ready message list that fits a hard token budget, puts the important content where models pay attention, keeps a byte-stable prefix so provider prompt caching actually hits, and gives you a map from every [Mem:N] citation back to your own record IDs. Like an operating system managing RAM, it has fixed capacity, explicit placement, eviction by priority, and caching. Prompts are compiled, never concatenated. The only core dependency is tiktoken. Membrane itself never calls an LLM or sends your prompts anywhere.

memories + rules + history + user turn
        │
        ▼
 Layout ──▶ Pre-flight ──▶ Firewall ──▶ Glaciator ──▶ Provider
 (U-shape)  (compress,     (hard token   (stable prefix  (messages,
            optional)      budget)       vs dynamic tail) cache hints)
        │
        ▼
 CompileResult: messages · ref_map · metadata (tokens, trimming, caching)

Results at a glance

Eleven stress scenarios (long chats, tool calls, memory floods, long pastes, tight budgets, paraphrased questions) sent to the Claude API on 2026-11-11, with the same token budget for every library. Accuracy means the reply contained the fact asked for. "Wrong" counts replies that stated a different value instead of saying it was unknown. Cost is input cost relative to LangChain = 100.

Read this with its limits: one seed per model, synthetic tasks, Claude models only, and the other libraries run with their documented defaults. The suite, the raw results and the scenarios where Membrane got worse or stayed behind are under Benchmarks.

Contents

• Results at a glance
• Why Membrane
• Install
• Quickstart
• Core concepts
• Providers and adapters
• Composing the stages yourself
• Async usage
• Configuration reference
• More tools
• Benchmarks
• Contributing
• License

Why Membrane

Most LLM apps join system instructions, retrieved memories, history and the user message into one big string. That breaks in four predictable ways:

Install

Requires Python 3.10+. The package is published as membrane-aol and imported as membrane.

pip install membrane-aol

Core features need only tiktoken. Optional features come as extras:

pip install "membrane-aol[all]"

pip install "membrane-aol[lod,redis]"

all does not include llm-anthropic, tokenizers, mcp or the framework adapters. Install those by name.

Two downloads happen on first use. tiktoken fetches its encoding files once; for offline machines, pre-seed TIKTOKEN_CACHE_DIR. With lod installed, fastembed downloads its embedding model the first time topic-shift detection or the coding/conversational deduplicator runs.

Quickstart

from membrane import CompileResult, ContextCompiler, MemoryItem

compiler = ContextCompiler(
    max_tokens=8192,
    model_id="openai/gpt-4o",
    core_content="You are a helpful assistant.",
)

memories = [
    MemoryItem(id="1", content="User prefers concise answers.", memory_type="semantic", priority=2),
    MemoryItem(id="2", content="Also discussed asyncio yesterday.", memory_type="semantic", priority=2),
]

result: CompileResult = compiler.compile(
    memories=memories,
    user_message="Show a producer-consumer example.",
)

print(result.messages)                   # send these to your provider
print(result.ref_map[1])                 # "1": [Mem:1] in the prompt is your record "1"
print(result.metadata["token_count"])    # exact size of the compiled prompt

result.messages is an OpenAI-style list. The system message holds your rules. The final user message holds the kept memories followed by the current turn:

[system] You are a helpful assistant.

         If a detail is missing from the provided context, do not hallucinate. Output exactly:
         <SYSTEM_INTERRUPT_FETCH[Your precise query for the missing detail]> ...
[user]   [Mem:1] [fact] kind=note | text=User prefers concise answers. | conf=1.00
         [Mem:2] [fact] kind=note | text=Also discussed asyncio yesterday. | conf=1.00

         --- Current user message ---
         Show a producer-consumer example.

The second paragraph of the system message is the interrupt protocol instruction. Pass include_interrupt_instruction=False to leave it out.

Pass the conversation so far as conversation_messages. It is sent as real messages between the system message and the final user message, with tool calls and content-part lists kept as given:

from membrane import ContextCompiler, MemoryItem

compiler = ContextCompiler(max_tokens=8192, model_id="openai/gpt-4o",
                           core_content="You are a helpful assistant.")
history = [
    {"role": "user", "content": "What's a good queue for asyncio?"},
    {"role": "assistant", "content": "asyncio.Queue is the standard choice."},
]
result = compiler.compile(
    memories=[MemoryItem(id="pref-42", content="User prefers concise answers.")],
    conversation_messages=history,
    user_message="Show me a producer-consumer example.",
)
assert [m["role"] for m in result.messages] == ["system", "user", "assistant", "user"]

Core concepts

The U-shaped layout

Models attend most strongly to the start and end of their context (Liu et al., 2023). ContextLayoutEngine builds every prompt in three regions:

With the default memory_placement="end", history goes between the start and the middle, so the memories sit right next to the question. See docs/u-shaped-layout.md.

Token budgets and priority tiers

max_tokens is a hard budget for the compiled prompt. When model_id is in the capability map, the budget is also capped at that model's context window. Unknown models never shrink your budget.

Every memory carries a priority tier. When the prompt is over budget, the firewall drops memories from the lowest-priority tier first. The start and end regions are always kept.

With end placement, nothing is dropped while something else still has its full text (thin_before_drop). Every memory has a keyframe: its densest sentence under keyframe_tokens (40), where dense means its words are rare across the sentences of the memories on offer and of the conversation. Under pressure the keyframes of all memories are placed first, then memories get their full text back in priority order while room lasts. A thinned memory keeps its [Mem:N] ref and ends in …, so the model can still cite it or ask for the full text. metadata["segments_thinned"] counts them. A memory with no sentence under the cap is kept whole or dropped, as before. A keyframe is one sentence: a fact spread over two sentences of a memory is only half there once that memory is thinned.

Kept memories are emitted in tier order (procedural, then semantic, then episodic). [Mem:N] numbers follow your input order, so a citation always points at the same input record. Each memory's text is shortened by default: 200 characters for facts, 280 for episodes and 500 for procedures. Set MEMBRANE_MEMORY_TEXT_MAX_CHARS to change this (0 disables it). metadata["memories_truncated"] reports how many were shortened.

from membrane import ContextCompiler, MemoryItem

compiler = ContextCompiler(max_tokens=200, core_content="You are a support agent.")
memories = [
    MemoryItem(id="howto", content="Escalate refunds over $500 to a human.", memory_type="procedural", priority=1),
    MemoryItem(id="fact", content="Customer is on the Pro plan.", memory_type="semantic", priority=2),
    *[
        MemoryItem(id=f"ep{i}", content=f"Earlier ticket #{i} about a billing question, resolved by email.",
                   memory_type="episodic", priority=3)
        for i in range(10)
    ],
]
result = compiler.compile(memories=memories, user_message="Can I get a refund?")

md = result.metadata
print(md["token_count"], "<=", md["tokens_budget"])            # always within budget
print(md["segments_kept"], "of", md["segments_offered"], "memories kept")
print(md["tokens_trimmed"], "tokens trimmed")
assert "howto" in result.ref_map.values()                      # procedural kept
assert "fact" in result.ref_map.values()                       # semantic kept

Token counts use tiktoken's cl100k_base encoding by default, for every model. That is exact for GPT-4 and GPT-3.5 but other tokenizers count the same text differently (Claude counts more). So the budget is divided by a per-model token_scale. It starts from a seed in the capability map (1.0 for OpenAI, 1.11 for Claude Opus 4.5 and 4.6, Sonnet 4.6 and Haiku 4.5, 1.51 for Claude Opus 4.7 and every later Claude model) and you correct it with what the provider actually billed:

from membrane import ContextCompiler, MemoryItem

compiler = ContextCompiler(max_tokens=8000, model_id="anthropic/claude-opus-5-5")
result = compiler.compile(memories=[MemoryItem(id="1", content="User is in Lagos.")],
                          user_message="What time zone am I in?")
# ... send the request, then report the prompt tokens the provider counted:
# usage = response.usage
# compiler.observe(result, usage.input_tokens + usage.cache_read_input_tokens
#                          + usage.cache_creation_input_tokens)
print(result.metadata["token_scale"], result.metadata["tokens_budget_counted"])  # 1.51 5298

observe smooths the ratio per model, adds a margin from its spread, and later compiles use the result. The scale in use moves in steps of 0.05, rises at once and falls only when it is 0.1 too high, because every change moves the history cut and breaks the cache once. It lives on the compiler object: nothing is downloaded, stored or sent. Pass token_scale=1.0 to turn it off. The Claude seeds were measured on 2026-11-11 with Anthropic's token-count endpoint (benchmarks/measure_token_scale.py) and are the ratio for English prose. Code counts higher (1.65 on Opus 4.7 and later, 1.27 on the older tokenizer) and digit-heavy text lower (1.11 and 1.05), so a first request that is mostly code can run about 9% over the budget until observe has seen it. Models other than Claude and OpenAI start at 1.0, unmeasured. observe is what makes the budget right.

To count with a model's own tokenizer instead, pass a TokenizerRegistry with patterns for your model ids. TokenizerRegistry.get() returns a shared registry preloaded with patterns for bare ids (gpt-4o* → o200k_base, claude* → cl100k_base). TokenizerRegistry() starts empty:

from membrane import ContextCompiler, MemoryItem
from membrane.tokenizers import TiktokenAdapter, TokenizerRegistry

registry = TokenizerRegistry()
registry.register("openai/gpt-4o*", TiktokenAdapter("o200k_base"))

compiler = ContextCompiler(max_tokens=8192, model_id="openai/gpt-4o")
result = compiler.compile(
    memories=[MemoryItem(id="1", content="User prefers concise answers.")],
    user_message="Hi",
    tokenizer_registry=registry,
)

Patterns are matched against the full model_id with fnmatch, so the preloaded gpt-4o* does not match openai/gpt-4o. Register a pattern for the ids you actually use. When no pattern matches, TokenizerRegistry.get() falls back to a character-based estimate, and a fresh TokenizerRegistry() falls back to cl100k_base. SentencePieceAdapter and HuggingFaceAdapter need the tokenizers extra.

The pipeline

Every compile runs the same deterministic stages:

1. Layout. ContextLayoutEngine serialises memories into [Mem:N] lines and builds the core, middle and end regions.
2. Pre-flight. If the middle exceeds MEMORY_PREFLIGHT_TOKEN_THRESHOLD (8,000 tokens by default) and you passed a compressor (compress_sync, or compress_async / compress_fn on the async paths), it is summarised to about MEMORY_PREFLIGHT_TARGET_TOKENS (500). Without a compressor the stage is skipped and the firewall trims as usual.
3. Firewall. PreFlightContextFirewall enforces the hard budget by trimming the middle by tier, then prunes ref_map to the memories that survived.
4. Glaciator. The output is split into a byte-stable prefix (system rules, then history) and a dynamic tail (memories, current turn, directives). Prompt caching reuses the prefix.
5. Provider. Messages are formatted for the target model, with caching metadata when the model supports prompt caching.

Optional modules (dedup, freshness decay, classifier, policies, reranker, profiler) plug in through a MembraneProfile. The built-in presets are minimal (the default), coding and conversational:

from membrane import ContextCompiler, MemoryItem
from membrane.profile import MembraneProfile

compiler = ContextCompiler(max_tokens=8192, model_id="openai/gpt-4o")
result = compiler.compile(
    memories=[MemoryItem(id="1", content="Use pytest for tests.", memory_type="procedural", priority=1)],
    user_message="Write a test for parse_date().",
    profile=MembraneProfile.coding(),
)

Glaciation and prompt caching

Providers cache the longest prefix a request shares with an earlier one. If memories sit before the conversation history, they change every turn and the cache never gets past them. Membrane "glaciates" the prompt: the frozen part comes first and the moving part last.

With the default memory_placement="end", the compiler emits:

1. a system message with only the core rules,
2. conversation_messages exactly as given,
3. one user message with the kept memories, the current turn and any dynamic directives.

Items 1 and 2 are byte-identical from turn to turn, so the provider serves most of each request from cache. History may hold history_share (default 0.75) of the budget left after the core. Past that, the oldest turns are dropped in chunks of half that share (metadata["history_messages_dropped"]). The cut depends only on the budget and the core, never on this turn's memories or question, so it stays put as the chat grows and the cache breaks only when another chunk goes.

Dropped turns are not lost. Each dropped message leaves one line in an index placed in the system message right after the core (eviction_index): its role and its keyframe (see Token budgets), scored over the conversation up to the cut that dropped it. A line never changes after it is written, so the index only changes when the cut moves, and at that moment the cache after the core is rewritten anyway: the index adds no cache breaks. It may hold up to index_share (one eighth) of the budget the core leaves. That room is not taken from history: it comes from what history, the question and one keyframe per memory on offer leave, and never less than half the share. The size is decided when a cut is first seen and kept until the next cut, so the block stays byte-stable in between; a new compiler object can size it differently, which costs one cache miss. Over its room, the oldest lines go, again only at a cut. Under memory pressure the index shrinks, and with a very small budget that loses old history (see Benchmarks). Tool results get a line, tool-call stubs do not, and a long message keeps one keyframe per 400 tokens of its length.

Passages come back in full where they match (recall_evicted_history): dropped messages are split into passages of about 60 words, tool results included, and ranked by the rare words they share with the current user message, then by recency. The best passages, up to one chunk, return in the dynamic tail and stay out of ref_map. Every memory is offered its place before any passage is, so passages cost the memories at most the two header lines that introduce them (which can be one memory line under a very tight budget). A passage that does not fit whole is cut to the sentence that matches the question.

When memories do not all fit, the ones that survive are chosen by reciprocal rank fusion of the order you passed (your retriever's rank) and the same lexical relevance (rank_fusion). The survivors keep your order, so nothing changes when everything fits. Both are word matching, not semantic search: they help when the question names what it asks for. To rank by meaning as well, pass an embedding function (see Ranking by meaning). Pass trim_history=False to manage history yourself. An overrun then sets metadata["budget_exceeded"] = True instead of dropping turns.

memory_placement="system" puts core, memories, the last turn and the user message into a single system message. It sends fewer tokens because it includes only the last turn of history, but the cache can never get past the core. Use it for local models without caching, and for one-off calls.

To split text you assemble yourself, use the standalone glaciate() helper. See Composing the stages yourself and docs/prompt-caching.md.

Ref-map citations

Each memory gets a short local index in the prompt, such as [Mem:3]. result.ref_map maps each index that survived trimming back to your record ID, and result.ref_map_type gives its kind. Ask the model to cite [Mem:N] and you can resolve every citation to a real record:

import re

from membrane import ContextCompiler, MemoryItem

compiler = ContextCompiler(max_tokens=4096)
result = compiler.compile(
    memories=[
        MemoryItem(id="doc-381", content="The refund window is 30 days."),
        MemoryItem(id="doc-112", content="Refunds go back to the original card."),
    ],
    user_message="How long do I have to request a refund?",
)

answer = "You have 30 days [Mem:1], refunded to your original card [Mem:2]."  # model output
cited = [result.ref_map[int(n)] for n in re.findall(r"\[Mem:(\d+)\]", answer)]
assert cited == ["doc-381", "doc-112"]
assert result.ref_map_type[1] == "semantic"

Trimmed memories are removed from the map, so a citation that doesn't resolve shows the model cited something it was never shown. Across turns, you can persist the map in Redis with set_ref_map_for_conversation and get_ref_map_for_conversation (requires the redis extra), or pass persist_ref_map_to_redis=True to compile_prompt_async.

Providers and adapters

Capability map

Membrane ships a capability map (llm_capabilities.json) that sets the context cap, prompt-caching support, cache prices and token scale for each model:

Lookups ignore case, a missing provider prefix and date suffixes, so gpt-4o and openai/gpt-4o-2024-08-06 both resolve to openai/gpt-4o. Unknown models get no caching and no context cap, and log a warning once. To use your own map, point LLM_CAPABILITY_MAP_PATH at a JSON file with the same capabilities and context_caps keys; it replaces the bundled map.

Two more sections of the map drive the layout. cache_pricing gives, per model, the cache write and read prices relative to plain input, the shortest prefix the provider caches, and whether caching is automatic (OpenAI) or marked (Anthropic). token_scale is the seed for calibrated budgets. Both are optional in your own map and both are lists of prices and ratios that providers change, so check them against your provider's current pricing page. They were last checked against the OpenAI and Anthropic pricing pages on 2026-11-11; models retired by then (gpt-4-turbo, claude-3-5-sonnet, claude-3-7-sonnet, claude-3-opus) were removed from the map and are now treated as unknown models. The map does not yet list OpenAI models newer than gpt-5.1; add them through your own map.

from membrane import get_cache_pricing, get_token_scale

print(get_cache_pricing("claude-opus-5-5"))  # mode='marked', write=1.25, read=0.05, min_tokens=512
print(get_token_scale("gpt-4o"))             # 1.0

from membrane import get_context_cap_for_model, has_prompt_caching

assert has_prompt_caching("gpt-4o")
assert get_context_cap_for_model("anthropic/claude-sonnet-4.5") == 200_000

OpenAI and OpenAI-compatible APIs

result.messages already has the Chat Completions shape:

from membrane import ContextCompiler, MemoryItem

compiler = ContextCompiler(max_tokens=8192, model_id="openai/gpt-4o-mini",
                           core_content="You are a helpful assistant.")
result = compiler.compile(memories=[MemoryItem(id="1", content="User is in Lagos.")],
                          user_message="What time zone am I in?")

request = {"model": "gpt-4o-mini", "messages": result.messages}
# from openai import OpenAI
# OpenAI().chat.completions.create(**request)

OpenAI caches long shared prefixes automatically, so end placement is all you need.

Anthropic

The Anthropic Messages API takes the system prompt as a separate system parameter and only caches what the request marks with cache_control. anthropic_request builds both from a compile result:

from membrane import ContextCompiler, MemoryItem, anthropic_request

compiler = ContextCompiler(max_tokens=8192, model_id="anthropic/claude-opus-5-5",
                           core_content="You are a helpful assistant.")
result = compiler.compile(memories=[MemoryItem(id="1", content="User is in Lagos.")],
                          user_message="What time zone am I in?")

request = {
    "model": "claude-opus-5-5",  # the API's model name, not Membrane's model_id
    "max_tokens": 1024,
    **anthropic_request(result),  # "system" blocks and "messages", cache_control placed
}
# import anthropic
# response = anthropic.Anthropic().messages.create(**request)
# usage = response.usage
# compiler.observe(result, usage.input_tokens + usage.cache_read_input_tokens
#                          + usage.cache_creation_input_tokens)

The markers go where metadata["cache_breakpoints"] says a cached prefix ends: after the core system prompt, and after the conversation history. Tool calls and tool results in the history are converted to tool_use and tool_result blocks. Nothing is marked below the model's minimum cacheable size (512 to 4,096 tokens depending on the model).

The history is always marked. An earlier version held the marker back when a cut was expected before the history could be read again; in a simulation of the 3,000-token scenario that saved about 0.1% of input cost, so the rule was removed. Pass ttl="1h" for the one-hour cache. anthropic_request needs end placement (the default); anthropic_system_blocks(prefix, tail) is still there for text you split yourself.

Local models

Local models in the map (Ollama-style ids such as llama3.1:8b) report no caching. Consider memory_placement="system" for them: it sends fewer tokens, and there is no cache to protect.

Framework adapters

Use Membrane instead of a framework's own conversation memory, not alongside it. With LangChain, disable_langchain_memory(chain) removes the chain's memory, and detect_memory_collision(chain, membrane_active=True) raises MemoryCollisionError if one is still set. See examples/drop3_mcp_and_adapters.md.

Memory types

compile() accepts MemoryItem rows or any object with an id and a memory_kind property (the KernelMemoryRecord protocol). Typed records are also available: SemanticMemory, EpisodicMemory, ProceduralMemory and KernelMemoryObject, with a JSON schema from load_kernel_memory_json_schema().

Composing the stages yourself

Every stage is a public building block. Use them separately when you already have your own prompt assembly:

from membrane import (
    TIER_EPISODIC,
    TIER_PROCEDURAL,
    TIER_SEMANTIC,
    ContextLayoutEngine,
    PreFlightContextFirewall,
    SemanticMemory,
    anthropic_system_blocks,
    build_system_messages,
    count_tokens,
    glaciate,
)

# 1. Layout: serialise (memory, tier) pairs into [Mem:N] segments.
layout = ContextLayoutEngine(core_content="You are a precise research assistant.")
retrieved = [
    (SemanticMemory(id="s1", subject="Membrane", predicate="license", object="Apache-2.0"), TIER_SEMANTIC),
]
middle, ref_map, ref_map_type = layout.build_middle(retrieved)
middle.append(("[Mem:2] Earlier session: user compared prompt compilers.", TIER_EPISODIC))
middle.append(("[Mem:3] Always cite sources as [Mem:N].", TIER_PROCEDURAL))
ref_map.update({2: "e7", 3: "p1"})

core = layout.build_core()
end = layout.build_end(last_turn_content=None, user_message="What license does Membrane use?")

# 2. Firewall: enforce the budget; trims the middle by tier and prunes the ref map.
firewall = PreFlightContextFirewall(max_tokens=4096)
text, kept_refs, total, stats = firewall.run(
    core_text=core, end_text=end, middle_segments=middle, ref_map=ref_map
)
assert total == count_tokens(text) <= 4096
print(stats["segments_kept"], "of", stats["segments_offered"], "segments kept")

# 3. Glaciator: split at a line boundary into a cacheable prefix and a dynamic tail.
g = glaciate(text, max_cacheable_tokens=2000)

# 4. Provider formatting.
openai_messages, cache_hint = build_system_messages("openai/gpt-4o", g.cacheable_prefix, g.dynamic_tail)
anthropic_system = anthropic_system_blocks(g.cacheable_prefix, g.dynamic_tail)

glaciate() never cuts mid-line. Its use_caching flag is True once the prefix reaches 256 tokens. Check the provider's own minimum cacheable length too.

Async usage

ContextCompiler.compile_async takes the same arguments as compile, plus an async pre-flight compressor and optional Redis caching of compressed output:

import asyncio

from membrane import ContextCompiler, MemoryItem


async def compress(raw_middle: str, target_tokens: int, user_message: str) -> str:
    # Call your LLM here. Runs only when the middle exceeds MEMORY_PREFLIGHT_TOKEN_THRESHOLD.
    return raw_middle[: target_tokens * 4]


async def main() -> None:
    compiler = ContextCompiler(max_tokens=8192, model_id="openai/gpt-4o")
    result = await compiler.compile_async(
        memories=[MemoryItem(id="1", content="User prefers concise answers.")],
        user_message="Summarise our plan.",
        compress_async=compress,
    )
    print(result.metadata["token_count"])


asyncio.run(main())

compile_prompt_async (alias compile_prompt_payload) is a higher-level entry point for chat backends. It takes pre-retrieved (memory, tier) pairs and the raw message list, detects topic shifts (with the lod extra) and can persist the ref map to Redis:

import asyncio

from membrane import TIER_EPISODIC, TIER_SEMANTIC, MemoryItem, compile_prompt_async


async def main() -> None:
    result = await compile_prompt_async(
        user_msg="Can you remind me what we decided?",
        messages_list=[
            {"role": "user", "content": "Let's pick a database."},
            {"role": "assistant", "content": "Postgres fits your workload."},
        ],
        user_id="user-1",
        conversation_id="conv-9",
        max_tokens=8192,
        model_id="openai/gpt-4o",
        core_content="You are a helpful assistant.",
        retrieved_objects=[
            (MemoryItem(id="d1", content="Team chose Postgres over MongoDB."), TIER_SEMANTIC),
            (MemoryItem(id="e1", content="Discussed sharding last week.", memory_type="episodic"), TIER_EPISODIC),
        ],
    )
    print(result.ref_map)


asyncio.run(main())

To write the compressor against a class instead of a bare function, implement the PreFlightAnalyst protocol (compress_retrieved_memory(...)), or wrap an async function with as_preflight_analyst. Membrane does not ship provider SDK clients. The llm-openai and llm-anthropic extras only install the SDKs for your own compressor.

Configuration reference

ContextCompiler

compiler.observe(result, actual_prompt_tokens) reports what the provider billed for a compiled prompt and returns the scale now in use. compile_anonymous(...) takes the same history options when called with memory_placement="end".

compile() / compile_async() arguments

CompileResult

Environment variables

More tools

Cost estimates

estimate() runs the full compile and adds an input-side quote. Membrane ships no price list, so you pass your provider's rate:

from membrane import ContextCompiler, MemoryItem

compiler = ContextCompiler(max_tokens=8192, model_id="openai/gpt-4o-mini")
quote = compiler.estimate(
    memories=[MemoryItem(id="1", content="User prefers concise answers.")],
    user_message="Hello",
    price_per_1k_input_tokens=0.00015,
)
print(quote.total_tokens, quote.estimated_cost_usd, quote.trimming_strategies)
messages = quote.compile_result.messages   # no need to compile twice

Interrupt protocol

When the context is missing a detail, the default system instruction tells the model to emit a fetch tag instead of guessing. Your host detects it, retrieves, and continues:

from membrane import contains_interrupt, parse_interrupt, strip_all_interrupts

text = "Let me check. <SYSTEM_INTERRUPT_FETCH[user's timezone preference]> One moment."
if contains_interrupt(text):
    before, query, after = parse_interrupt(text)
    assert query == "user's timezone preference"
clean = strip_all_interrupts(text)

See docs/interrupt-protocol.md for streaming integration.

Ranking by meaning (experimental)

Off by default, and in our own measurements it did not improve results (details below). It is here for you to test on your data, not as a recommended setting.

By default Membrane decides what gets its full text by the words an item shares with the question. Pass embedder= and it also uses similarity of meaning, for two choices only: which memories keep their full text under pressure, and which passages of dropped history come back. Keyframes, the index of dropped history and the history cut do not use it, so the cached prefix is the same with or without it.

from membrane import ContextCompiler
from membrane.lod import fastembed_embedder  # pip install "membrane-aol[lod]"

compiler = ContextCompiler(max_tokens=8000, model_id="openai/gpt-4o",
                           embedder=fastembed_embedder())

embedder is any function from a list of texts to one vector each, so you can pass your own model or an API client instead. fastembed_embedder() runs BAAI/bge-small-en-v1.5 locally on the CPU; fastembed downloads the model (about 130 MB) the first time. Membrane itself still makes no network calls and has no new dependency. Each text is embedded once per compiler and the question once per turn; if the function raises, that compile falls back to word matching. metadata["semantic_ranking"] is set when it was used. The similarity is fused with the lexical order by reciprocal rank fusion (and with your retriever's order for memories), so it is one vote, not a replacement.

Measured on the offline hard suite (eleven scenarios, 5 seeds, BAAI/bge-small-en-v1.5; run it with --strategies membrane,membrane-embed), it did not help. Ten scenarios are identical with and without it, in both modes. One got worse: history recall in the reworded spread-fact scenario fell from 93.8% to 91.2% (two more of 80 questions missed); the likely cause, not verified, is a passage that resembles the question displacing one that carried part of the fact. Cost did not move. Each turn took roughly 40 to 220 ms longer on a laptop CPU (the model, not Membrane). The suite is a weak test of it: memory recall is already 100% there, and the history it still misses is lost when a fact is spread over sentences, not when the wrong passage is ranked first. It is likelier to pay off where questions and stored text use different words and many memories compete, which this suite's paraphrase scenarios only partly model. Treat it as untested on real conversations and measure before turning it on.

Separately from embedder: with the lod extra installed, every compile that has history also embeds the last user messages for topic-shift detection (below), which cost 100 to 200 ms per compile on the same (busy) machine whether or not you pass embedder.

Topic-shift detection (LOD)

With the lod extra, compute_attention_shift compares the current message with the centroid of recent user messages, using local fastembed embeddings:

from membrane import compute_attention_shift

shifted = compute_attention_shift(
    ["What's the best Python testing framework?", "How does pytest discover tests?"],
    "How do I deploy to Kubernetes?",
)

It returns False when numpy or fastembed is missing. To use your own embedder, call compute_attention_shift_with_embeddings.

CLI diagnostics

The membrane command runs the compiler locally and calls no LLM:

membrane inspect --user-message "Explain the retry policy." --memory "policy|2|Retries use exponential backoff with jitter." --max-tokens 2048

membrane quote --user-message "Summarize the release plan." --memory "release|2|Version 0.2 ships after the integration tests pass." --price-per-1k-input-tokens 0.00015

membrane profile explain coding

--memory takes id|tier|content and can be repeated. Add --json for machine-readable output.

Benchmarks

Prompt-cache savings (benchmarks/cache_savings.py): a 20-turn chat on openai/gpt-4o with a 2,000-token system prompt and 5 new memories retrieved every turn. The simulated provider caches the longest prefix shared with the previous request (minimum 1,024 tokens).

With the same full history, end placement cuts input cost by 23–60%, because memories no longer break the cache every turn. Anthropic bills cache reads at about 0.1× and OpenAI at about 0.5×. "system" is cheaper only because it sends just the last turn of history. The simulation is deterministic; your savings depend on prompt sizes and pricing.

python -m benchmarks.cache_savings

Peer comparison (benchmarks/peer_comparison.py): a 24-turn chat under a hard budget, 5 seeds. Each turn asks for one fact that sits either in a retrieved memory or in a user message 10 turns back. Tokens are counted with OpenAI chat framing. Recall means the answer was in the prompt, an upper bound on accuracy. The --live mode measures real accuracy and cached_tokens against OpenAI but has not been run yet. priompt reimplements Priompt's priority-cutoff search rather than importing the package.

Recall costs tokens: with recall_evicted_history=False the 6,000-token run caches 69% and costs 66 / 44. It now finds every history fact too, through the index of dropped history (before the index it found none); this benchmark states each fact in one sentence, the best case for the index. With a 12,000-token budget Membrane caches 81% and costs 68 / 35. Under memory pressure (4,000 tokens, 40 memories) it costs 87 / 73 and keeps the needed memory every time, against 95% for naive.

python -m benchmarks.peer_comparison [--budget N] [--memories N] [--live]

Hard suite (benchmarks/hard_suite.py): eleven stress scenarios, 5 seeds each, against LangChain trim_messages, LlamaIndex ChatMemoryBuffer and Memory, Priompt and a naive loop. Offline; recall means the answer was in the prompt, an upper bound on accuracy. Every strategy gets the same budget in tiktoken tokens (Membrane runs with token_scale=1.0). Mean over all scenarios (cost relative to naive = 100):

The Priompt row runs the real package with one element per conversation turn and memories ranked above history; its recall equals the Python reimplementation's in all eleven scenarios. Ranking the last three turns above memories instead changes its recall by less than a point.

Per scenario, Membrane defaults against the best of the others. "Before" is the same suite on the previous state of the code (commit cb43a15: thin-before-drop, the first eviction index, incremental counting), so the two columns show what the later changes did (index room, neighbour sentences in keyframes, questions ranked behind statements, tool calls named in the index):

With recall_evicted_history=False (the cheap mode: the index alone carries old history), history recall before and after: 80-turn chat 92% to 95%, tool scenario 30% to 97%, memory flood 83% to 100%, two-fact 87% to 99%, 3,000-token budget 86% to 89%, paraphrase 48% to 96%, two-sentence fact 49% to 72%, spread fact 45% to 70%, spread and reworded 50% to 72%; pastes and the 24,000-token budget stay at 100%. Mean recall 91% to 96%.

Where Membrane got worse or is behind:

• With a 3,000-token budget, history recall fell from 100% to 97%. The index now shrinks when memories need the room; without shrinking it stays at 100% but 13 of 9,890 offered memories lose their line.
• The paste scenario costs more: 91 / 80 against 86 / 75, because more history is kept word for word. The 80-turn chat costs one point more at 0.5× cache reads (76 against 75).
• The cheap mode costs more in five scenarios (for example 80-turn chat 55 / 29 against 51 / 28, tool scenario 63 / 41 against 59 / 40), 69 / 47 on average against 69 / 50.
• Median compile time rose by about 0.5 ms in most scenarios (80-turn chat 0.9 to 1.9 ms, memory flood 2.6 to 3.3 ms), from keyframes that keep neighbour sentences. Pastes fell from 3.3 to 0.8 ms.
• A fact spread over several sentences is still the weak spot: 94% to 96% with recall on, 70% to 72% in the cheap mode. A keyframe keeps a neighbour sentence only when that sentence opens with a pronoun or is very short; the two-sentence scenario's second sentence opens with "The", so the rule does not fire there.
• The rule that ranks questions behind statements is flattered by this suite: every user message in it opens with a question and then states a fact. Expect less on real conversations. It accounts for most of the paraphrase gain (72% to 96%).
• LangChain, Priompt and the naive loop keep 100% of two-fact answers and of old history in the tool scenario; Membrane now matches them there, at lower cost, but does not beat them.

Membrane picks dropped history and surviving memories partly by the words they share with the question; most scenarios name what they ask for, which helps that. The paraphrase and reworded scenarios remove the help. The benchmark text is synthetic and templated; nothing in the keyframe scoring was fitted to it.

The peers put retrieved memories in the system message, as their RAG chat patterns do. Moving them next to the question by hand gives the same caching in any of them. Median compile time is 0.5 to 3.6 ms per turn at these sizes (1.9 ms on the 80-turn chat); a turn that moves the history cut takes 2 to 9 ms. LangChain and the Priompt reimplementation take 0.1 to 0.2 ms with token counts supplied by the harness, real Priompt 7 to 16 ms including the round trip to Node, LlamaIndex 2 to 80 ms.

uv run --extra langchain --extra llamaindex python -m benchmarks.hard_suite

The real Priompt rows need node and one npm install in benchmarks/priompt_node; without them the suite skips those rows.

Live run (benchmarks/hard_suite_live.py): the suite's requests sent to the Claude API. Accuracy means the reply contained the fact. "Wrong" counts replies that stated a different code instead of saying it was unknown. Cost is uncached input

• 1.25 × cache writes + 0.1 × cache reads, relative to LangChain = 100 (Claude Opus 5.5 bills reads at 0.05×, which widens the gaps).

Latest run, 2026-11-11, one seed (seed 1) on each model, all eleven scenarios, 430 requests per strategy, on the code at commit 61b8df6. claude-opus-5-5 ran four strategies; claude-haiku-5-5 also ran real Priompt. Mean over scenarios:

Before and after on claude-opus-5-5, same seed, the nine scenarios both runs have. "Before" is the one-seed run on cb43a15 described below. Accuracy, then cost against LangChain = 100:

The two new scenarios (no "before"): Membrane 98% in both at cost 64, cheap mode 92% at 46 and 32, LangChain 90%, LlamaIndex Memory 78%.

What got worse or stayed behind in the live runs:

• Cheap mode, 3,000-token budget: 97% to 93% on Opus (history 86% to 71%), and 93% on Haiku.
• Cheap mode costs more than before in four scenarios on Opus: 80-turn chat 21 to 27, pastes 38 to 47, paraphrase 60 to 65, two-sentence fact 29 to 42. Defaults cost more in two: memory flood 68 to 69, 24,000-token budget 23 to 26.
• On Haiku 5.5 the defaults were below the cheap mode in the tool scenario (tool facts 92% against 100%) and below LangChain and Priompt on two-fact answers (96% against 100%), and gave 7 confident wrong answers, against 2 on Opus.
• Facts spread over sentences: 94% of those history questions on Opus, 88% on Haiku.
• The paraphrase jump (80% to 100%) rests on 10 history questions in one seed.

This is one seed per model: 30 to 80 questions per scenario, so a difference of one or two questions (3 to 7 points in the smaller scenarios) is within what the same strategy shows between runs. LangChain's requests did not change between the two Opus runs, yet its accuracy moved from 85% to 88% in the tool scenario and from 97% to 100% on two-fact answers. Three seeds were planned; one was run to keep the spend under the $35 the run was allowed (about $29 for Opus and $1.30 for Haiku at list prices).

The run before that: one seed, nine scenarios, 350 requests per strategy, claude-opus-5-5 on 2026-11-10, on the code at commit cb43a15 (before the index-room, neighbour-sentence, question and tool-call changes above). Mean over scenarios:

In that run Membrane had the best or equal accuracy in every scenario and cost less in eight of nine; under a 3,000-token budget it cost 102. With recall_evicted_history=False it was below LangChain in two-fact questions (90% against 97%) and the two-sentence scenario (82% against 90%). One seed means 30 to 80 questions per scenario, so single-scenario differences of a few points are noise.

Things to know when reading the live numbers. The paraphrase scenario's memory questions are left out of accuracy, because they name projects by an alias no model can resolve. Assistant turns are scripted, so every strategy sees the same history. An earlier run on the release before these mechanisms (eight scenarios, one seed: Membrane 92% at cost 69) is kept in benchmarks/results/ with the other raw results.

uv run --extra langchain --extra llamaindex --extra llm-anthropic python -m benchmarks.hard_suite_live

The run needs Claude API credentials and costs API credits: about $26 per seed at list prices for the default four strategies on Claude Opus 5.5 over eleven scenarios, about $10 more with priompt-node.

Also in benchmarks/: token_savings_bench.py (savings from compression and trimming) and trimming_bench.py (trimming keeps high-priority content and the U-shaped layout).

Contributing

Bug reports, fixes, docs and new adapters are welcome. For anything bigger than a small fix, open an issue first. Every pull request needs the Contributor License Agreement box ticked. See CONTRIBUTING.md for details. Report security issues privately as described in SECURITY.md.

git clone https://github.com/adeyomilawal/membrane.git

cd membrane && uv sync --locked --extra dev

Run the same checks as CI:

uv lock --check && uv run ruff check src tests && uv run pytest && uv build

Without uv, run pip install -e ".[dev]" and then pytest. Tests that need optional stacks (MCP, LangChain and so on) are skipped when those packages aren't installed.

Further reading: Architecture · U-shaped layout · Prompt caching · Interrupt protocol · Changelog

License

Membrane is licensed under the Apache License, Version 2.0. You can use, modify and distribute it in commercial and non-commercial products. See LICENSE and NOTICE.