scene-memory

A scene layer for conversational agent memory: turns of conversation become a graph of versioned slots that knows which version of each fact still holds. Ask "where do I live?" and it answers with the current value; ask "where did I live before?" and it answers from the superseded history — nothing is deleted, facts are invalidated, not forgotten.

Two things live in this repository:

• The library (scene_memory/) — pure-stdlib core, no pip install, no network. Extraction and reading use a local LLM through Ollama.
• The benchmark system (results/extractor_experiment/) — the research harness that routes each question by form to the right memory and reader, and holds the headline result below.

Headline results

LongMemEval-S, official judging protocol: 477/500 = 95.4% — above Mastra (gpt-5-mini) on all six indicators.

Every number comes from a single reproducible pass over the 500 questions (sonda_e2e_canonica.py), verified against a canonical state file whose guard refuses to write if anything drifts. Every mechanism behind it was integrated with a pre-registered prediction committed before measuring, an explicit falsification bar, and ten falsified arms published with the same prominence as the wins. The full audit trail is in results/extractor_experiment/RELATORIO_ARQUITETURAS.md (Portuguese).

And the library result that motivates the scene itself: reading the compressed scene (~550 words) lets a small local model answer better than the same model reading the raw sessions (~9,000 words) — +0.102 accuracy at 16× less input — matching a much stronger reader on raw text. The scene removes exactly the noise that confuses readers: duplicates, stale values, scattered updates.

Honest framing: these are benchmark results with registered caveats, not a product promise. Router triggers and absence gates are regexes calibrated on LongMemEval's English corpus; see Limits.

Quickstart

Core is pure stdlib. For extraction/reading you need a local Ollama with the model used by the frozen research protocol:

ollama pull gemma4:12b

from scene_memory import SceneMemory, make_client

client = make_client("ollama:gemma4:12b")
memory = SceneMemory("./cache", client, conversation_id="demo")

# WRITE — assertions become facts; questions never touch the memory
memory.remember("I live in Lisbon and work as a software engineer.")
memory.remember("My cat is called Whiskers.")
memory.remember("Actually, I moved to Porto last month.")

# READ — answers from the scene, not by re-reading the conversation
memory.answer("Where do I live?")          # -> "porto"
memory.answer("Where did I live before?")  # -> "lisbon"  (superseded history)

# PROVENANCE — the answer carries the facts that produced it
answer = memory.answer("Where do I live?")
answer == "porto"                          # it IS a string; old call sites unchanged

for fact in answer.fatos:
    print(fact.indice, fact.slot, fact.valor, fact.corrente)
# 4 user|location porto True
# 5 user|date+move last month True

for alert in answer.alertas:
    print(alert.codigo)
# valor_ausente_no_texto   -- the reader cited fact #5 but "last month" never
#                             reached the answer text. Informational, not an
#                             error: paraphrase legitimately drops values.

# INSPECT — the scene is a plain, walkable structure
for key, slot in memory.scene.slots.items():
    current = slot.current
    print(key, "->", current.display if current else None,
          "| history:", [v.display for v in slot.superseded])

What you get back is honest about its own limits, in two ways.

It shows its work. answer() returns a Resposta, which subclasses str — it compares, prints and serialises like the plain string it replaced, so existing call sites need no change — and carries .fatos, the scene facts the reader said it used, resolved back to concrete slots and values. Each one knows whether the value it cites is the slot's current value or a superseded one. (Note the one trap: string operations like .strip() return a plain str and drop the provenance. Keep the object if you need it.)

And the code checks that work. .alertas carries what the verifier contested about the citation, structurally, without pretending to judge semantics: valor_superado when a cited value is no longer current, indice_invalido for a fact number that does not exist, sem_citacao when the reader cited nothing. A superseded citation is not automatically wrong — it is the right answer to a question about the past — so the alert says so rather than accusing.

Measured cost of asking for citations, paired over the 422 held-out questions: 120/422 → 122/422, McNemar p = 0.75. It is free (results/portao_citacao/).

The lower-level ask() still carries abstained (the scene had no matching slot — not the same as "I don't know") and may_be_partial (other slots also matched the question, and single-slot reading returned just one).

How the scene works, in 30 seconds

Every asserted fact becomes a (subject, relation, object) triple filed under a canonical key — user|location — so paraphrases land in the same slot without embeddings. A slot keeps all values it ever had, ordered by logical time; the newest is current, the rest are superseded:

user|location
  t1  lisbon    superseded
  t3  porto     current

Updates supersede, enumerations coexist, and questions about the past read the history. Writing is immutable: each turn produces a new scene, so a failure mid-turn never leaves memory half-written.

Live demo — chat with the scene at its side

A minimal chat where you watch every slot being born, updated and superseded, turn by turn. Under each answer it prints the facts that produced it — fact number, slot, value, and a SUPERSEDED mark when the reader cited a value the scene has since replaced. The right-hand panel is a full inspector: click a slot for its complete value history with origin text, filter slots live, see which slot answered. There is a button to reset the scene without restarting the server:

ollama serve                                   # in another terminal
python3 demo/chat/servidor.py                  # http://127.0.0.1:8000
# from another device on your Tailscale network:
python3 demo/chat/servidor.py --host $(tailscale ip -4)

The demo is also an instrument: its API-friction findings (24 so far, 15 already fixed upstream) are logged in demo/chat/ACHADOS.md. The open ones are worth reading — they are where this system currently lies to you, written down in the same detail as the wins.

Reproducing the evaluation

python3 tests/test_core.py                     # core tests, no LLM
python3 -m pytest tests/ -q                    # full suite (475 tests)

# retrieval baselines (no LLM):
python3 -m scene_memory.cli run --dataset lme_ku_s --retriever bm25 grep vector --k 1 5

# structural scene on real conversation (needs Ollama + gemma4:12b):
python3 -m scene_memory.cli scene-query-conv --dataset lme_ku_s \
    --resolver-model ollama:gemma4:12b --select-model ollama:gemma4:12b

Benchmark datasets are not shipped in the repository. Fetch scripts live in eval/ (fetch_mab.py for MemoryAgentBench via HuggingFace; LongMemEval per its official instructions into data/lme_raw/). The benchmark-system caches under results/extractor_experiment/ are likewise local-only; the scripts re-extract on demand.

Repository layout

Limits

Declared, measured, and kept visible — not fine print:

• Benchmark numbers are benchmark numbers. The router triggers, absence gate and date anchors are regexes calibrated on LongMemEval's English corpus; in another language or domain they do not fire without re-measurement.
• Two indicators sit at their measured oracle ceiling (temporal at 128, multi-session above its own ceiling): further progress there requires a stronger reader model, not better retrieval — this is measured, not assumed.
• Single-slot reading answers one slot. Questions whose answer spans several facts get a partial answer with may_be_partial=True; composing across slots is the multi-hop path, a separate experiment.
• Reported speech is an open finding (demo finding #15): "they said there was gold" is extracted as a plain fact and the negation is lost as structure. The extractor already marks modality informally; making it a contract — and making the reader state it — is the mapped next research step.
• Frozen defaults are the research protocol (v4, passes=2, merge_sim=0.5). Changing them breaks comparability with every published measurement, so changing them is an explicit caller decision.

Research documentation

The journey from a 12-example probe to the full result — including the pre-registered held-out evaluation where the central thesis initially failed and what was done about it honestly — is the paper: docs/research-journey.md (Portuguese original). The final benchmark campaign (arms 28–50, six integrated mechanisms, two external instruments, ten falsifications, one retraction) is in results/extractor_experiment/RELATORIO_ARQUITETURAS.md. The campaign report is in Portuguese; the commit history narrates the same story with predictions committed before every measurement.

Contributing

CONTRIBUTING.md has the seven open findings in a table with the shape of the work for each, the frozen-protocol rule, and what a change has to look like to be believable here. Three ways in, by effort: run the live demo and report what breaks; take an open finding; or replicate where these numbers explicitly do not claim to hold — another language, another model, another domain. A clean negative result gets published as one.

License

MIT.