personos 0.1.3

Multimodal long-term memory for robots, smart glasses and agents.
Video, images and conversations in — memory about people, with names, out.

English · 简体中文

The film · Why identity · Highlights · Benchmarks · Quickstart · How it works · Install · Roadmap

The film

https://github.com/user-attachments/assets/7252ab1e-9fe8-4c83-b2b1-d3da03b83504

Who ate the Halloween candy? Pebble, a little home robot, watches a party through its one big lens. Three kids arrive as strangers, earn their names from what they call each other, vanish under identical white sheets — and in the middle of a blackout, one of them puts on someone else's sheet to frame him.

The notebook on the right is Pebble's memory, and it works the way PersonOS works: the screenplay is written the moment things happen, with neutral codes (P1, P2…); who's who is kept separately, revised as evidence comes in, and only applied to the screenplay when the night's memory is saved. Next morning everyone blames the boy whose sheet it was. Pebble doesn't.

Names are earned. "Whoa… Mia, the candy's over there!" turns stranger P1 into a guess, Mia? — confirmed only when it happens again.	No faces, still known. Under the sheets, shoes, height, a bell on a shoelace and voices keep every ghost attached to the right kid.	Contradictions are caught. The ketchup sheet says Leo; Leo's voice comes from the kitchen. One person can't be in two places.
…and repaired. A high giggle and pink sneakers move the ghost to Mia's card. What was recorded is never edited — only who it refers to.	Identity lands last. When the night is saved, codes resolve to names across the whole screenplay: one complete record, every person correct.	Answers cite evidence. "Who was it?" — Mia, with the exact lines and a snapshot of the sneakers that gave her away.

_{An original, code-drawn short (p5.js + p5.brush); voices and sound by ElevenLabs. The story is staged, the mechanics are the real ones — see How it works.}

Why identity

A memory system that understands video but not people writes this down:

- [21:15] A ghost in a ketchup-stained sheet went to the candy bowl.
- [21:16] A ghost took a candy.  [21:22] A ghost took a candy.  [21:29] A ghost took a candy.

Which ghost? Ask "who ate the candy?" and the best it can do is follow the sheet — straight to the wrong kid. PersonOS writes this instead:

+ [21:15] Mia — wearing Leo's ketchup-stained sheet — went to the candy bowl.
+ [21:16] Mia took a candy.  [21:22] Mia took a candy.  [21:29] Mia took a candy.

Every line points at a stable character that persists across clips and sessions. Names are earned, not enrolled — someone becomes "Mia" because people call her Mia; until then she is a stable handle you can still ask about. And the device itself is a character too, because "who did this?" sometimes has the answer "you did".

Highlights

Benchmarks

M3-Bench-robot — long-video memory from a robot's point of view: 100 videos, 1,276 questions. This is the benchmark identity is built for.

Video-MME, long videos, no subtitles — answered from memory alone: the video is ingested into PersonOS and the answering model never sees it.

Text memory, no compromise — the same store on conversational benchmarks:

_{Accuracy in %. Video memory built with Qwen3.5-Omni-Plus, questions answered from memory by GPT-5.5. ¹ Same dataset, memory model and answer model as PersonOS; prompt templates and retrieval budgets are not guaranteed identical. Text benchmarks reproduce from scripts/bench; video evaluation code ships with the accompanying paper.}

Quickstart

pip install personos
export PERSONOS_LLM_API_KEY=sk-...
export PERSONOS_LLM_BASE_URL=https://api.openai.com/v1   # any OpenAI-compatible gateway

from personos import Memory

m = Memory()   # SQLite under ~/.personos — nothing to provision

m.add("I moved from Hangzhou to Shanghai in June", user_id="alice", session_id="s1")
m.end_session(user_id="alice", session_id="s1", sync=True)

print(m.search("where do I live?", user_id="alice").ans.answer)
# -> Shanghai

Give it eyes. Same three calls, now with video and person identity:

pip install 'personos[identity]'
export PERSONOS_VIDEO_BACKEND=real
export PERSONOS_MLLM_API_KEY=sk-...  PERSONOS_MLLM_MODEL=...   # any model that accepts video

for clip in ["clip000.mp4", "clip001.mp4", "clip002.mp4"]:          # consecutive clips
    m.add([{"role": "user", "content": "", "video": clip}],
          user_id="robot", session_id="living-room")

m.end_session(user_id="robot", session_id="living-room", sync=True)  # identities are committed here

out = m.search("What's the name of the girl at the table, and what does she study?", user_id="robot")
print(out.ans.answer)       # Alice ... math
print(out.ans.cited_cells)  # the episodes (and clips) the answer came from

Not sure what your setup can do? Run personos doctor. More in examples/: a week of conversation · a photo · video with person identity.

How it works

① Perceive. A multimodal model watches each clip and writes a screenplay: who is on screen — as clip-local codes, not names — who says what, what happens, when.

② Resolve identity. Codes are matched against known characters through a face / body / voiceprint cloud. Code-level checks look for physical contradictions (one person in two places, two people sharing one identity) and send them back to the model for repair. Within a session, character chains collect evidence — a first sighting, a better face, a clearer voice, a name said out loud — and are adjudicated into persistent characters when the session closes.

③ Remember. The screenplay is resolved through those identities and written into a layered, append-only store:

evidence  ──►  memcell (episode)  ──►  atom  ──►  atom_chain
raw input      the narrative unit      the          the same fact over time,
never edited   used for answering      retrieval    grouped, never collapsed
                                       anchor

Atoms are short propositions that embed well — the retrieval index. Episodes are what the model actually reads.

④ Recall. search() rewrites the question, resolves who it is about, retrieves, reranks, adjudicates a draft and escalates to a multi-step agent when needed — and returns how it got there:

out = m.search("who ate the candy?", user_id="pebble")
out.ans.answer   # the answer
out.rw.subject   # who the question was resolved to be about
out.hits         # atoms retrieved, in fusion order
out.reviews      # what the adjudicator said about the draft
out.to_public()  # ...or a plain dict

Built for

Installation

The core has 7 dependencies — no torch, no LangChain, no web framework. Everything heavier is opt-in:

MissingCapability: video understanding is unavailable: no multimodal model is configured
  To enable it: set PERSONOS_MLLM_API_KEY and PERSONOS_MLLM_MODEL

m.add(messages, user_id=..., session_id=...)   # text, a dict with an image, or video clips
m.end_session(user_id=..., session_id=...)     # close the segment; build memories and commit identities
m.search(query, user_id=..., mode="auto")      # "auto" | "fast" | "deep"
m.profile(user_id=...)                         # distilled user profile
m.trace(node_id, user_id=...)                  # provenance, both directions
m.capabilities()                               # what this configuration can do
m.reset(user_id=...)                           # delete one user's data

Roadmap

• Layered append-only memory with fast + deep recall
• Image ingest and recall
• Video memory with face / body / voiceprint identity, self-repair and cross-session recognition
• Async ordered ingestion, HTTP server
• Memory inspector UI — timeline, character gallery, answer traces
• MCP server for Claude Code, Cursor and other MCP clients
• Integrations: LangGraph, OpenAI Agents SDK, ROS 2
• AsyncMemory
• Streaming ingestion for live camera feeds
• Lighter identity install (faces without torch)
• Paper + full video evaluation code

Status

0.1.x — the memory pipeline runs in production; the packaging around it is new, and the public API may still move before 1.0. Releases follow RELEASE.md.

Contributing

Issues and PRs are welcome — see CONTRIBUTING.md. The test suite runs with pytest and needs no external services.

Acknowledgements

• The film is original work, drawn in code with p5.js and p5.brush; voices and sound effects by ElevenLabs.
• The M3-Bench-robot benchmark and the video samples used in the examples come from M3-Agent / M3-Bench (ByteDance-Seed, CC BY-NC-SA 4.0). Samples are downloaded on demand and are not redistributed in this repository.

License

Apache-2.0. See LICENSE.