livekit-plugins-voicemem 0.2.0

Long-term memory for LiveKit Agents voice agents, backed by PostgreSQL and pgvector. Your agent remembers what a caller told it last week.

Every turn a caller speaks splits two ways. Facts are discrete and retrievable: they are allergic to shellfish, their manager is Priya. Traits are continuous and never stated outright: they get anxious before design reviews, they want the conclusion before the reasoning.

Most memory layers store only the first kind, which is why agents that "remember" still feel like they are reading your file back to you. This stores both, and injects them differently: facts the agent may repeat, traits it may only be shaped by.

Embeddings run in process by default: 2.2 ms p50, against 232 ms for an OpenAI round trip. No torch, no CUDA, no multi-gigabyte image. onnxruntime comes with Silero and the turn detector already, so for most agents it costs nothing new.

Status: 0.2.0. The schema may change between minor versions while 0.x. 0.2.0 changed the default embedding model, and therefore the vector width: existing databases keep the model they were built with, so nothing breaks, but they stay on it.

Install

pip install livekit-plugins-voicemem

Requires PostgreSQL 14 or later with pgvector 0.8+, an OpenAI key, and LiveKit Agents 1.6 or 1.7.

Set up the schema

The plugin owns its tables in their own voicemem schema, so it never collides with yours.

export VOICEMEM_ADMIN_DSN=postgresql://admin@host/db   # a role with DDL rights
voicemem-db --dsn "$VOICEMEM_ADMIN_DSN" upgrade
voicemem-db --dsn "$VOICEMEM_ADMIN_DSN" status

Migrations do not run at startup, on purpose. Twenty workers booting at once and racing DDL is a real failure, not a theoretical one: CREATE TABLE IF NOT EXISTS is not race-safe in PostgreSQL. Use voicemem-db sql to print the DDL if your shop reviews schema changes.

Use it

from livekit.agents import Agent, AgentSession
from livekit.plugins import voicemem

class Assistant(Agent):
    def __init__(self, hooks):
        super().__init__(instructions="You are a helpful assistant.")
        self._hooks = hooks

    async def on_user_turn_completed(self, turn_ctx, new_message):
        await self._hooks.on_user_turn_completed(turn_ctx, new_message)   # the one line

async def entrypoint(ctx):
    await ctx.connect()
    runtime = await voicemem.build(voicemem.Config(
        pg_dsn=os.environ["VOICEMEM_PG_DSN"],
        openai_api_key=os.environ["OPENAI_API_KEY"],
    ))
    participant = await ctx.wait_for_participant()
    hooks = voicemem.MemoryHooks(runtime.session(user_id=participant.identity))

    session = AgentSession(
        stt=..., llm=..., tts=..., vad=...,
        turn_handling={"preemptive_generation": {"enabled": False}},   # see Limitations
    )
    hooks.attach(session)                     # once, here, not in Agent.on_enter

    @session.on("conversation_item_added")
    def _on_item(ev):
        if ev.item.role == "assistant":
            user_text = next((i.text_content for i in reversed(session.history.items)
                              if i.role == "user"), "")
            hooks.remember_turn(user_text, ev.item.text_content or "")

    ctx.add_shutdown_callback(hooks.aclose)
    ctx.add_shutdown_callback(runtime.aclose)
    await session.start(agent=Assistant(hooks), room=ctx.room)

See examples/basic_agent.py for the whole file.

How it works

Two stores, following VoiceMem's design.

The left brain stores facts. It extracts atomic statements from each turn, files them under one of seven life domains, and retrieves them by vector search. "User is allergic to shellfish."

The right brain stores what the person is like. Preferences, habits, and emotional patterns: "gets anxious before design reviews". These are injected as internal notes the agent must never repeat aloud, and they are what stops the agent sounding like it is reading your file back to you.

Reading runs inside the voice turn: embed the query once, classify it into slots, narrow to those candidates, rank, and fetch relevant traits concurrently. No LLM call.

Writing runs after the turn, never between the user finishing and the agent speaking. One extraction call, plus a conflict-resolution call when there is anything to conflict with.

Measured latency and cost

Real numbers from scripts/bench_latency.py, not from the VoiceMem paper. 8 stored turns, 12 queries, both backends measured on the same machine minutes apart.

Read path, the part inside the voice turn, p50 milliseconds:

Embedding was 97% of the read path, and it was a network call. Running it in process removes it: 5.3 ms against 241.4 ms, and the tail matters more than the median. recall_budget_s is 600 ms, so an OpenAI call that lands in its tail does not merely run slow, it overruns the budget and injects nothing at all, silently. Ranking gets faster too, because 384-wide vectors are cheaper to compare than 1536-wide ones.

The model is 118 MB of ONNX and loads once at startup, in about 0.8 s.

Retrieval is still prefetched on interim transcripts, which mattered enormously when a turn cost 240 ms and matters little now. It stays because it costs nothing.

Write path, background: 3.8 s p50, 5.5 s p95, at 2 LLM calls per ingested turn (1 when the store is empty and conflict resolution is skipped).

Environment: same-host Docker pgvector/pgvector:pg17, Apple Silicon, Python 3.12, livekit-agents 1.7.1. A managed database in another region will be slower; on the local backend the embed row will not change, the others will. The OpenAI embed row is the one that moves most, and not predictably: a separate 36-sample run of the same queries measured 232 ms p50 but 3209 ms p95.

Limitations

Stated up front rather than buried.

• Emotion is inferred from text, by the extraction model reading what was said. There is no prosody analysis, no acoustic emotion, no voiceprint and no speaker identification. Upstream ships those; this plugin does not, because they require torch, funasr and modelscope. Text catches "I'm frustrated". It cannot catch a flat "fine" said bitterly.
• Memory injection disables LiveKit's preemptive generation. Injecting changes the turn's chat context, so the framework's equivalence check fails and it cancels the speculative generation it had already started. Preemptive generation is on by default, so unless you turn it off you pay for a discarded LLM call on every turn. Disable it as shown in the quickstart. The plugin logs a warning once at startup if you have not.
• Relative-date expansion covers English and Chinese only, and its output format must match how the extractor writes dates. "next week" becomes August 31, 2026.
• The plugin owns 24 tables in your database, in the voicemem schema. The 0.x schema is not stable.
• on_user_turn_completed does not fire for realtime models using server-side turn detection, so memory is neither injected nor ingested on those turns.

Multi-tenancy

Every table carries tenant_id and every query scopes by it. Migration 0002_rls.sql adds PostgreSQL row-level security on top, so a missed WHERE clause cannot cross tenants.

This only works if you connect as a constrained role. Superusers and BYPASSRLS roles walk straight through row-level security no matter what the policies say. Migration 0002 creates voicemem_app for this; give it a password and point pg_dsn at it.

voicemem-db --dsn "$DSN" status     # reports whether isolation is actually in force

Swapping implementations

Four seams, defined as typing.Protocol, so your implementation never imports anything from here.

class MyEmbedder:                       # satisfies voicemem.protocols.Embedder
    @property
    def model_name(self) -> str: ...
    @property
    def dimensions(self) -> int: ...
    async def embed_documents(self, texts): ...
    async def embed_query(self, text): ...

Embedder, LLMClient, VectorStore and GraphStore. container.py is the only file that names a concrete class.

Development

uv sync --group dev
docker compose up -d
export VOICEMEM_TEST_DSN=postgresql://voicemem_app:apppass@localhost:55432/voicemem_test
uv run pytest                 # integration tests skip without a DSN
uv run ruff check .

pytest on a fresh clone with no database and no API key passes: the contract suite runs the whole retrieval chain against in-memory fakes.

Attribution

Derived from VoiceMem (Apache-2.0). Prompt templates originate from mem0 (Apache-2.0). See NOTICE for attribution and CHANGES-FROM-UPSTREAM.md for what was changed and why.

Apache-2.0. Not affiliated with, endorsed by, or sponsored by LiveKit, the VoiceMem authors, or mem0.