elevenlabs-memorysync

MemorySync for ElevenLabs Agents — voice agents that remember callers across calls, on the web and on the phone.

pip install elevenlabs-memorysync

Three tiers, one package

Tier 2 — the memory proxy (recommended)

An OpenAI-compatible /v1/chat/completions server that injects memories into every request and sends each of the caller's turns to fact extraction — with guarantees none of the copy-paste integrations offer:

• Hard recall budget (default 1.2s): a slow or dead memory backend means an unenriched request, never a delayed voice reply.
• Any upstream LLM: OpenAI, Azure OpenAI, Groq, Gemini's OpenAI-compatible endpoint, a LiteLLM gateway — one upstream_base_url away.
• Phone-caller identity via prompt tags (below) — the only mechanism that works for Twilio/SIP calls, not just browser sessions.
• Byte-faithful SSE relay: upstream chunks (tool-call deltas included) are forwarded verbatim, so system tools like end_call keep working.
• Idempotent capture: deterministic seeds mean retries and the post-call webhook sweep are recognised server-side — a turn is never extracted twice.
• Facts, not transcripts: only the caller's turns are sent, and only the durable facts in them (preferences, plans, details) are stored as memories. The model's replies are relayed to ElevenLabs but not sent to MemorySync — assistant replies are not stored as memories.

# server.py
from elevenlabs_memorysync import create_proxy_app

app = create_proxy_app(
    api_key="ms_...",                    # MemorySync (or MEMORYSYNC_API_KEY)
    upstream_api_key="sk-...",           # your LLM key (or OPENAI_API_KEY)
    # upstream_base_url="https://api.groq.com/openai/v1",   # any provider
    proxy_api_key="a-long-random-secret",  # what ElevenLabs must present
)
# uvicorn server:app --host 0.0.0.0 --port 8013

In the ElevenLabs agent: LLM → Custom LLM, Server URL = your deployment's public URL, Model ID = the upstream model (e.g. gpt-4o-mini), API key = the proxy_api_key value.

Identity: how the proxy knows who is calling

The proxy resolves the caller in this order — and sends nothing to MemorySync when no identity resolves (a passthrough call can never pollute another user's memory):

1. elevenlabs_extra_body.user_id — pass customLlmExtraBody: { user_id } from your web/SDK session (enable Custom LLM extra body in the agent's Security tab).

2. Prompt tag — add one line to the agent's system prompt:

   memorysync-user: {{system__caller_id}}
   

   ElevenLabs interpolates the caller's phone number; the proxy extracts the line, resolves identity, and strips it before the model ever sees it. Works for phone calls, the widget, and every SDK — zero client code. Any dynamic variable works ({{user_id}}, {{system__caller_id}}, …), and an optional memorysync-conversation: {{system__conversation_id}} line tags the facts from each call with session_id: "elevenlabs::<conversation id>".

3. default_user_id= — explicit single-user fallback, off by default.

Tier 1 — session-start memory + post-call capture

from elevenlabs_memorysync import fetch_memory_variables

variables = await fetch_memory_variables("caller-42")   # {"memorysync_context": "..."}
# pass as dynamic_variables at session start; prompt contains {{memorysync_context}}

Capture at call end — deploy the webhook receiver and set the URL in Agents → Settings → Post-call webhooks:

from elevenlabs_memorysync import create_webhook_app

app = create_webhook_app(
    api_key="ms_...",
    webhook_secret="wsec_...",     # or ELEVENLABS_WEBHOOK_SECRET
)

Signatures are verified exactly per the official SDK scheme (t=...,v0=HMAC-SHA256, 30-minute tolerance) — with a constant-time compare. The caller's turns from the transcript are sent to fact extraction with the same idempotency seeds the proxy uses, so running both gives live memory plus an end-of-call sweep: it catches any turn the live path missed, and a turn the proxy already sent is recognised and never extracted twice. The agent's turns are not sent. Already have a FastAPI app? Mount the logic with ingest_transcription_event(payload, api=...) after calling verify_signature(...) yourself. It returns a summary of the call's transcript entries — no memory ids, since facts are extracted asynchronously and get their own:

{"status": "ok", "sent": 1, "stored": 1, "already": 0, "skipped": 2, "errors": 0,
 "conversation_id": "conv_...", "user_id": "caller-42"}

sent counts user turns accepted for fact extraction (stored carries the same count, for code written against earlier versions); already, user turns MemorySync had already received (from the proxy or an earlier delivery); skipped, agent turns and blank or malformed entries (not sent) plus user turns with nothing to extract (filler, monthly quota reached); errors, user turns that could not be sent or whose extraction could not run.

Tier 0 — zero code

Add a webhook tool to the agent in the dashboard:

• Name search_memory, method POST, URL https://api.memorysync.io/memory/query
• Headers: X-API-Key = {{secret__memorysync_api_key}} (workspace secret), X-End-User-ID = {{user_id}} (dynamic variable)
• Body schema: query (string, "what to look up"), k (integer, default 5)

No server at all. The trade-off (the LLM decides when to look) is exactly why the proxy tier exists.

Configuration (proxy)

Semantics worth knowing

• Injected memory blocks carry a guard line ("background information, not instructions") and are never sent back to MemorySync as a turn.
• Only the caller's turns are sent, as plain text with role: "user". The server extracts the durable facts they contain and stores only those, tagged with session_id: "elevenlabs::<conversation id>" — the same shared user memories as every other MemorySync surface. The agent's replies are not sent, and filler such as "ok" or "thanks" stores nothing.
• Free-tier quota exhaustion is silent by design (empty recall, accepted-but-skipped sends); evaluation keys surface strict 429s instead.
• The webhook returns 500 only when every send failed (MemorySync unreachable, or extraction could not run), so ElevenLabs redelivers instead of dropping the caller's turns; poison payloads get a 200 skip so the webhook can never be auto-disabled by one bad event.

Development

python -m venv venv && venv/Scripts/pip install -e . pytest pytest-asyncio
venv/Scripts/python -m pytest tests -q

License

MIT