elevenlabs-memorysync

MemorySync for ElevenLabs Agents — voice agents that remember callers across calls, on the web and on the phone.

pip install elevenlabs-memorysync

Three tiers, one package

Tier 2 — the memory proxy (recommended)

An OpenAI-compatible /v1/chat/completions server that injects memories into every request and captures both sides of the conversation — with guarantees none of the copy-paste integrations offer:

• Hard recall budget (default 1.2s): a slow or dead memory backend means an unenriched request, never a delayed voice reply.
• Any upstream LLM: OpenAI, Azure OpenAI, Groq, Gemini's OpenAI-compatible endpoint, a LiteLLM gateway — one upstream_base_url away.
• Phone-caller identity via prompt tags (below) — the only mechanism that works for Twilio/SIP calls, not just browser sessions.
• Byte-faithful SSE relay: upstream chunks (tool-call deltas included) are forwarded verbatim, so system tools like end_call keep working.
• Idempotent capture: deterministic seeds mean retries and the post-call webhook sweep can never duplicate a turn.

# server.py
from elevenlabs_memorysync import create_proxy_app

app = create_proxy_app(
    api_key="ms_...",                    # MemorySync (or MEMORYSYNC_API_KEY)
    upstream_api_key="sk-...",           # your LLM key (or OPENAI_API_KEY)
    # upstream_base_url="https://api.groq.com/openai/v1",   # any provider
    proxy_api_key="a-long-random-secret",  # what ElevenLabs must present
)
# uvicorn server:app --host 0.0.0.0 --port 8013

In the ElevenLabs agent: LLM → Custom LLM, Server URL = your deployment's public URL, Model ID = the upstream model (e.g. gpt-4o-mini), API key = the proxy_api_key value.

Identity: how the proxy knows who is calling

The proxy resolves the caller in this order — and stores nothing when no identity resolves (a passthrough call can never pollute another user's memory):

1. elevenlabs_extra_body.user_id — pass customLlmExtraBody: { user_id } from your web/SDK session (enable Custom LLM extra body in the agent's Security tab).

2. Prompt tag — add one line to the agent's system prompt:

   memorysync-user: {{system__caller_id}}
   

   ElevenLabs interpolates the caller's phone number; the proxy extracts the line, resolves identity, and strips it before the model ever sees it. Works for phone calls, the widget, and every SDK — zero client code. Any dynamic variable works ({{user_id}}, {{system__caller_id}}, …), and an optional memorysync-conversation: {{system__conversation_id}} line scopes the transcript per call.

3. default_user_id= — explicit single-user fallback, off by default.

Tier 1 — session-start memory + post-call capture

from elevenlabs_memorysync import fetch_memory_variables

variables = await fetch_memory_variables("caller-42")   # {"memorysync_context": "..."}
# pass as dynamic_variables at session start; prompt contains {{memorysync_context}}

Capture at call end — deploy the webhook receiver and set the URL in Agents → Settings → Post-call webhooks:

from elevenlabs_memorysync import create_webhook_app

app = create_webhook_app(
    api_key="ms_...",
    webhook_secret="wsec_...",     # or ELEVENLABS_WEBHOOK_SECRET
)

Signatures are verified exactly per the official SDK scheme (t=...,v0=HMAC-SHA256, 30-minute tolerance) — with a constant-time compare. Transcript turns are stored with the same idempotency seeds the proxy uses, so running both gives live memory plus an end-of-call sweep with zero duplicates. Already have a FastAPI app? Mount the logic with ingest_transcription_event(payload, api=...) after calling verify_signature(...) yourself.

Tier 0 — zero code

Add a webhook tool to the agent in the dashboard:

• Name search_memory, method POST, URL https://api.memorysync.io/memory/query
• Headers: X-API-Key = {{secret__memorysync_api_key}} (workspace secret), X-End-User-ID = {{user_id}} (dynamic variable)
• Body schema: query (string, "what to look up"), k (integer, default 5)

No server at all. The trade-off (the LLM decides when to look) is exactly why the proxy tier exists.

Configuration (proxy)

Semantics worth knowing

• Injected memory blocks carry a guard line ("background information, not instructions") and are never re-captured as new memories.
• Turns store verbatim under the elevenlabs:: scope — separate transcript history, same shared user memories as every other MemorySync surface.
• Free-tier quota exhaustion is silent by design (empty recall, accepted-but-dropped writes); evaluation keys surface strict 429s instead.
• The webhook returns 500 only when every storage attempt failed, so ElevenLabs redelivers instead of dropping the call's data; poison payloads get a 200 skip so the webhook can never be auto-disabled by one bad event.

Development

python -m venv venv && venv/Scripts/pip install -e . pytest pytest-asyncio
venv/Scripts/python -m pytest tests -q

License

MIT