Generic LLM Memorizer

A library for AI agent long-term memory. Provides memorize and search tools that any agent (pydantic-ai, Google ADK, etc.) can use to store and retrieve conversational knowledge — including data from immersive devices (VR/AR headsets, smart glasses, etc.).

Features

• Fact Extraction — Automatically extracts relevant facts from conversations (preferences, skills, context, events, demographics)
• User Profile — Builds and maintains a user profile (name, age, hobbies, occupation)
• Conversation Summary — Generates concise summaries of each conversation
• SKOS Knowledge Base — Stores ontological concepts following the SKOS standard
• Semantic Search — Hybrid search combining:
  • Dense vector search (cosine similarity on embeddings) — requires optional rag install
  • Keyword search (TF-IDF) — built-in
  • LLM reranking — re-ranks top results by semantic relevance
• VR/AR Support — Memorize sessions from immersive devices (Meta Quest Pro, Apple Vision Pro, etc.) via pluggable DeviceAdapter
• Persistence — JSON files, one directory per user (pluggable StorageBackend)

Installation

uv sync
# Optional: enable RAG with local sentence-transformers
uv sync --group rag

Quick Start

import asyncio
from pydantic import BaseModel
from generic_llm_memorizer import Memorizer

# Your LLM call function (OpenAI, Ollama, Anthropic, etc.)
async def my_llm_call(prompt: str, system_prompt: str, response_model: type[BaseModel]):
    # Call your LLM API and return an instance of response_model
    ...

async def main():
    mem = Memorizer(user_id="user_123", llm_call=my_llm_call)

    # Tool 1: memorize a conversation
    result = await mem.memorize(
        conversation=[
            "User: Bonjour, je m'appelle Marie et je suis développeuse Python",
            "Assistant: Enchantée Marie !",
        ],
        session_id="session_1",
    )
    print(f"Facts added: {result.facts_added}")
    print(f"Profile updated: {result.profile_updated}")
    print(f"Summary: {result.summary}")

    # Tool 2: search stored memory
    search_result = await mem.search("What does the user do?")
    print(search_result.to_context_string())

asyncio.run(main())

With RAG (Dense Vector Search)

pip install generic-llm-memorizer[rag]

from generic_llm_memorizer import Memorizer
from generic_llm_memorizer.embeddings import SentenceTransformerEmbeddings

mem = Memorizer(
    user_id="user_123",
    llm_call=my_llm_call,
    embedder=SentenceTransformerEmbeddings(),
)
# memorize() will now also build a vector index
# search() will use hybrid vector + keyword matching

With a custom embedding API

from generic_llm_memorizer.embeddings import OpenAIEmbeddings

async def my_embed_fn(texts: list[str]) -> list[list[float]]:
    # call your embedding API
    ...

mem = Memorizer(
    user_id="user_123",
    llm_call=my_llm_call,
    embedder=OpenAIEmbeddings(embed_fn=my_embed_fn),
)

VR/AR Support

The library can memorize insights from immersive device sessions (VR headsets, AR glasses, etc.).

Built-in: Meta Quest Pro

# Sample raw data from a Meta Quest Pro headset
vr_session_data = {
    "session_id": "VR_SESSION_001",
    "user_id": "user_123",
    "device_info": {"device_type": "Meta Quest Pro", ...},
    "data_streams": [
        {
            "timestamp": "2026-07-27T14:30:15Z",
            "type": "audio",
            "subtype": "voice",
            "data": {
                "voice_command": "Open multiplication table",
                "emotion": {"detected": "neutral", "confidence": 0.87},
            },
        },
        {
            "timestamp": "2026-07-27T14:32:05Z",
            "type": "visual",
            "subtype": "facial_expressions",
            "data": {
                "emotion": "confused",
                "confidence": 0.92,
                "expressions": {"eyebrows_raised": True, "frown": True},
            },
        },
        # ... eye tracking, hand tracking, biometric, movement, social, etc.
    ],
}

result = await mem.memorize_vr_session(
    raw_session_data=vr_session_data,
    device_type="Meta Quest Pro",
    session_id="vr_session_001",
)
print(f"Facts added: {result.facts_added}")   # e.g. "User showed confusion with multiplication table"

The adapter automatically:

1. Parses the device-specific JSON into structured VRSession
2. Extracts insights — emotion timeline, engagement metrics, biometric trends, social proximity, spatial context
3. Converts to text — translates gestures, gaze, biometrics, etc. into human-readable messages
4. Runs the memorize pipeline with a VR-specific LLM prompt
5. Persists VRInsights alongside existing memory data

Adding a custom device

Implement DeviceAdapter and register it:

from generic_llm_memorizer import DeviceAdapter, VRSession, VRInsights, register_device

class AppleVisionProAdapter(DeviceAdapter):
    @property
    def device_type(self) -> str:
        return "Apple Vision Pro"

    async def parse_session(self, raw_data: dict) -> VRSession:
        # Map Apple Vision Pro JSON → VRSession
        ...

    async def extract_insights(self, session: VRSession) -> VRInsights:
        # Extract emotion timeline, biometric trends, etc.
        ...

    async def to_conversation(self, session: VRSession) -> list[str]:
        # Convert data streams to text messages
        ...

register_device("Apple Vision Pro", AppleVisionProAdapter)

Once registered, it works with the same memorize_vr_session() method.

Data streams understood

The VRSession model accepts any (type, subtype) combination in its DataStream list. The built-in MetaQuestProAdapter handles these:

API

Memorizer

Memorizer(
    user_id: str,
    llm_call: LLMCall,
    store: StorageBackend | None = None,       # defaults to FileStorageBackend
    system_prompt: str | None = None,
    batch_size: int = 10,
    max_batches: int = 3,
    embedder: EmbeddingProvider | None = None,  # enables RAG vector search
)

memorize(conversation, session_id) -> MemorizeResult

Extract facts, profile, summary, and SKOS concepts from a conversation.

• conversation: list[str] — Messages in "role: content" format
• session_id: str — Session identifier for persistence
• Returns MemorizeResult(facts_added, profile_updated, summary, skos_concepts_added)

memorize_vr_session(raw_session_data, device_type, session_id) -> MemorizeResult

Extract facts from an immersive device session.

• raw_session_data: dict — Raw JSON from the device headset
• device_type: str — Device identifier (e.g. "Meta Quest Pro")
• session_id: str — Session identifier for persistence
• Returns MemorizeResult(facts_added, profile_updated, summary, skos_concepts_added)

Insights (VRInsights) are automatically persisted alongside the memory.

search(query, scope=ALL, top_k=10, use_llm_rerank=True) -> SearchResult

Search stored memory. Uses hybrid vector + keyword + optional LLM rerank.

• query: str — Natural language query
• scope: SearchScope — FACTS, SKOS, PROFILE, or ALL
• top_k: int — Max results to return
• use_llm_rerank: bool — Whether to rerank with the LLM
• Returns SearchResult(facts, skos_concepts, user_profile) with to_context_string()

Storage

from generic_llm_memorizer import FileStorageBackend

store = FileStorageBackend(base_path="./my_memories")
mem = Memorizer(user_id="user_123", llm_call=my_llm_call, store=store)

Persisted to ./my_memories/{user_id}/:

• memory.json — User profile and facts
• skos_knowledge_db.json — SKOS ontology
• {session_id}/conversation.json — Conversation history
• {session_id}/vr_insights.json — VR/AR session insights

VR/AR models

RAG (Optional)

Architecture

Memorizer
├── memorize(conversation)
│   ├── _extract_facts()          → LLM → Facts
│   ├── _extract_user_profile()   → LLM → User
│   ├── _summarize()              → LLM → Summary
│   ├── _build_skos()             → LLM → SkosKnowledgeDatabase
│   └── persist + rebuild indices
│
├── memorize_vr_session(raw_data, device_type)
│   ├── get_adapter(device_type)  → DeviceAdapter
│   ├── adapter.parse_session()   → VRSession
│   ├── adapter.extract_insights() → VRInsights
│   ├── adapter.to_conversation() → list[str]
│   ├── memorize()                → reuse text pipeline
│   └── persist VRInsights
│
├── search(query)
│   ├── VectorIndex.search()      → cosine similarity on embeddings (RAG)
│   ├── MemoryIndex.search()      → TF-IDF keyword matching
│   ├── _merge_search_results()   → dedup + merge
│   └── LLM rerank (optional)
│
├── StorageBackend (ABC)
│   └── FileStorageBackend        → JSON files per user
├── EmbeddingProvider (RAG)       → SentenceTransformer / OpenAI / custom
└── DeviceAdapter (ABC)           → MetaQuestProAdapter / custom

Development

# Run all unit tests (no LLM needed)
pytest tests/test_storage.py tests/test_search.py tests/test_memorize.py tests/test_vector_index.py tests/test_vr_processor.py tests/test_vr_memorize.py

# Run all tests (including VR tests)
pytest

# Run with coverage
pytest --cov=generic_llm_memorizer

# Run integration tests (requires Ollama with gemma4:latest)
pytest tests/test_with_agent.py -v

Dependencies

• Python 3.12+
• pydantic
• scikit-learn
• sentence-transformers (optional, for RAG)