embenx 1.5.0

The Agentic Memory Layer & Universal Retrieval Toolkit. Synthetic data generation, 15+ vector backends, hybrid search, and MCP native memory for AI agents.

Embenx 🚀

The Agentic Memory Layer & Universal Retrieval Toolkit.
Synthetic data generation, 15+ vector backends, hybrid search, and MCP native memory for AI agents.

Documentation · Report Bug · Request Feature

---

What is Embenx?

Embenx is a Python-native retrieval library that sits between raw vector indices and full-blown vector databases. It provides a high-level Collection API for managing embeddings and metadata, supporting advanced features like filtering, reranking, and quantization across 15+ backends.

Library Usage

from embenx import Collection

# 1. Initialize a collection
col = Collection(dimension=768, indexer_type="faiss-hnsw")

# 2. Add data
# Supports incremental O(1) additions and bulk 'add_batch'
col.add(
    vectors=[[0.1, 0.2, ...], [0.3, 0.4, ...]],
    metadata=[{"category": "AI", "id": 1, "text": "The quick brown fox"}]
)

# Bulk batch ingestion with progress bar
col.add_batch(large_vectors, batch_size=1000, show_progress=True)

# 3. Search with filtering
results = col.search(
    query=[0.1, 0.2, ...],
    top_k=5,
    where={"category": "AI"}
)

# 4. Export to production
col.export_to_production(backend="qdrant", connection_url="http://localhost:6333")

# 5. Generate synthetic data (training/eval)
# Supports LiteLLM (v1.83.0+) and local Ollama
pairs = col.generate_synthetic_queries(
    n_queries_per_doc=2,
    output_path="synthetic_data.jsonl"
)

Agentic Memory (MCP)

Embenx ships with a built-in Model Context Protocol (MCP) server. This allows AI agents (like Claude Desktop) to use Embenx collections as their own long-term memory.

1. Start the server

embenx mcp-start

Visual Explorer

Embenx provides a built-in web UI to visualize your vector collections, including an interactive HNSW Graph Visualizer and a RAG Playground.

embenx explorer

Synthetic Data Generation 🧪

Generate high-quality query-document pairs to train or evaluate your retrieval pipelines. Embenx supports LiteLLM (for 100+ providers like OpenAI, Anthropic, Gemini) and local Ollama models.

from embenx import Collection

col = Collection.load("my_collection")

# Generate 2 synthetic queries for each of the first 100 documents
results = col.generate_synthetic_queries(
    n_queries_per_doc=2,
    num_docs=100,
    model="gpt-4o-mini",  # Or "ollama/llama3"
    output_path="eval_data.jsonl"
)

Features

• Synthetic Data Generation — Create high-quality query-document pairs using LLMs for training and evaluation.
• Multimodal Support — Native support for image embeddings (CLIP).
• RAG Playground — Test retrieval quality with an integrated LLM chat loop.
• HNSW Graph Visualizer — Interactive 3D visualization of navigation layers.
• Export to Production — One-click migration to Qdrant or Milvus clusters.
• Unified Collection API — Table-like interface for vectors and metadata.
• Retrieval Zoo — Instant access to pre-indexed collections (SQuAD, MS-MARCO, etc.).
• Agentic Memory (MCP) — Native Model Context Protocol support for AI agents.
• Self-Healing Retrieval — Integrated feedback loops to automatically improve ranking accuracy.
• Temporal Memory (Echo) — Recency-biased retrieval and time-window filtering (arXiv:2502.16090).
• Spatial Memory (ESWM) — Neuroscience-inspired spatial cognitive maps for navigation (ICLR 2026).
• TurboQuant Compression — 1-bit sign-based quantization for activation tensors (arXiv:2504.19874).
• ClusterKV Optimization — Semantic clustering for high-throughput retrieval (arXiv:2412.03213).
• Hybrid Search — Combine dense vectors with sparse BM25 retrieval using RRF.
• KV Cache Offloading (RA-KVC) — Store and retrieve high-dimensional LLM activations using safetensors.
• SSM State Hydration — Persist and prime hidden states ($h_0$) for State Space Models (Mamba-2).
• Trajectory Retrieval — Search for similar state/action sequences for World Models.
• Visual Explorer — Built-in web UI to visualize vector clusters and metadata.
• Universal model support — Integrated LiteLLM for any embedding provider.
• Portable Formats — Native support for Parquet, NumPy (.npy/.npz), and FAISS (.index).

Supported Indexers

Indexer	Family	Best For
faiss	HNSW, IVF, Flat	Production-grade local search
scann	Tree-AH	State-of-the-art speed/recall (Linux)
usearch	HNSW	High-performance C++, low latency
pgvector	Postgres	Embeddings next to relational data
lancedb	Columnar	Large disk-based datasets
simple	NumPy	Exact search baseline

Installation

pip install embenx

Roadmap

See ROADMAP.md for our journey towards production-grade agentic retrieval.

License

Distributed under the MIT License.

---

Built with ❤️ for the AI engineering community by adityak74