lexigram-vector

Vector storage infrastructure for the Lexigram Framework with Qdrant, ChromaDB, PGVector, Pinecone, and in-memory backends. Provides embedding clients, vector search, hybrid retrieval, reranking, and Named DI multi-store support.

Overview

Vector storage infrastructure for the Lexigram Framework with Qdrant, ChromaDB, PGVector, Pinecone, and in-memory backends. Provides embedding clients, vector search, hybrid retrieval, reranking, and Named DI multi-store support.

Architecture note: This package provides infrastructure and data-layer functionality for vector storage and retrieval. While it is commonly used by AI and RAG features (lexigram-ai-rag), it is a general-purpose vector database abstraction suitable for any use case requiring semantic search, similarity matching, or high-dimensional data storage.

Full documentation: docs.lexigram.dev

Deep-dive guide: Vector Stores in lexigram-vector

Install

uv add lexigram lexigram-vector

# With Qdrant support
uv add "lexigram-vector[qdrant]"

# With ChromaDB support
uv add "lexigram-vector[chroma]"

# With PGVector support (requires lexigram-sql for database access)
uv add "lexigram-vector[pgvector]"

# With Pinecone support
uv add "lexigram-vector[pinecone]"

# With embedding support
uv add openai  # or anthropic, cohere, etc.

Quick Start

import asyncio

from lexigram import Application
from lexigram.contracts.data.vector import (
    CollectionConfig,
    SearchQuery,
    VectorRecord,
    VectorStoreProtocol,
)
from lexigram.vector import VectorModule


async def main() -> None:
    async with Application.boot(modules=[VectorModule.stub()]) as app:
        store = await app.container.resolve(VectorStoreProtocol)

        # Create a collection
        await store.create_collection(
            CollectionConfig(name="documents", dimension=1536)
        )

        # Upsert vectors through the collection handle
        collection = await store.get_collection("documents")
        await collection.upsert(
            [
                VectorRecord(
                    id="doc1",
                    vector=[0.1] * 1536,
                    metadata={"title": "Document 1", "category": "tech"},
                ),
                VectorRecord(
                    id="doc2",
                    vector=[0.2] * 1536,
                    metadata={"title": "Document 2", "category": "science"},
                ),
            ]
        )

        # Search
        results = await collection.search(SearchQuery(vector=[0.15] * 1536, top_k=5))

        for result in results:
            print(f"{result.id}: {result.metadata['title']} (score: {result.score})")


if __name__ == "__main__":
    asyncio.run(main())

Configuration

Zero-config usage: Call VectorModule.configure() with no arguments to start with all built-in defaults — no config file or environment variables needed. See the Config reference below for all default values.

Option 1 — YAML file (use when config lives in a single explicit file)

Declare config in a YAML file loaded at a fixed, explicit path. LEX_* environment variables override YAML values at startup.

config_section = "vector" is already set on this class — section= can be omitted in all calls. Pass an explicit section= only to override the default (e.g. when this config is nested under a non-standard key).

# application.yaml — copy example.yaml for a fully-annotated starting point
vector:
  backend: "qdrant"             # memory, pgvector, pinecone, qdrant, chroma
  default_dimension: 1536       # 1536 = OpenAI text-embedding-3-small
  upsert_batch_size: 100
  qdrant:
    url: "http://localhost:6333"
    api_key: null               # LEX_VECTOR__QDRANT__API_KEY

Then load and wire it in your composition root:

from lexigram.vector.config import VectorConfig
from lexigram.vector import VectorModule

config = VectorConfig.from_yaml("application.yaml")
app.add_module(VectorModule.configure(config))

Environment variables override YAML values and use the LEX_VECTOR__ prefix:

LEX_VECTOR__BACKEND=qdrant

Option 2 — Profiles + Environment Variables (recommended for production, staging, Docker, CI/CD)

Loads a base application.yaml, then overlays an environment-specific file (application.production.yaml, application.staging.yaml, etc.) based on the LEX_PROFILE environment variable. LEX_* env vars are applied last as the final override layer.

# Set LEX_VECTOR__* env vars before starting the process
export LEX_VECTOR__ENABLED=true

from lexigram.vector.config import VectorConfig
from lexigram.vector import VectorModule

config = VectorConfig.from_env_profile()
app.add_module(VectorModule.configure(config))

Loading order: application.yaml (base) → application.{profile}.yaml (overlay, if LEX_PROFILE is set) → LEX_* environment variables (final override). Missing files are silently skipped so this is safe to call in all environments.

Option 3 — Python (use when config is dynamic or computed at boot)

Build config in code at boot time. Use this when settings are derived at runtime — e.g. secrets fetched from a vault, per-tenant configurations, or when you need multiple module instances with different settings.

from lexigram.vector import VectorModule
from lexigram.vector.config import QdrantConfig, VectorConfig

app.add_module(
    VectorModule.configure(
        VectorConfig(
            backend="qdrant",
            qdrant=QdrantConfig(
                url="http://localhost:6333",
            ),
        )
    )
)

Config reference

VectorConfig

Top-level configuration loaded from application.yaml's vector: key or from LEX_VECTOR__* environment variables.

When backends is non-empty, each entry is registered under Annotated[VectorStoreProtocol, Named(entry.name)]. The first entry (or the one with primary=True) also receives the unnamed VectorStoreProtocol binding for backward compatibility.

Backend-Specific Configuration

QdrantConfig

PgVectorConfig

Note: PGVector requires lexigram-sql and a configured DatabaseProviderProtocol. The database field refers to a named database backend from db.backends.

PineconeConfig

MemoryConfig

NamedVectorConfig

Configuration for a single named vector store backend (used in multi-store setups):

Example multi-store setup:

from lexigram.vector.config import (
    NamedVectorConfig,
    PgVectorConfig,
    QdrantConfig,
    VectorConfig,
)

VectorModule.configure(
    VectorConfig(
        backends=[
            NamedVectorConfig(
                name="primary",
                primary=True,
                backend="qdrant",
                qdrant=QdrantConfig(
                    url="http://qdrant-primary:6333",
                ),
            ),
            NamedVectorConfig(
                name="rag",
                backend="pgvector",
                pgvector=PgVectorConfig(
                    database="rag",
                    schema="vectors",
                ),
            ),
        ]
    )
)

Inject named stores:

from typing import Annotated
from lexigram.contracts.data.vector.protocols import VectorStoreProtocol
from lexigram.di.markers import Named


class MyService:
    def __init__(
        self,
        store: VectorStoreProtocol,  # primary
        rag: Annotated[VectorStoreProtocol, Named("rag")],
    ) -> None:
        self.store = store
        self.rag = rag

Module Factory Methods

Key Features

• Multi-backend vector storage — Qdrant, ChromaDB, PGVector, Pinecone, and in-memory backends
• Embedding client — OpenAI-compatible async client for generating embeddings (OpenAICompatibleEmbeddingClient)
• Embedding cache — In-memory and persistent caching to reduce embedding API calls (EmbeddingCache, InMemoryEmbeddingCache)
• Vector search — Similarity search with metadata filtering and distance metrics
• Hybrid retrieval — BM25 + vector search with reciprocal rank fusion (HybridRetriever, BM25Retriever, RRFReranker)
• Reranking — Cross-encoder reranking, diversity reranking, and similarity reranking for improved relevance (CrossEncoderReranker, DiversityReranker, RerankerPipeline)
• Metadata filtering — Structured filtering on metadata fields with backend-specific filter compilers
• Named DI multi-store — Multiple vector stores registered as Annotated[VectorStoreProtocol, Named("rag")]
• Collection management — Create, delete, list collections with automatic schema inference
• Batch operations — Efficient batch upsert, delete, and search with configurable batch sizes
• Distance metrics — Cosine, Euclidean, and dot product similarity metrics
• Index types — HNSW, IVFFlat, and backend-specific index configuration

Testing

async with Application.boot(modules=[VectorModule.stub()]) as app:
    store = await app.container.resolve(VectorStoreProtocol)
    # Test with the in-memory backend
    ...

Key Source Files