VLite Storage

VLite Storage is a Python library that provides vector similarity search capabilities using SQLite and the vectorlite packages. It allows you to store documents with their vector embeddings and perform efficient similarity searches. For creating text embeddings, it integrates with the Ollama server.

Features

• Document storage with vector embeddings
• Efficient similarity search using HNSW algorithm
• Integration with Ollama for text embeddings
• Support for document metadata
• Simple and intuitive API

Installation

pip install vlite-storage

Prerequisites

• Python 3.7+
• Ollama server running locally (default: http://localhost:11434)
• VectorLite SQLite extension

Quick Start

Here's a simple example of how to use VLite Storage:

from vlite_storage.embedders import OllamaEmbedder
from vlite_storage.storages import Storage

# Initialize embedder and storage
embedder = OllamaEmbedder()
dim = embedder.dimensions()
storage = Storage(db_name="my_database.db", dim=dim, embedding_fn=embedder)

# Add documents
storage.add(
    content="This is a sample document",
    metadata={"source": "example", "category": "sample"}
)

# Search for similar documents
results = storage.search("sample document", k=5)
for doc, distance in results:
    print(f"Content: {doc.content}")
    print(f"Metadata: {doc.metadata}")
    print(f"Distance: {distance}")

# Close the connection
storage.close()

API Reference

Storage Class

The main class for document storage and retrieval.

Storage(db_name: str, dim: int, embedding_fn: Optional[Callable[[str], np.ndarray]] = None)

Methods:

• add(content: str, metadata: dict): Add a document with content and metadata
• remove(rowid: int): Remove a document by ID
• update(rowid: int, content: Optional[str], metadata: Optional[dict]): Update document content and/or metadata
• get(rowid: int) -> Document: Retrieve a document by ID
• search(text: str, k: int) -> List[Tuple[Document, float]]: Find k most similar documents
• close(): Close the database connection

OllamaEmbedder Class

Class for generating text embeddings using Ollama models.

OllamaEmbedder(base_url: str = "http://localhost:11434", model_name: str = "bge-m3:latest")

Methods:

• dimensions() -> int: Get embedding dimensions
• __call__(texts: List[str]) -> np.ndarray: Generate embeddings for texts

Example Usage

Check the examples/ directory for more detailed examples, including:

• Text chunk processing and storage
• Similarity search in large documents
• Metadata handling

License

This project is licensed under the MIT License - see the LICENSE file for details.