Skip to main content

A modular text-based database manager for retrieval-augmented generation (RAG), seamlessly integrating with the LoLLMs ecosystem.

Project description

LoLLMsVectorDB

LoLLMsVectorDB: A modular text-based database manager for retrieval-augmented generation (RAG), seamlessly integrating with the LoLLMs ecosystem. Supports various vectorization methods and directory bindings for efficient text data management.

Features

  • Flexible Vectorization: Supports multiple vectorization methods including TF-IDF and Word2Vec.
  • Directory Binding: Automatically updates the vector store with text data from a specified directory.
  • Efficient Search: Provides fast and accurate search results with metadata to locate the original text chunks.
  • Modular Design: Easily extendable to support new vectorization methods and functionalities.

Installation

pip install lollmsvectordb

Usage

Example with TFIDFVectorizer

from lollmsvectordb import TFIDFVectorizer, VectorDatabase, DirectoryBinding

# Initialize the vectorizer
tfidf_vectorizer = TFIDFVectorizer()
tfidf_vectorizer.fit(["This is a sample text.", "Another sample text."])

# Create the vector database
db = VectorDatabase("vector_db.sqlite", tfidf_vectorizer)

# Bind a directory to the vector database
directory_binding = DirectoryBinding("path_to_your_directory", db)

# Update the vector store with text data from the directory
directory_binding.update_vector_store()

# Search for a query in the vector database
results = directory_binding.search("This is a sample text.")
print(results)

Adding New Vectorization Methods

To add a new vectorization method, create a subclass of the Vectorizer class and implement the vectorize method.

from lollmsvectordb import Vectorizer

class CustomVectorizer(Vectorizer):
    def vectorize(self, data):
        # Implement your custom vectorization logic here
        pass

Contributing

Contributions are welcome! Please fork the repository and submit a pull request.

License

This project is licensed under the MIT License.

Contact

For any questions or suggestions, feel free to reach out to the author:

Acknowledgements

Special thanks to the LoLLMs community for their continuous support and contributions.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

lollmsvectordb-1.1.0.tar.gz (28.9 kB view hashes)

Uploaded Source

Built Distribution

lollmsvectordb-1.1.0-py3-none-any.whl (33.6 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page