A modular text-based database manager for retrieval-augmented generation (RAG), seamlessly integrating with the LoLLMs ecosystem.
Project description
LoLLMsVectorDB
LoLLMsVectorDB: A modular text-based database manager for retrieval-augmented generation (RAG), seamlessly integrating with the LoLLMs ecosystem. Supports various vectorization methods and directory bindings for efficient text data management.
Features
- Flexible Vectorization: Supports multiple vectorization methods including TF-IDF and Word2Vec.
- Directory Binding: Automatically updates the vector store with text data from a specified directory.
- Efficient Search: Provides fast and accurate search results with metadata to locate the original text chunks.
- Modular Design: Easily extendable to support new vectorization methods and functionalities.
Installation
pip install lollmsvectordb
Usage
Example with TFIDFVectorizer
from lollmsvectordb import TFIDFVectorizer, VectorDatabase, DirectoryBinding
# Initialize the vectorizer
tfidf_vectorizer = TFIDFVectorizer()
tfidf_vectorizer.fit(["This is a sample text.", "Another sample text."])
# Create the vector database
db = VectorDatabase("vector_db.sqlite", tfidf_vectorizer)
# Bind a directory to the vector database
directory_binding = DirectoryBinding("path_to_your_directory", db)
# Update the vector store with text data from the directory
directory_binding.update_vector_store()
# Search for a query in the vector database
results = directory_binding.search("This is a sample text.")
print(results)
Adding New Vectorization Methods
To add a new vectorization method, create a subclass of the Vectorizer
class and implement the vectorize
method.
from lollmsvectordb import Vectorizer
class CustomVectorizer(Vectorizer):
def vectorize(self, data):
# Implement your custom vectorization logic here
pass
Contributing
Contributions are welcome! Please fork the repository and submit a pull request.
License
This project is licensed under the MIT License.
Contact
For any questions or suggestions, feel free to reach out to the author:
- Twitter: @ParisNeo_AI
- Discord: Join our Discord
- Sub-Reddit: r/lollms
- Instagram: spacenerduino
Acknowledgements
Special thanks to the LoLLMs community for their continuous support and contributions.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for lollmsvectordb-1.0.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9711fbcd6bb7c42a5a4fe3f549b52a5b559783d82ddcfdc5758549b5af0e6bfc |
|
MD5 | 780aa36adac2cc9ee2df0e09e7c662d7 |
|
BLAKE2b-256 | dd032505b345f43a4b28ab7191c268d6c3da512b9431a03561434824c2214cab |