easyfaiss 0.1.3

No project description provided

Easy Faiss

Easy Faiss is a package for creating vector indexes / knowledge base for LLM chatbots. This provides an easy-to-use alternative over using vector databases that can come up a high cost.

Example script:

from easyfaiss.vectorembeddings import bert_setup, create_embeddings
from easyfaiss.vectorspace import FlatIndex, HNSWFlatIndex
import time

# Set up BERT model and tokenizer
model, tokenizer = bert_setup()

# Sample data
data = [
    "Hello, how are you?",
    "What's the weather today?",
    "How does photosynthesis work?",
    "What's the capital of France?",
    "Tell me about the history of the Roman Empire.",
    "How can I improve my coding skills?",
    "What's the recipe for a classic margarita?",
    "Explain the theory of relativity.",
    "What's the population of Tokyo?",
    "How can I learn a new language quickly"
]

# Create embeddings for the sample data
embeddings = create_embeddings(model=model, tokenizer=tokenizer, dataset=data)

# Define a user query
user_query = "What's the population in Tokyo, Japan?"

# Create embeddings for the user query
query_vector = create_embeddings(model=model, tokenizer=tokenizer, dataset=[user_query])

# Create a FlatIndex and update it with embeddings
flat_index = FlatIndex(name='flat_index')
flat_index.create_index(dimensions=768)
flat_index.update_index(embeddings=embeddings, dataset=data)

# Print details of the index
print("FlatIndex Details:")
print(flat_index.details())

# Perform similarity search on FlatIndex
print("\nFlatIndex Similarity Search:")
start_time = time.time()
indices, distances = flat_index.similarity_search(query_vector=query_vector, k=3)
end_time = time.time()
print("Time taken for kNN search in FlatIndex: {:.4f} seconds".format(end_time - start_time))

# Create an HNSWFlatIndex and update it with embeddings
hnsw_index = HNSWFlatIndex(name='hnsw_index')
hnsw_index.create_index(dimensions=768)
hnsw_index.update_index(embeddings=embeddings, dataset=data)

# Print details of the HNSWFlatIndex
print("\nHNSWFlatIndex Details:")
print(hnsw_index.details())

# Perform similarity search on HNSWFlatIndex
print("\nHNSWFlatIndex Similarity Search (Approximate Nearest Neighbors - ANN):")
start_time = time.time()
indices, distances = hnsw_index.similarity_search(query_vector=query_vector, k=3)
end_time = time.time()
print("Time taken for ANN search in HNSWFlatIndex: {:.4f} seconds".format(end_time - start_time))

EasyFaiss currently provides 2 types of faiss indexes

FlatIndex:

• Speed: This index is relatively slower for nearest neighbor search, especially as the number of vectors increases.
• Scale: Suitable for smaller datasets with low to moderate memory requirements.
• Accuracy: Provides exact nearest neighbor search, making it the most accurate option.

HNSWFlatIndex:

• Speed: Hierarchical Navigable Small World (HNSW) allows for fast approximate nearest neighbor search, especially on large datasets.
• Scale: Well-suited for large datasets, as it uses approximate nearest neighbor search, which makes it more memory-efficient.
• Accuracy: Provides approximate results that might have a small trade-off in accuracy compared to FlatIndex.

The choice between these two index types depends on your specific use case:

• Use FlatIndex when you have a relatively small dataset and need precise, exact nearest neighbor search.
• Use HNSWFlatIndex when you have a larger dataset, and you can tolerate a small reduction in accuracy in exchange for faster search and lower memory requirements.

For FlatIndex, an upper limit for the number of vectors depends on the available memory, but it's typically suitable for a few thousand vectors (e.g., up to 10,000 or more, depending on the dimension of the vectors and available RAM). Beyond that, you might start experiencing memory limitations. HNSWFlatIndex is a good choice for larger datasets, including millions of vectors, provided you can accept the trade-offs in accuracy.