Retrieving the most relevant context for your LLMs
Project description
Quick start
Installation
pip install embedstore
Get API Key
Get an API key here
# Set the API Key
import os
os.environ["EMBEDSTORE_API_KEY"] = <YOUR API KEY>
Retrieve contexts
In a few lines of code, you can now start retrieving relevant contexts to get better answers from your LLM. We have already embedded Podcast Transcripts and Arxiv Research Papers, with new dataset drops every week (full list here)
from embedstore.rag.retrievers import EmbedStoreRetriever
# Initialize the retriever
arxiv_retriever = EmbedStoreRetriever(dataset_id = "arxiv_01", num_docs=3)
# Query the retriever
contexts = arxiv_retriever.query("What is the state of the art in Autonomous Driving security and safety?")
# Examine the retrieved contexts and then append it to the prompt before you call your LLM
print(contexts[0])
Read more here for detailed usage guidelines.
What is this?
TBD
Usage
More about EmbedStoreRetriever
class
EmbedStoreRetriever
serves as the single point of entrance to interact with the embedstore.
Parameters to initialize the class :
dataset_id (str)
: Dataset to query and retrieve contexts from (eg:arxiv_01
,podcasts_01
)num_docs (int)
: Number of contexts to retrieve from the embeddings
Advanced filtering while querying
You can use EmbedStoreRetriever.query()
to get contexts based on the user prompt and also perform filtering.
The query()
function takes two inputs:
prompt (str)
: The user prompt that you are querying forpost_processing_config (dict)
: A dictionary of dataset specific filters (check here for available options)
Example : Temporal filter on Arxiv research papers
from embedstore.rag.retrievers import EmbedStoreRetriever
# Initialize the retriever
arxiv_retriever = EmbedStoreRetriever(dataset_id = "arxiv_01", num_docs=3)
# Query the retriever
user_query = "What are some recent papers about CRISPR? Give me a brief summary of major trends."
# Filter to do semantic search on papers published in the last 6 months
post_processing_config = {"filters":{"publish_date_start":"2022-12-01T0:0:0Z"}}
# Calling the `query()` function
contexts = arxiv_retriever.query(user_query,post_processing_config)
Example : Categorical filter on Podcast Transcripts
# Initialize the retriever
podcast_retriever = EmbedStoreRetriever(dataset_id = "podcasts_01", num_docs=3)
# Query the retriever
user_query = "What are people saying about the new Twitter CEO?"
# Filter to search on Business and Technology podcast episodes
post_processing_config = {"filters":{"category":["Business","Technology"]}}
# Calling the `query()` function
contexts = podcast_retriever.query(user_query,post_processing_config)
Datasets
Dataset | dataset_id | Description | Status | Filters Available | More Details |
---|---|---|---|---|---|
Podcast Transcripts | podcasts_01 | 30 most recent podcasts across Finance, Business and Technology | Live | - category : ["Finance","Business","Technology"] |
Link |
arXiv | arxiv_01 | 2.2M arxiv papers | Live | - category : ['Computer Science', 'Quantitative Biology', 'Economics', 'Quantitative Finance', 'Statistics', 'Electrical Engineering and Systems Science', 'Mathematics', 'Physics']- publish_date_start : To filter papers published after this date (string in %Y-%m-%dT%H:%M:%SZ ) |
- |
Wikipedia | wikipedia_01 | - | Launching Soon | ||
News | news_01 | - | Launching Soon |
Get involved
We are early, and we want to know how you use this library and what else would you want to make the most out of it.
Join us on our discord, or feel free to email us at hello@embedding.store!
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for embedstore-0.0.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a4f83cee06eb9ad9d48d28280c429b2a7cabac16108c52c699179cfd6fd769a6 |
|
MD5 | edcfd2cad8d961369f05841416f09cde |
|
BLAKE2b-256 | bf76b8e059a72709d5757b6a999148a06baf47690daa4578cc7affcbdd4033a7 |