Skip to main content

AgentSearch: An open source framework and dataset for webscale local search.

Project description

AgentSearch: A framework for powering search agents and enabling customizable local search.

AgentSearch Banner

AgentSearch is a framework for powering search agents by seamlessly integrating LLM technologies from various providers with different search engines. This integration enables search agents to perform a wide range of functions through Retrieval-Augmented Generation (RAG), including summarizing search results, generating new queries, and retrieving detailed downstream results.

Features of AgentSearch

  • Search Agent Integration: Effortlessly build a search agent by connecting any search-specialized LLM, such as Sensei-7B, with a supported search engine.
  • Customizable Search: Utilize the AgentSearch dataset in conjunction with this framework to deploy a customizable local search engine.
  • API Endpoint Integration: Seamlessly integrate with a variety of hosted provider APIs for diverse search solutions, offering ease of use and flexibility, including Bing, SERP API, and AgentSearch. Additionally, support is provided for LLMs from SciPhi, HuggingFace, OpenAI, Anthropic, and more.

Quickstart Guide

Installation

pip install agent-search

Configuration

Get your free API key from SciPhi and set it in your environment:

export SCIPHI_API_KEY=$MY_SCIPHI_API_KEY

Usage

Call a pre-configured search agent endpoint:

# Requires SCIPHI_API_KEY in the environment
from agent_search import SciPhi

client = SciPhi()

# Search, then summarize result and generate related queries
agent_summary = client.get_search_rag_response(query='latest news', search_provider='bing', llm_model='SciPhi/Sensei-7B-V1')
print(agent_summary)
# { 'response': '...', 'other_queries': '...', 'search_results': '...' }

Standalone searches and from the AgentSearch search engine are supported:

# Requires SCIPHI_API_KEY in the environment
from agent_search import SciPhi

client = SciPhi()

# Perform a search
search_response = client.search(query='Quantum Field Theory', search_provider='agent-search')

print(search_response)
# [{ 'score': '.89', 'url': 'https://...', 'metadata': {...} }]

Code your own custom search agent workflow:

# Requires SCIPHI_API_KEY in the environment
from agent_search import SciPhi

client = SciPhi()

# Specify instructions for the task
instruction = "Your task is to perform retrieval augmented generation (RAG) over the given query and search results. Return your answer in a json format that includes a summary of the search results and a list of related queries."
query = "What is Fermat's Last Theorem?"

# construct search context
search_response = client.search(query=query, search_provider='agent-search')
search_context = "\n\n".join(
      f"{idx + 1}. Title: {item['title']}\nURL: {item['url']}\nText: {item['text']}"
      for idx, item in enumerate(search_response)
).encode('utf-8')

# Prefix to enforce a JSON response 
json_response_prefix = '{"summary":'

# Prepare a prompt
formatted_prompt = f"### Instruction:{instruction}\n\nQuery:\n{query}\n\nSearch Results:\n${search_context}\n\nQuery:\n{query}\n### Response:\n{json_response_prefix}",

# Generate a completion with Sensei-7B-V1
completion = json_response_prefix + client.completion(formatted_prompt, llm_model_name="SciPhi/Sensei-7B-V1")

print(completion)
# {
#   "summary":  "\nFermat's Last Theorem is a mathematical proposition first prop ... ",
#   "other_queries": ["The role of elliptic curves in the proof of Fermat's Last Theorem", ...]
# }

Community & Support

  • Engage with Us: Join our Discord community for discussions and updates.
  • Feedback & Inquiries: Contact us via email for personalized support.

Additional Notes

  • Execute commands from the root directory of the AgentSearch project.
  • User Guide coming soon!

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

agent_search-0.1.0.tar.gz (18.2 kB view details)

Uploaded Source

Built Distribution

agent_search-0.1.0-py3-none-any.whl (21.1 kB view details)

Uploaded Python 3

File details

Details for the file agent_search-0.1.0.tar.gz.

File metadata

  • Download URL: agent_search-0.1.0.tar.gz
  • Upload date:
  • Size: 18.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.10.8

File hashes

Hashes for agent_search-0.1.0.tar.gz
Algorithm Hash digest
SHA256 4f94c15b878088f06fc9e8b93356c6c25a63b977821ad9eb2490332d038499cc
MD5 98e5273f8a2f673a4e0b73c916a5d6ad
BLAKE2b-256 f6941996dfd0ffd2a916b3cce3b8273cdd45b92e2c673cdc3d9b13b957e15f14

See more details on using hashes here.

File details

Details for the file agent_search-0.1.0-py3-none-any.whl.

File metadata

  • Download URL: agent_search-0.1.0-py3-none-any.whl
  • Upload date:
  • Size: 21.1 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.10.8

File hashes

Hashes for agent_search-0.1.0-py3-none-any.whl
Algorithm Hash digest
SHA256 833ed6a56aedb506ef7408efafdcb42cae1614af87c3906c65ed0bf29fb46b1b
MD5 5ba225917977823640be7a4890b54840
BLAKE2b-256 74b25470d8c34e99560f832b17ea98acb972c784d30f820ef441c0a275a53c5d

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page