Serpex Haystack Integration

Serpex integration for Haystack - bringing web search to your Haystack pipelines.

Overview

Serpex is the web search API and extract API for AI agents: ranked web results, optionally with page content as markdown. This integration allows you to seamlessly incorporate web search results into your Haystack RAG (Retrieval-Augmented Generation) pipelines and AI applications.

Key Features

• 🔍 Web Search: One search engine, nothing to configure; optional page content as markdown
• ⚡ Reliable: Automatic retries with exponential backoff
• 🎯 Rich Results: Get organic search results with titles, snippets, and URLs
• 🕒 Time Filters: Filter results by day, week, month, or year
• 🔒 Type-Safe: Fully typed with comprehensive type hints
• 📝 Haystack Native: Seamless integration with Haystack 2.0+ components

Installation

pip install serpex-haystack

Quick Start

Get Your API Key

Sign up at Serpex.dev to get your API key.

Basic Usage

from haystack import Pipeline
from haystack.components.builders import PromptBuilder
from haystack.components.generators import OpenAIGenerator
from haystack.utils import Secret
from haystack_integrations.components.websearch.serpex import SerpexWebSearch

# Create a web search component
web_search = SerpexWebSearch(
    api_key=Secret.from_env_var("SERPEX_API_KEY"),
)

# Use it standalone
results = web_search.run(query="What is Haystack AI?")
for doc in results["documents"]:
    print(f"Title: {doc.meta['title']}")
    print(f"URL: {doc.meta['url']}")
    print(f"Snippet: {doc.content}\n")

RAG Pipeline Example

from haystack import Pipeline
from haystack.components.builders import PromptBuilder
from haystack.components.generators import OpenAIGenerator
from haystack.utils import Secret
from haystack_integrations.components.websearch.serpex import SerpexWebSearch

# Create a simple RAG pipeline with web search
prompt_template = """
Based on the following search results, answer the question.

Search Results:
{% for doc in documents %}
- {{ doc.meta.title }}: {{ doc.content }}
  Source: {{ doc.meta.url }}
{% endfor %}

Question: {{ query }}

Answer:
"""

pipe = Pipeline()
pipe.add_component("search", SerpexWebSearch(api_key=Secret.from_env_var("SERPEX_API_KEY")))
pipe.add_component("prompt", PromptBuilder(template=prompt_template))
pipe.add_component("llm", OpenAIGenerator(api_key=Secret.from_env_var("OPENAI_API_KEY")))

pipe.connect("search.documents", "prompt.documents")
pipe.connect("prompt", "llm")

# Run the pipeline
result = pipe.run({
    "search": {"query": "Latest developments in AI agents"},
    "prompt": {"query": "Latest developments in AI agents"}
})

print(result["llm"]["replies"][0])

Advanced Features

Page content

# Each Document carries the page content (markdown) for the top 5 results;
# the snippet moves to meta["snippet"].
web_search = SerpexWebSearch(include_content=True, content_results=5)
results = web_search.run(query="AI news")

# Or per call
results = web_search.run(query="Python tutorials", include_content=True)

Best-effort: a page that can't be extracted keeps the snippet as content and carries meta["content_error"].

Deprecated parameters

Serpex is one search engine, so there is nothing to select. engine (in __init__ and run()) and time_range (in run()) are deprecated and ignored by the Serpex API. They are still accepted so existing code keeps running, emit a DeprecationWarning, and are not sent. Pipelines saved with an older version still load: their engine value is dropped. Removed in 2.0.

Timeouts and retries

web_search = SerpexWebSearch(
    api_key=Secret.from_env_var("SERPEX_API_KEY"),
    timeout=60.0,  # Request timeout in seconds
    retry_attempts=3  # Total attempts on a transport error, 429 or 5xx
)

The default timeout is 60 s (100 s with include_content), longer than the server's own budget, so a slow search isn't abandoned after the server has billed it. Only transport errors, 429 and 5xx are retried; 4xx errors (bad request, invalid key, no credits) fail immediately.

Component Reference

SerpexWebSearch

A Haystack component for fetching web search results via the Serpex API.

Parameters

• api_key (Secret, optional): Serpex API key. Defaults to SERPEX_API_KEY environment variable.
• timeout (float, optional): Request timeout in seconds. Defaults to 60, or 100 with include_content.
• retry_attempts (int, optional): Total attempts on a transport error, 429 or 5xx. Defaults to 2.
• include_content (bool, optional): Also fetch page content (markdown) for the top results. Defaults to False.
• content_results (int, optional): How many top results get content, 5 or 10. Defaults to 5.
• engine (str, optional): Deprecated — ignored by the Serpex API and not sent.

Inputs

• query (str): The search query string.
• include_content (bool, optional): Overrides the component setting for this call.
• engine, time_range (optional): Deprecated — ignored by the Serpex API and not sent.

Outputs

• documents (List[Document]): List of Haystack Document objects containing search results.

Each document includes:

• content: The page content (markdown) when include_content is on and the page was extracted; otherwise the snippet
• meta:
  • title: Result title
  • url: Result URL
  • position: Position in search results
  • query: Original search query
  • snippet, content_error: Only with include_content
  • engine: Deprecated, always "auto"; removed in 2.0

Examples

Check out the examples directory for more use cases:

• Basic Search
• RAG Pipeline
• Agent with Web Search

Why Serpex?

• 🌐 Real-Time Web Search: current results for any query, as structured JSON
• 📄 Page Content Extraction: turn URLs into LLM-ready markdown
• 🤖 Built for AI: made for agents, LLM tools and RAG pipelines

Documentation

• Serpex API Documentation
• Haystack Documentation
• Integration Examples

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

Development Setup

# Clone the repository
git clone https://github.com/divyeshradadiya/serpex-haystack.git
cd serpex-haystack

# Install with development dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Run linting
ruff check .
black --check .

# Run type checking
mypy src/

License

This project is licensed under the Apache 2.0 License - see the LICENSE file for details.

Support

• 📧 Email: support@serpex.dev
• 💬 Discord: Join our community
• 🐛 Issues: GitHub Issues
• 📖 Docs: docs.serpex.dev

Acknowledgments

Built with ❤️ for the Haystack community by Divyesh Radadiya

---

Note: This is a community-maintained integration. For Serpex API support, visit serpex.dev.