search-engine-tool-mcp 0.4.3

A Python MCP Server for web search and extraction with Jina and Tavily providers

Search Engine Tool MCP

A Python MCP (Model Context Protocol) Server for web search and extraction with multiple providers.

Version

Current version: 0.4.3

Features

• 🔍 Web Search: Search the web using You.com or Tavily APIs
• 📄 Content Extraction: Extract clean content from any URL (free local extraction or Tavily API)
• 🔄 Auto Provider Selection: Automatically chooses the best available provider
• 🔑 Optional API Keys: Works without API keys using local extraction (Tavily requires API key for enhanced extraction)
• 🚀 MCP Compatible: Full support for Model Context Protocol

Installation

From PyPI (Recommended)

✅ Latest version 0.4.3 is now available on PyPI!

PyPI Package Page: https://pypi.org/project/search-engine-tool-mcp/0.4.3/

# Using pip
pip install search-engine-tool-mcp==0.4.3

# Or using uv (recommended)
uv pip install search-engine-tool-mcp==0.4.3

# Or using uvx (no installation required)
uvx --from search-engine-tool-mcp==0.4.3 search-engine-tool-mcp

From Source

git clone https://github.com/YiHarvest/search-engine-tool.git
cd search-engine-tool

# Using uv (recommended)
uv sync

# Or using pip
pip install -e .

Configuration

Environment Variables

• SEARXNG_BASE_URL (recommended): Your SearXNG instance URL. Example: http://127.0.0.1:8080
• TALORDATA_API_KEY (recommended): Your TalorData SERP API key for web search. Get it at https://dashboard.talordata.com. Note: TalorData only supports web_search, not web_extract.
• YDC_API_KEY (recommended): Your You.com API key. Get it at https://you.com/platform.
• TAVILY_API_KEY (optional): Your Tavily API key for advanced features and fallback extraction.

Auto Provider Selection Logic:

For web_search:

• If TALORDATA_API_KEY exists → uses TalorData (SERP API provider)
  • If TalorData fails or returns empty results → falls back to Tavily or You.com
• Otherwise, if SEARXNG_BASE_URL exists → uses SearXNG (free, self-hosted)
  • If SearXNG fails or returns empty results → falls back to TalorData, Tavily, or You.com
• Otherwise, if YDC_API_KEY or YOU_API_KEY exists → uses You.com
• Otherwise, if TAVILY_API_KEY exists → uses Tavily
• If none configured → throws error

For web_extract:

• Default provider is local (free, no API key required)
• If provider="auto" and local extraction fails:
  • Falls back to Tavily if TAVILY_API_KEY is available
  • Otherwise returns error
• If provider="tavily" → requires TAVILY_API_KEY
• Note: TalorData does NOT support web_extract

Quick Configuration

1. Copy the example configuration file:

cp .env.example .env

2. Edit the .env file and fill in your API keys:

YDC_API_KEY="your-ydc-api-key-here"
TAVILY_API_KEY="your-tavily-api-key-here"
SEARXNG_BASE_URL="http://127.0.0.1:8080"
TALORDATA_API_KEY="your-talordata-api-key-here"

Or Set Environment Variables Manually

# Set SearXNG base URL (recommended for free self-hosted search)
export SEARXNG_BASE_URL="http://127.0.0.1:8080"

# Set TalorData API key (recommended for SERP API search)
export TALORDATA_API_KEY="your-talordata-api-key"

# Set You.com API key (alternative)
export YDC_API_KEY="ydc-sk-your-api-key"

# Or set Tavily API key (alternative)
export TAVILY_API_KEY="tvly-your-api-key"

Run SearXNG Locally

SEARXNG_BASE_URL is not an API key. It is the URL of a SearXNG instance that you run. This MCP server uses SearXNG's JSON search endpoint, so json must be enabled in settings.yml.

1. Create the configuration directory and searxng/settings.yml:

mkdir -p searxng

# searxng/settings.yml
use_default_settings: true

search:
  formats:
    - html
    - json

server:
  # Replace this even for local use: openssl rand -hex 32
  secret_key: "replace-with-a-random-secret"
  limiter: false
  image_proxy: true

2. Start a SearXNG container bound only to localhost:

docker run -d \
  --name searxng \
  --restart unless-stopped \
  -p 127.0.0.1:8080:8080 \
  -v "$(pwd)/searxng:/etc/searxng:rw" \
  docker.io/searxng/searxng:latest

3. Verify the JSON endpoint:

curl "http://127.0.0.1:8080/search?q=hello&format=json"

After it returns JSON containing results, configure:

SEARXNG_BASE_URL="http://127.0.0.1:8080"

Use docker stop searxng and docker start searxng to stop and restart it. For production or public deployment, add a reverse proxy, HTTPS, access controls, and a random secret_key; see the official SearXNG container documentation.

MCP Client Configuration

Credentials must be passed to the MCP server process as environment variables. All values below are placeholders. Keep only the providers you use, and never commit real credentials to Git.

Trae (JSON)

Add or import the following configuration in Trae's MCP settings:

{
  "mcpServers": {
    "search-engine-tool": {
      "command": "uvx",
      "args": [
        "--from",
        "search-engine-tool-mcp==0.4.3",
        "search-engine-tool-mcp"
      ],
      "env": {
        "TALORDATA_API_KEY": "your-talordata-api-key",
        "TALORDATA_BASE_URL": "https://serpapi.talordata.net/serp/v1/request",
        "SEARXNG_BASE_URL": "http://127.0.0.1:8080",
        "YDC_API_KEY": "your-you-api-key",
        "TAVILY_API_KEY": "tvly-your-tavily-api-key"
      }
    }
  }
}

Restart or refresh the MCP server after saving. If Trae and SearXNG run on different hosts or container networks, 127.0.0.1 refers to the environment running Trae/the MCP process; replace it with a SearXNG URL reachable from that environment.

Codex (TOML)

Codex CLI and the Codex IDE extension share config.toml. Add this to the user configuration at ~/.codex/config.toml, or to .codex/config.toml in a trusted project:

[mcp_servers.search-engine-tool]
command = "uvx"
args = [
  "--from",
  "search-engine-tool-mcp==0.4.3",
  "search-engine-tool-mcp",
]
startup_timeout_sec = 30
tool_timeout_sec = 60

[mcp_servers.search-engine-tool.env]
TALORDATA_API_KEY = "your-talordata-api-key"
TALORDATA_BASE_URL = "https://serpapi.talordata.net/serp/v1/request"
SEARXNG_BASE_URL = "http://127.0.0.1:8080"
YDC_API_KEY = "your-you-api-key"
TAVILY_API_KEY = "tvly-your-tavily-api-key"

Restart Codex, then run codex mcp list or use /mcp in a Codex session to verify the connection. See the official Codex MCP documentation for configuration options.

Usage

Available Tools

The MCP server exposes two tools:

1. web_search

Search the web for information.

Parameters:

Parameter	Type	Required	Default	Description
query	string	✅	-	Search query string
provider	string	❌	"auto"	Provider to use: "auto", "searxng", "talordata", "you", or "tavily"
max_results	integer	❌	5	Maximum number of results (1-20)
search_depth	string	❌	"basic"	Search depth: "basic" or "advanced" (Tavily only)
include_answer	boolean	❌	false	Include AI-generated answer (Tavily and TalorData only)

Response:

{
  "query": "search query",
  "provider": "jina",
  "count": 5,
  "results": [
    {
      "href": "https://example.com",
      "title": "Result Title",
      "abstract": "Result snippet..."
    }
  ]
}

2. web_extract

Extract content from a specific URL.

Parameters:

Parameter	Type	Required	Default	Description
url	string	✅	-	URL to extract content from
provider	string	❌	"auto"	Provider to use: "auto", "local", or "tavily"

Provider Options:

• local (default free provider): Free local extraction, no API key required. Uses trafilatura + BeautifulSoup for content extraction.
  • ⚠️ Limitations:
    • Does not execute JavaScript
    • Cannot bypass login walls, paywalls, or CAPTCHA
    • Dynamic/spa-style pages may have incomplete extraction
    • Timeout: 20 seconds (default)
• tavily (enhanced provider): Tavily API extraction, requires TAVILY_API_KEY.
  • ✅ Better for complex/dynamic pages
  • ✅ Can handle more challenging sites
• auto (default): Tries local first, falls back to Tavily if TAVILY_API_KEY is available and local fails.

Response:

{
  "url": "https://example.com",
  "content": "Extracted content...",
  "provider": "local"
}

Provider Selection Logic

• provider="searxng": Always use SearXNG (requires SEARXNG_BASE_URL)
  • ⚠️ Limitations:
    • Requires self-hosted SearXNG instance
    • Timeout: 15 seconds
    • If returns empty results → throws error
• provider="you": Always use You.com (requires YDC_API_KEY)
• provider="tavily": Always use Tavily (requires TAVILY_API_KEY)
• provider="auto"** (default):
  • Initial selection order: TalorData → SearXNG → You.com → Tavily
  • On failure or empty results, switches through the configured fallback chain for the current provider
  • If no provider is configured → throws an error

TalorData SERP API

TalorData is a SERP (Search Engine Results Page) API provider that offers structured search results from Google, Bing, and other search engines. Important: TalorData is ONLY for web_search, NOT for web_extract.

Features

• ✅ High quality: Top-ranked SERP API provider
• ✅ Multi-engine support: Google, Bing, DuckDuckGo, and more
• ✅ Structured output: Clean JSON format with detailed metadata
• ✅ AI Overview: Supports AI-generated answers (include_answer parameter)
• ✅ Optional fields: Returns position, source, display_link for advanced analysis
• ✅ Fast response: Average response time under 1 second
• ✅ Pay for success: Only charges for successful requests
• ⚠️ API key required: Requires TALORDATA_API_KEY
• ⚠️ Web search only: Does NOT support web_extract

Get API Key

1. Visit TalorData Dashboard
2. Sign up for a free account
3. Navigate to API Playground to get your API token
4. Set the environment variable:

export TALORDATA_API_KEY="your-api-key-here"

Usage Example

# Web search with TalorData
results = await web_search(
    query="OpenAI",
    provider="talordata",
    max_results=10,
    include_answer=True  # Get AI-generated overview
)

# Response includes:
# - href: URL of the result
# - title: Title of the result
# - abstract: Description or snippet
# - position: Position in search results (optional)
# - source: Source name (optional)
# - display_link: Displayed link (optional)
# - answer: AI-generated overview (if include_answer=True)

SearXNG Setup

SearXNG is a free, self-hosted metasearch engine that aggregates results from multiple search engines.

Installation

1. Using Docker (recommended):

docker run -d --name searxng -p 8080:8080 searxng/searxng:latest

2. Manual installation: Follow the SearXNG documentation

Configuration

After setting up SearXNG, set the environment variable:

export SEARXNG_BASE_URL="http://127.0.0.1:8080"

Features

• ✅ Free: No API key required
• ✅ Privacy-focused: No tracking
• ✅ Multiple engines: Aggregates results from Google, Bing, DuckDuckGo, etc.
• ✅ Customizable: Choose specific engines with engines parameter
• ⚠️ Self-hosted: Requires running your own instance
• ⚠️ Timeout: 15-second limit to prevent MCP hanging

Development

Setup Development Environment

# Clone the repository
git clone https://github.com/YiHarvest/search-engine-tool.git
cd search-engine-tool

# Using uv (recommended)
uv sync --all-extras

# Or using pip
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
pip install -e ".[dev]"

Run Tests

# Using uv
uv run pytest

# Or using pytest directly
pytest

Build Package

# Using uv (recommended)
uv build

# Or using build directly
python -m build

Check Package

# Using uv
uv run twine check dist/*

# Or using twine directly
twine check dist/*

API Reference

Jina Provider

• Search API: https://api.jina.ai/search
• Reader API: https://r.jina.ai/{url}
• Requirements: No API key needed
• Rate Limits: Subject to Jina's public API limits

Tavily Provider

• Search API: https://api.tavily.com/search
• Extract API: https://api.tavily.com/extract
• Requirements: API key required (TAVILY_API_KEY)
• Features: Advanced search depth, AI-generated answers

Project Structure

search-engine-tool/
├── src/search_engine_tool_mcp/
│   ├── __init__.py
│   ├── server.py          # MCP server implementation
│   ├── search.py          # Web search functionality
│   ├── extract.py         # Content extraction
│   ├── schemas.py         # Pydantic data models
│   └── providers/
│       ├── __init__.py
│       ├── you.py         # You.com provider (search)
│       ├── tavily.py      # Tavily provider (search + extract)
│       └── local_extract.py  # Local extraction provider (free)
├── tests/
│   ├── test_search.py
│   └── test_providers.py
│   └── test_extract.py
├── pyproject.toml
└── README.md

License

MIT License

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Support

For issues and feature requests, please use the GitHub Issues page.