Search Engine Tool MCP

A Python MCP (Model Context Protocol) Server for web search and extraction with multiple providers.

Version

Current version: 0.5.0

Features

• 🔍 Web Search: Zero-configuration search with DDGS, plus SearXNG, TalorData, You.com, and Tavily providers
• 📄 Content Extraction: Extract text from public HTTP(S) URLs with local extraction or Tavily
• 🔄 Fixed Auto Search: Uses SearXNG when configured, then DDGS
• 🔑 No Search Key Required: DDGS search and local extraction work without API keys
• 🔒 Safe Local Extraction: DNS-aware SSRF checks, per-hop redirect validation, and bounded responses
• 🚀 MCP Compatible: Full support for Model Context Protocol

Installation

From PyPI (Recommended)

# Using pip
pip install search-engine-tool-mcp

# Or using uv (recommended)
uv pip install search-engine-tool-mcp

# Or using uvx (no installation required)
uvx search-engine-tool-mcp@latest

From Source

git clone https://github.com/YiHarvest/search-engine-tool.git
cd search-engine-tool

# Using uv (recommended)
uv sync

# Or using pip
pip install -e .

Configuration

Environment Variables

• SEARXNG_BASE_URL (optional): Your SearXNG instance URL. Example: http://127.0.0.1:8080
• DDGS_REGION, DDGS_BACKEND, DDGS_SAFESEARCH, DDGS_TIMEOUT, and DDGS_PROXY (optional): Tune DDGS. DDGS requires no API key.
• SEARCH_TOTAL_TIMEOUT (optional, default 20): Maximum seconds for the complete web_search operation, including automatic fallback.
• EXTRACT_MAX_BYTES (optional, default 5000000): Maximum downloaded response size for local extraction.
• EXTRACT_MAX_CHARS (optional, default 100000): Maximum extracted character count.
• EXTRACT_MAX_REDIRECTS (optional, default 5): Maximum manually followed redirects; every hop is validated.
• TALORDATA_API_KEY (optional): Your TalorData SERP API key for explicit provider="talordata" searches. TalorData supports web_search, not web_extract.
• YDC_API_KEY (optional): Your You.com API key for explicit provider="you" searches.
• TAVILY_API_KEY (optional): Used for explicit Tavily search/extraction and automatic extraction fallback.

The server loads only .env from its current working directory. It does not search parent or home directories.

Provider Capabilities

Provider	Configuration	Search	Extract	Participates in web_search auto
DDGS	None	✅	❌	✅
SearXNG	SEARXNG_BASE_URL	✅	❌	First, when configured
TalorData	TALORDATA_API_KEY	✅	❌	❌
You.com	YDC_API_KEY	✅	❌	❌
Tavily	TAVILY_API_KEY	✅	✅	❌
Local	None	❌	✅	Not applicable

Fixed Auto Rules

For web_search:

• If SEARXNG_BASE_URL exists: SearXNG, then DDGS when SearXNG fails or returns no results.
• Without SEARXNG_BASE_URL: DDGS.
• TalorData, You.com, and Tavily never participate in auto. They require API keys and explicit provider selection, preventing accidental API usage.

For web_extract:

• Default provider is local (free, no API key required)
• If provider="auto" and local extraction fails:
  • Falls back to Tavily if TAVILY_API_KEY is available
  • Otherwise returns error
• provider="tavily" requires TAVILY_API_KEY

Quick Configuration

1. Copy the example configuration file:

cp .env.example .env

2. Optional: edit .env only when you want SearXNG, custom DDGS settings, or an explicitly selected paid provider:

# SEARXNG_BASE_URL="http://127.0.0.1:8080"
DDGS_REGION="us-en"
DDGS_BACKEND="auto"
DDGS_SAFESEARCH="moderate"
DDGS_TIMEOUT="10"
# DDGS_PROXY="socks5h://127.0.0.1:9150"
SEARCH_TOTAL_TIMEOUT="20"
EXTRACT_MAX_BYTES="5000000"
EXTRACT_MAX_CHARS="100000"
EXTRACT_MAX_REDIRECTS="5"

# Paid providers: uncomment only when explicitly needed
# TALORDATA_API_KEY="your-talordata-api-key-here"
# YDC_API_KEY="your-ydc-api-key-here"
# TAVILY_API_KEY="your-tavily-api-key-here"

Run SearXNG Locally

SEARXNG_BASE_URL is not an API key. It is the URL of a SearXNG instance that you run. This MCP server uses SearXNG's JSON search endpoint, so json must be enabled in settings.yml.

1. Create the configuration directory and searxng/settings.yml:

mkdir -p searxng

# searxng/settings.yml
use_default_settings: true

search:
  formats:
    - html
    - json

server:
  # Replace this even for local use: openssl rand -hex 32
  secret_key: "replace-with-a-random-secret"
  limiter: false
  image_proxy: true

2. Start a SearXNG container bound only to localhost:

docker run -d \
  --name searxng \
  --restart unless-stopped \
  -p 127.0.0.1:8080:8080 \
  -v "$(pwd)/searxng:/etc/searxng:rw" \
  docker.io/searxng/searxng:latest

3. Verify the JSON endpoint:

curl "http://127.0.0.1:8080/search?q=hello&format=json"

After it returns JSON containing results, configure:

SEARXNG_BASE_URL="http://127.0.0.1:8080"

Use docker stop searxng and docker start searxng to stop and restart it. For production or public deployment, add a reverse proxy, HTTPS, access controls, and a random secret_key; see the official SearXNG container documentation.

MCP Client Configuration

No environment variables are required for DDGS. Optional SearXNG settings and paid-provider credentials must be passed to the MCP server process. Never commit real credentials to Git.

JSON clients (Trae and similar)

Zero-configuration DDGS search:

{
  "mcpServers": {
    "search-engine-tool": {
      "command": "uvx",
      "args": [
        "search-engine-tool-mcp@latest"
      ]
    }
  }
}

To use SearXNG or a paid provider, add only the variables you actually use and replace every placeholder:

{
  "mcpServers": {
    "search-engine-tool": {
      "command": "uvx",
      "args": [
        "search-engine-tool-mcp@latest"
      ],
      "env": {
        "SEARXNG_BASE_URL": "http://127.0.0.1:8080",
        "TALORDATA_API_KEY": "<your-talordata-api-key>",
        "YDC_API_KEY": "<your-you-api-key>",
        "TAVILY_API_KEY": "<your-tavily-api-key>"
      }
    }
  }
}

Delete unused entries. In particular, do not leave a placeholder TAVILY_API_KEY: web_extract(provider="auto") treats any non-empty value as configured and may attempt Tavily fallback. Other variables from the environment-variable list can be passed through the same env object.

Restart or refresh the MCP server after saving. If Trae and SearXNG run on different hosts or container networks, 127.0.0.1 refers to the environment running Trae/the MCP process; replace it with a SearXNG URL reachable from that environment.

Codex (TOML)

Codex CLI and the Codex IDE extension share config.toml. Add this to the user configuration at ~/.codex/config.toml, or to .codex/config.toml in a trusted project:

[mcp_servers.search-engine-tool]
command = "uvx"
args = [
  "search-engine-tool-mcp@latest",
]
startup_timeout_sec = 30
tool_timeout_sec = 60

# Add only the entries you use.
[mcp_servers.search-engine-tool.env]
SEARXNG_BASE_URL = "http://127.0.0.1:8080"
TALORDATA_API_KEY = "<your-talordata-api-key>"
YDC_API_KEY = "<your-you-api-key>"
TAVILY_API_KEY = "<your-tavily-api-key>"

The environment table is optional. Remove unused entries and never commit a configuration file containing real credentials.

Restart Codex, then run codex mcp list or use /mcp in a Codex session to verify the connection. See the official Codex MCP documentation for configuration options.

Usage

Available Tools

The MCP server exposes two tools:

1. web_search

Search the web for information.

Parameters:

Parameter	Type	Required	Default	Description
query	string	✅	-	Search query string
provider	string	❌	"auto"	Provider: "auto", "searxng", "ddgs", "talordata", "you", or "tavily"
max_results	integer	❌	5	Maximum number of results (1-20)
search_depth	string	❌	"basic"	Search depth: "basic" or "advanced" (Tavily only)
include_answer	boolean	❌	false	Include AI-generated answer (Tavily and TalorData only)

Response:

{
  "query": "search query",
  "provider": "ddgs",
  "count": 5,
  "results": [
    {
      "href": "https://example.com",
      "title": "Result Title",
      "abstract": "Result snippet..."
    }
  ]
}

2. web_extract

Extract content from a public HTTP(S) URL.

Parameters:

Parameter	Type	Required	Default	Description
url	string	✅	-	Public HTTP(S) URL to extract
provider	string	❌	"auto"	Provider to use: "auto", "local", or "tavily"

Provider Options:

• local (default free provider): Free local extraction, no API key required. Uses trafilatura + BeautifulSoup for content extraction.
  • ⚠️ Limitations:
    • Does not execute JavaScript
    • Cannot bypass login walls, paywalls, or CAPTCHA
    • Dynamic/spa-style pages may have incomplete extraction
    • Timeout: 20 seconds (default)
    • Only public HTTP(S) targets are allowed; all resolved IPs and every redirect hop are validated
    • Downloads, extracted characters, redirects, and response Content-Type are bounded
• tavily: Tavily API extraction, requires TAVILY_API_KEY.
• auto (default): Tries local first, falls back to Tavily if TAVILY_API_KEY is available and local fails.

Response:

{
  "url": "https://example.com",
  "content": "Extracted content...",
  "provider": "local"
}

Development

Setup Development Environment

# Clone the repository
git clone https://github.com/YiHarvest/search-engine-tool.git
cd search-engine-tool

uv sync

Run Tests

uv run pytest -v

Build Package

uv build

Check Package

uv run twine check dist/*

Project Structure

search-engine-tool/
├── src/search_engine_tool_mcp/
│   ├── __init__.py
│   ├── server.py          # MCP server implementation
│   ├── search.py          # Web search functionality
│   ├── extract.py         # Content extraction
│   ├── errors.py          # Stable, client-safe error types
│   ├── schemas.py         # Pydantic data models
│   ├── security/
│   │   └── url_validator.py  # DNS-aware SSRF validation
│   └── providers/
│       ├── __init__.py
│       ├── ddgs.py        # DDGS provider (free search)
│       ├── searxng.py     # SearXNG provider (self-hosted search)
│       ├── talordata.py   # TalorData provider (search)
│       ├── you.py         # You.com provider (search)
│       ├── tavily.py      # Tavily provider (search + extract)
│       └── local_extract.py  # Local extraction provider (free)
├── tests/
│   ├── test_search.py
│   ├── test_ddgs.py
│   ├── test_url_validator.py
│   ├── test_local_extract.py
│   ├── test_providers.py
│   └── test_extract.py
├── pyproject.toml
└── README.md

License

MIT License

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Support

For issues and feature requests, please use the GitHub Issues page.