Skip to main content

py-search-helper

CI PyPI version Python 3.14+ License: MIT

Overview

py-search-helper is a Python library that offers a unified interface for searching the internet and extracting web content. It integrates various search engines and provides functionality for clean text extraction from web pages. Additionally, it features an MCP (Model Context Protocol) server, enabling seamless integration with AI agents for research and information gathering tasks.

Note: Single Page Applications (SPAs) cannot be parsed correctly from the initial HTML response because substantive content is dynamically rendered by JavaScript after load.

Features

  • Search provider: DuckDuckGo (wraps several search engines)
  • Domain-specific search filtering (restrict results to specific websites)
  • Extract clean text content from web pages using dual-extractor fallback (markitdown → trafilatura)
  • Discover available search engines programmatically
  • MCP (Model Context Protocol) server for AI agent integration
  • Configurable result limits and character truncation
  • Type-safe API with comprehensive exception handling

Prerequisites

  • Python 3.14+ (Download Python)
  • uv (recommended for dependency management and environment setup)
  • Internet connection for search and content extraction

Installation

git clone https://github.com/sanyokkua/py_search_helper.git
cd py-search-helper
uv sync --all-extras

From PyPI (for library/mcp users)

pip install py-search-helper

Verify Installation

uv run python -c "from py_search_helper import get_search_engines; print(get_search_engines())"
# Expected output (or similar): [('ddgs', 'General web search (DuckDuckGo)')]

Configuration

MCP

Configuration Example:

{
  "mcpServers": {
    "py-search-helper": {
      "command": "uv",
      "args": ["run", "python", "-m", "py_search_helper.mcp"]
    }
  }
}

Quick Start

Library Usage

from py_search_helper import get_search_engines, search, open_url

# Discover available search engines
engines = get_search_engines()
print(engines)
# Output: [('ddgs', 'General web search (DuckDuckGo)'), ('pyside', 'Qt for Python official documentation'), ('wikipedia', 'Wikipedia encyclopedia')]

# Search with default settings (max_results=10)
results = search(engine="ddgs", query="python asyncio")
print(results)  # Markdown-formatted search results

# Search with custom limit
results = search(engine="ddgs", query="python", max_results=5)

# Search specific domain
docs_results = search(engine="ddgs", query="asyncio tutorial", site="docs.python.org")

# Search PySide documentation
pyside_results = search(engine="pyside", query="QPushButton", max_results=5)
print(pyside_results)  # PySide6 documentation results

# Search Wikipedia
wiki_results = search(engine="wikipedia", query="Python programming language", max_results=5)
print(wiki_results)  # Wikipedia articles (all languages)

# Extract content with default limit (max_chars=500)
content = open_url("https://example.com")
print(content)  # First 500 characters

# Extract unlimited content
full_content = open_url("https://example.com", max_chars=None)

CLI Usage

Once installed, the py-search-helper command provides access to the library's core functionalities directly from your terminal.

List available search engines:

py-search-helper get-engines
# Example Output:
# Available Search Engines:
#   - ddgs: General web search (DuckDuckGo)
#   - pyside: Qt for Python official documentation
#   - wikipedia: Wikipedia encyclopedia

Perform a web search:

# Basic search
py-search-helper search ddgs "python typer"

# Search with custom result limit
py-search-helper search ddgs "python requests" -m 5

# Search a specific domain
py-search-helper search ddgs "asyncio tutorial" -s docs.python.org

Extract content from a URL:

# Extract content with default character limit (500 chars)
py-search-helper open-page https://www.example.com

# Extract full content (unlimited characters)
py-search-helper open-page https://www.example.com -c 0

MCP Server Usage

Start the MCP server for AI agent integration:

Use MCP Inspector for testing MCPs

# STDIO mode
uv run py-search-helper-mcp

For MCP Inspector use command uv and as parameters:
`--directory ~path_to_th_project/py-search-helper run py-search-helper-mcp`

# HTTP mode (requires fastmcp[http])
fastmcp run py_search_helper.mcp.server:mcp --transport http --port 8000

API Reference

get_search_engines()

Returns list of available search engines.

Returns: list[tuple[str, str]] - List of (engine_id, description) tuples

Raises: EngineError - If engine registry fails

Example:

engines = get_search_engines()
for engine_id, description in engines:
    print(f"{engine_id}: {description}")

search(engine, query, max_results=10, *, site=None)

Search using specified engine with optional domain filtering.

Parameters:

  • engine (str): Engine ID (e.g., "ddgs", "pyside", "wikipedia")
  • query (str): Search query string
  • max_results (int): Maximum results to return (default: 10, max recommended: 30)
  • site (str | None): Optional domain to restrict search (default: None)

Returns: str - Markdown-formatted search results

Raises:

  • ValueError - If query is empty, max_results < 1, or site is empty
  • EngineNotFoundError - If engine is not registered
  • SearchProviderError - If search operation fails

Examples:

# General search
results = search(engine="ddgs", query="python typing", max_results=5)

# Search specific domain
results = search(
    engine="ddgs",
    query="machine learning",
    site="arxiv.org",
    max_results=10
)

# Search Stack Overflow
results = search(engine="ddgs", query="async await", site="stackoverflow.com")

open_url(url, max_chars=500)

Extract content from a URL.

Parameters:

  • url (str): URL to open (must start with http:// or https://)
  • max_chars (int | None): Maximum characters to return (default: 500, None for unlimited)

Returns: str - Markdown-formatted page content

Raises:

  • ValueError - If URL is invalid or max_chars < 1
  • URLNotFoundError - If URL returns 404
  • URLTimeoutError - If request times out
  • URLError - If extraction fails

Example:

# With character limit
content = open_url("https://example.com", max_chars=1000)

# Unlimited
full_content = open_url("https://example.com", max_chars=None)

MCP Server Tools

The MCP server exposes three tools for AI agents:

get_engines

List available search engines.

Parameters: None

Returns: List of (engine_id, description) tuples

search_web

Search using specified engine with optional domain filtering.

Parameters:

  • engine (str): Engine ID
  • query (str): Search query
  • max_results (int): Maximum results (default: 10)
  • site (str | None): Optional domain to restrict search (default: None)

Returns: Markdown-formatted search results

Examples:

# General search
search_web(engine="ddgs", query="python asyncio", max_results=5)

# Search specific domain
search_web(engine="ddgs", query="asyncio", site="docs.python.org", max_results=5)

search_web_ddg

Search using DuckDuckGo engine with optional domain filtering.

Parameters:

  • query (str): Search query
  • max_results (int): Maximum results (default: 10)
  • site (str | None): Optional domain to restrict search (default: None)

Returns: Markdown-formatted search results

Examples:

# General search
search_web_ddg(query="python asyncio", max_results=5)

# Search specific domain
search_web_ddg(query="asyncio", site="docs.python.org", max_results=5)

open_page

Extract content from a URL.

Parameters:

  • url (str): URL to open
  • max_chars (int | None): Maximum characters (default: 500)

Returns: Markdown-formatted page content

Error Handling

The library uses a hierarchical exception system:

PySearchHelperError (base)
├── EngineError
│   └── EngineNotFoundError
├── SearchError
│   └── SearchProviderError
└── URLError
    ├── URLNotFoundError
    └── URLTimeoutError

Example:

from py_search_helper import search, open_url
from py_search_helper.exceptions import (
    EngineNotFoundError,
    SearchProviderError,
    URLError,
)

try:
    results = search(engine="ddgs", query="python")
except EngineNotFoundError:
    print("Engine not found")
except SearchProviderError:
    print("Search failed")

try:
    content = open_url("https://example.com")
except URLError as e:
    print(f"URL error: {e}")

Available Search Engines

  • Engine ID: ddgs
  • Description: General web search using DuckDuckGo
  • Features: Rate limiting, retry logic, configurable result limits
  • Max Results: 30 (recommended limit)

PySide Documentation

  • Engine ID: pyside
  • Description: Search Qt for Python documentation
  • Implementation: Uses DuckDuckGo with site:doc.qt.io/qtforpython-6 filter
  • Scope: PySide6 API documentation, tutorials, examples, and guides

Wikipedia

  • Engine ID: wikipedia
  • Description: Search Wikipedia encyclopedia
  • Implementation: Uses DuckDuckGo with site:wikipedia.org filter
  • Scope: Wikipedia articles and encyclopedia content (all language editions)

Examples

See the examples/ directory for complete usage examples:

  • basic_usage.py - Core API demonstrations (8 examples)
  • site_filtering.py - Examples with a search with site filter

Run examples:

uv run python examples/basic_usage.py
uv run python examples/site_filtering.py

Limitations

  • DDGS provider subject to DuckDuckGo rate limits (6 requests/minute default)
  • Content extraction may fail for JavaScript-heavy sites
  • Some sites block automated scraping

Documentation

Architecture & Design:

Development:

MCP Server:

See Development Guide for detailed setup instructions.

License

MIT License - See LICENSE file for details

Metadata

Release files for py-search-helper 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for py-search-helper 0.1.0
File Size Uploaded Provenance Actions
py_search_helper-0.1.0.tar.gz 245.8 kB Without provenance Details for py_search_helper-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for py-search-helper 0.1.0
File Interpreter ABI Platform Actions
py_search_helper-0.1.0-py3-none-any.whl Python 3 none any Details for py_search_helper-0.1.0-py3-none-any.whl

Total release size: 268.4 kB

Release files / py_search_helper-0.1.0.tar.gz

Download URL py_search_helper-0.1.0.tar.gz
Size 245.8 kB
Tags Source
SHA-256 checksum
How to use checksums
a18cd3e96e1384a295e0e9546ea4ce205cb0217cb1ef2acb22646f8231a671b3
BLAKE2b-256 checksum
How to use checksums
ecb7ac0d26c4306c8481e031a5388226e32582f8c6e00cf2f1f0d440a1c30c03
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.10.2 {"installer":{"name":"uv","version":"0.10.2","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / py_search_helper-0.1.0-py3-none-any.whl

Download URL py_search_helper-0.1.0-py3-none-any.whl
Size 22.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
1e28e9d12402be5a3a9a84693acdde3001ab3ca7d32f52855a2de8429247810e
BLAKE2b-256 checksum
How to use checksums
653e75288f95f08f94dabc464e06702aa7df26cbade047bdeab338d67929dbcd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.10.2 {"installer":{"name":"uv","version":"0.10.2","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for py-search-helper 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page