mcp-web-tools 0.1.0

A powerful MCP server to equip LLMs with web access, search, and content extraction capabilities

MCP Web Tools

This package provides a powerful MCP server to equip LLMs with web access, going beyond naive methods of searching, fetching and extracting content.

Introduction

I created this package out of the frustration that most MCP servers enabling web access to LLMs, didn't perform as well as I hoped. Some of these shortcomings I wanted fix, include:

• Good search results without requiring an API key
• Sophisticated fetching for more complex JavaScript sites
• Extracting content in nicely formatted Markdown
• Support for extracting content from PDFs
• Usage options for advanced cases like loading raw HTML

Internals

The package is written in Python using powerful libraries and services under the hood to improve results.

Searching

We use the Brave Search API if an API key is provided with fallbacks a scraped Google workaround and finally DuckDuckGO if everything else fails. While we'd recommend getting a free Brave API key, the fallbacks should work more than well enough for most use cases and work loads.

Fetching

The fetching of web content is based on Zendriver, a fork of nodriver for next level webscraping and performance. It should stay undetected for most anti-bot solutions and fetch content even from complex JS-based sites.

Extracting

For web extraction, we use Trafilatura which consistently outperforms other alternatives for extracting content from HTML pages. For PDFs, we use PyMuPDF4LLM which similarly extracts content in an easy-to-read format for LLMs, with advanced layout support.

Contributing

While it's impossible to support all pages and layouts, we thrive to make this package better over time. For unsupported sites, problems, or feature requests open an issue.