NetLens

An MCP server for unobstructed web reading. It fetches any URL directly with browser-like headers — past robots.txt and naive bot blocks — and returns the full page as clean, ad-stripped Markdown, not a summary. Plus web search that returns real links. Zero dependencies: pure Python standard library.

Built for AI Agents

AI agents constantly hit pages their built-in tools can't read. NetLens fixes the three usual reasons a fetch comes back empty or useless:

It does not try to defeat JavaScript/Cloudflare challenge pages or CAPTCHAs — that's out of scope by design. When a page is a hard block, the HTTP status is surfaced honestly rather than faked.

Installation

npm (via npx):

{
  "mcpServers": {
    "netlens": {
      "command": "npx",
      "args": ["-y", "netlens-mcp"]
    }
  }
}

PyPI (via uvx):

{
  "mcpServers": {
    "netlens": {
      "command": "uvx",
      "args": ["netlens-mcp"]
    }
  }
}

Add either to your MCP client config (e.g. .mcp.json for Claude Code), then restart the session so the tools load.

Tools

web_search

Search the web and return real result links (title, URL, snippet), parsed locally — links, not summaries. Follow up with web_fetch to read a result.

A search fetches a single result page (~10 results), returned in full by default so nothing at position 9/10 is dropped. There's no deep pagination — if the answer isn't in the first page, refine the query.

web_fetch

Fetch any page and return its full content as clean Markdown.

On link-heavy pages the URLs themselves are a large share of the output — around 40% of a big encyclopedia article — so links="none" roughly halves it when you only need the prose. Links pointing back into the same page are always rendered as plain text.

If a page turns out to be a client-rendered shell, web_fetch says so rather than returning an empty result as a success:

(note: Only 10 characters of readable text were found in 6,856 characters of HTML. This page appears to be rendered client-side by JavaScript…)

Workflow: web_search to find pages, then web_fetch to read them.

Search engines

Search is a pluggable, selectable registry. In auto mode NetLens tries engines in order and returns the first with results, so a rate-limit/challenge page on one falls through to the next.

Pick per call with the engine argument, or set a default with NETLENS_SEARCH_ENGINE.

Configuration

How it works

• Direct fetch. Requests go straight to the target site via the system curl (better TLS/HTTP-2/compression, so it looks like a real browser), falling back to urllib. No third-party proxy or reader is involved.
• Local conversion. HTML → Markdown happens in-process with a hand-rolled html.parser converter — headings, lists, links (relative URLs resolved), code blocks, and GFM tables with colspan/rowspan.
• Content selection, not deletion. NetLens picks the page's main content region — the HTML5 landmark (<main> / <article> / [role=main]) when one exists, otherwise the subtree holding the most prose relative to its link density — and converts only that. Because it selects a winner rather than deleting anything that matches a name pattern, extraction cannot silently return an empty page. Nothing inspects CSS class or id names to decide what is content.
• Pruning by measurement. Within that region, blocks that are overwhelmingly link anchors (navboxes, tag clouds, "more from this site" grids) are dropped based on their link density. Non-rendering elements (<script>, <style>, …), explicitly hidden elements, and third-party ad-network slots (identified by vendor names like adsbygoogle, which cannot collide with real prose) are removed outright.
• Response charset is honored (from Content-Type or <meta>), so non-UTF-8 pages don't come back garbled.

Usage from the CLI

The server is also a plain script — handy for testing before a client loads it:

python -m netlens_mcp.server search "http caching best practices"
python -m netlens_mcp.server fetch  https://example.com/article
python -m netlens_mcp.server full   https://example.com   # whole body
python -m netlens_mcp.server raw    https://example.com   # unconverted HTML

python -m netlens_mcp runs the stdio MCP server; python -m netlens_mcp.server <cmd> runs the CLI.

Development

pip install -e ".[dev]"
python -m pytest        # run the test suite
ruff check .            # lint

Requirements

• Python 3.10+ (and the system curl, which ships with modern Windows/macOS/Linux; falls back to urllib if absent)

License

Apache License 2.0 — see LICENSE and NOTICE.