ByteCrawl

Give your AI agent focused web crawling. ByteCrawl is an MCP server (and a small Python library) that doesn't just scrape a page — it crawls a whole site and returns the pages most relevant to your topic first, using Shark-Search and OPIC in pure Python.

• Webpage: https://bytecrawl.vercel.app/
• Hosted MCP endpoint: https://bytecrawl.vercel.app/mcp

Quick start (MCP — nothing to install)

Point any MCP-capable agent (Claude Code, Claude Desktop, Cursor...) at the hosted endpoint:

claude mcp add --transport http bytecrawl https://bytecrawl.vercel.app/mcp

Now the agent has four tools:

The hosted server is static-only, rate-limited per IP, caps crawls at 10 pages, and refuses non-public URLs (SSRF guard). For heavy use or JS-rendered sites, run it locally:

pip install bytecrawl[mcp]
claude mcp add bytecrawl -- bytecrawl-mcp                 # full power, on your machine
pip install bytecrawl[browser] && playwright install chromium   # + JS rendering

Why focused crawling?

Most crawlers visit pages in whatever order they find them. With a limited request budget, order is everything — Shark-Search chases the branches that smell like your query and lets the rest decay, so 100 requests get you the 100 most useful pages, not the 100 closest to the seed.

from bytecrawl import SharkSearch

result = SharkSearch(query="vector databases").crawl(
    "https://example.com", max_pages=100)

for page in result.top(10):
    print(f'{page["relevance"]:.3f}  {page["url"]}')

• BFS — level by level, closest to the seed first.
• Shark-Search (Hersovici et al., 1998) — topical best-first; links inherit their parent's relevance with decay.
• OPIC (Abiteboul et al., 2003) — live PageRank via "cash" flow, no full graph needed (a pagerank() implementation is included to compare against).

Versus the alternatives: Scrapy is a framework you wire up yourself, Firecrawl is a paid SaaS — ByteCrawl is a plain library with a 3-package core and these frontier strategies built in.

Library API

from bytecrawl import Scraper

bot = Scraper()
page = bot.fetch("https://books.toscrape.com")   # auto: static, browser fallback
books = page.extract("article.product_pod",
                     {"title": "h3 a::attr(title)", "price": "p.price_color::text"})
page.markdown()   # clean Markdown for LLMs   ·   page.tokens()   # token estimate

bot.static(url)                                   # plain HTML
bot.api(url, params={...})                        # hidden JSON API
bot.browser(url, wait="div.results")              # JS via Playwright
bot.crawl(url, item="article", fields={...},
          next_page="li.next a::attr(href)")      # pagination
bot.session().login(url, data, csrf_field="csrf_token")   # authenticated

Install

pip install bytecrawl            # slim core (requests + beautifulsoup4 + lxml)
pip install bytecrawl[llm]       # + Markdown for LLMs
pip install bytecrawl[browser]   # + Playwright
pip install bytecrawl[mcp]       # + local MCP server
pip install bytecrawl[all]

Learn each scraping technique

A guided walkthrough with a runnable example against a practice site: static HTML · dynamic JS · hidden APIs · pagination · login · graph crawling · Markdown for LLMs · ethics

Contributing

pip install -e ".[llm,dev,mcp]"
pytest              # 109 tests, no network required
pytest -m live      # + live browser tests (needs the browser extra)
ruff check bytecrawl tests

Scrape responsibly: respect robots.txt, terms of service and rate limits. ByteCrawl ships with a configurable delay between requests.

License

MIT — see LICENSE.