mudfish

Python bindings for Mudfish Crawler, a Rust-native web crawling and intelligence engine.

This package wraps the Rust crawl engine directly (via PyO3) — there is no subprocess or network hop to a separate server. It exposes a typed, Pythonic API over the engine; the Rust project itself is at an early stage (see the main repo's ROADMAP_HONEST.md for full status).

Install

pip install mudfish

Usage

import mudfish

result = mudfish.crawl("https://example.com", depth=2, concurrency=30)

print(result.stats.urls_fetched, result.stats.duration_secs)
for page in result:                          # CrawlResult is iterable over its pages
    print(page.status_code, page.url, page.title)

crawl() blocks until the crawl finishes (or hits max_urls / max_duration_secs) and returns a typed CrawlResult:

Every typed object also supports the raw engine dict shape (result["stats"], page["status_code"], page.to_dict()), so code written against earlier versions keeps working. If you only want the raw dict, use mudfish.crawl_dict(...).

Async

crawl_async() runs the blocking engine on a worker thread, so it composes with asyncio and several crawls can run concurrently:

import asyncio
import mudfish

async def main() -> None:
    results = await asyncio.gather(
        mudfish.crawl_async("https://a.example"),
        mudfish.crawl_async("https://b.example"),
    )
    for result in results:
        print(result.stats.urls_fetched)

asyncio.run(main())

Reusable configuration

crawler = mudfish.Crawler(depth=1, concurrency=10)
home = crawler.crawl("https://example.com")
blog = await crawler.crawl_async("https://example.com/blog")

Errors

try:
    mudfish.crawl("not-a-url")
except mudfish.InvalidURLError:      # also a ValueError
    ...
except mudfish.CrawlFailedError:     # also a RuntimeError
    ...

Everything derives from mudfish.MudfishError.

Options

All keyword-only, matching the CLI's flags (also available as the CrawlOptions dataclass):

Limitations

• Chrome is not bundled in the PyPI wheel. With render_js=True the engine drives a headless Chrome that must already be installed on the host; if it cannot be launched, the crawl logs a warning and falls back to static HTML. A separate bundled wheel that ships a chrome-headless-shell (so render_js works with no host install) is attached to the GitHub release for each version — look for the asset named mudfish-<version>+chrome-*.whl.
• Single-platform wheels initially. Built and published from the maintainer's machine, not a cross-platform CI matrix yet — check PyPI for which platforms have wheels; others will need a Rust toolchain to build from source.

Bundled-Chrome builds

To build the bundled wheel locally:

python3 tools/fetch_chrome_headless_shell.py   # pins + downloads chrome-headless-shell
cd crates/python && maturin build --release

The binary is fetched (never committed) into python/mudfish/_chrome/, the wheel picks it up automatically, and at runtime mudfish.bundled_chrome_path() returns it (overridable with MUDFISH_CHROME).

License

Apache-2.0.