matrx-scraper 0.1.40

Web scraping engine, HTML parsing, and search integration for the Matrx ecosystem

matrx-scraper

Web scraping + HTML parsing + site crawling + search client for Python. An 8-stage parser pipeline turns raw HTML into clean, AI-ready content plus structured extractions (tables, code blocks, links by category, metadata). Designed to work standalone with just httpx, with optional extras for headless browsing, PDF extraction, OCR, and a FastAPI server front-end.

Install

pip install matrx-scraper                  # core: HTTP fetch + parse + crawl + Brave Search
pip install "matrx-scraper[browser]"       # + Playwright / curl_cffi for JS-rendered pages
pip install "matrx-scraper[pdf]"           # + PyMuPDF for PDF extraction
pip install "matrx-scraper[ocr]"           # + Tesseract OCR
pip install "matrx-scraper[connect]"       # + matrx-connect (stream events to a Matrx app)
pip install "matrx-scraper[server]"        # + FastAPI server + uvicorn + asyncpg
pip install "matrx-scraper[all]"           # everything

Python 3.12+ required. Depends on matrx-utils; matrx-connect is optional.

What's in the box

• Scraping (matrx_scraper.scraper, matrx_scraper.orchestrator): scrape(url, **opts), scrape_many(urls), scrape_many_stream(urls), ScrapeResult, ScrapeOptions, ScrapeService.
• Parser pipeline (matrx_scraper.parser): 8-stage HTML pipeline — normalize → NoiseRemover → ScrapeFilter → ElementExtractor → LinkExtractor → metadata (extruct) → hashing (MinHash/SimHash) → markdown conversion. Entry points: parse_html(html, **opts) and ParserOrchestrator.
• Crawling (matrx_scraper.crawler): crawl_site(base_url), SiteCrawler — async BFS site traversal with an explicit robots.txt switch. The authenticated direct marketing transport persists only to canonical Supabase web.*; see web_crawl/FEATURE.md.
• Search (matrx_scraper.search): BraveSearchClient.
• Caching (matrx_scraper.cache): CacheBackend with MemoryCache and TwoTierCache (memory + Postgres, via the optional server extras).
• Per-URL / per-domain config (matrx_scraper.domain_config): DomainConfigBackend — default is static, Postgres-backed variant available via the optional extras.
• Browser automation (optional): PlaywrightBrowserPool.
• FastAPI server (optional): matrx-scraper CLI at server/__main__.py; routers under api/.

Usage

One-off scrape

from matrx_scraper import scrape

result = await scrape("https://example.com/article")
print(result.title)
print(result.ai_content)           # clean, AI-ready markdown
print(result.links)                # categorized links
print(result.tables)               # parsed tables
print(result.organized_data)       # structured JSON of the page

ScrapeResult is a rich dataclass with ~20 fields: url, success, content_type, title, ai_content, ai_research_content, markdown_renderable, organized_data, tables, code_blocks, links, hashes, and more.

Parse raw HTML (no HTTP)

from matrx_scraper import parse_html

parsed = parse_html(open("page.html").read())
print(parsed.main_content)

Crawl a full site

from matrx_scraper import crawl_site

async for page in crawl_site("https://example.com", max_pages=100):
    print(page.url, page.title)

Brave Search

from matrx_scraper.search import BraveSearchClient

client = BraveSearchClient(api_key=settings.BRAVE_API_KEY)
results = await client.search("matrx-scraper python")

Integration with a Matrx host

When used inside a host that has matrx-connect available, you can stream scrape progress as typed events:

import matrx_scraper

matrx_scraper.configure_ext(
    info_payload_cls=InfoPayload,
    warning_payload_cls=WarningPayload,
    # … other Matrx event types
)

After this, scrape_many_stream and ScrapeService will emit matrx-connect event payloads. If configure_ext is not called, the package still works — it just doesn't emit stream events.

Local development (aidream monorepo)

Browser automation, homepage previews, and screenshots run in a separate scraper-service container (Chromium / Playwright is not installed in the aidream venv). The Matrix frontend calls the canonical crawler commands on this service directly; AI Dream is not an intermediary. Older dashboard features may still proxy unrelated preview routes via MATRX_SCRAPER_URL.

Quick start (from monorepo root — in VS Code: Terminal → New Terminal, Ctrl+` / Cmd+`):

1. Start Docker Desktop.
2. ./scripts/scraper-local.sh up — builds if needed, listens on http://localhost:8001.
3. In aidream .env:

   MATRX_SCRAPER_URL=http://localhost:8001
   MATRX_SCRAPER_TOKEN=<token>
   

4. Run aidream (uv run run.py) + dashboard; refresh the scraper tab.

Command	Purpose
./scripts/scraper-local.sh status	Container + health probe
./scripts/scraper-local.sh logs	Follow logs (Ctrl+C detaches)
./scripts/scraper-local.sh down	Stop
./scripts/scraper-local.sh restart	Restart after code changes
./scripts/scraper-local.sh rebuild	Full image rebuild (slow)

After a machine reboot, run up again. Production: set MATRX_SCRAPER_URL on the aidream container to the deployed scraper-service URL (see DEVELOPER_GUIDE.md).

Dependency posture

Core dependencies are a small set of well-known libraries (httpx, beautifulsoup4, selectolax, markdownify, tldextract, tabulate, python-dotenv) plus matrx-utils. All heavier dependencies (Playwright, PyMuPDF, Tesseract, FastAPI) live behind optional extras so lean installs stay lean.

Documentation

Doc	Purpose
DEVELOPER_GUIDE.md	Production server setup, API contract, scraper-postgres env vars, retry queue — hand this to external devs
SCHEMA.md	Supabase web-crawler schema (scraper.* tables)
MIGRATION_GUIDE.md	/api/v1 → /api/scraper API migration
STANDALONE_USAGE.md	Embed or run as microservice
matrx_scraper/web_crawl/FEATURE.md	Direct command/stream contract and canonical web.* persistence
../../docs/scraper/README.md	Monorepo scraper doc index

When deployment or API behavior changes, update DEVELOPER_GUIDE.md in the same PR.

Migration notes

This package replaces the legacy root-level scraper/ folder in the aidream monorepo and parts of research/. Internal docs (MIGRATION_STATUS.md, GAPS_TO_FIX.md, LEGACY_AUDIT.md, MIGRATION_GUIDE.md) track what has been ported and what hasn't.

Contributing

See CLAUDE.md for package-specific rules. This package lives in the aidream monorepo at github.com/AI-Matrix-Engine/aidream-current.

License

MIT.