matrx-scraper

Web scraping + HTML parsing + site crawling + search client for Python. An 8-stage parser pipeline turns raw HTML into clean, AI-ready content plus structured extractions (tables, code blocks, links by category, metadata). Designed to work standalone with just httpx, with optional extras for headless browsing, PDF extraction, OCR, and a FastAPI server front-end.

Install

pip install matrx-scraper                  # core: HTTP fetch + parse + crawl + Brave Search
pip install "matrx-scraper[browser]"       # + Playwright / curl_cffi for JS-rendered pages
pip install "matrx-scraper[pdf]"           # + PyMuPDF for PDF extraction
pip install "matrx-scraper[ocr]"           # + Tesseract OCR
pip install "matrx-scraper[connect]"       # + matrx-connect (stream events to a Matrx app)
pip install "matrx-scraper[server]"        # + FastAPI server + uvicorn + asyncpg
pip install "matrx-scraper[all]"           # everything

Python 3.12+ required. Depends on matrx-utils; matrx-connect is optional.

What's in the box

  • Scraping (matrx_scraper.scraper, matrx_scraper.orchestrator): scrape(url, **opts), scrape_many(urls), scrape_many_stream(urls), ScrapeResult, ScrapeOptions, ScrapeService.
  • Parser pipeline (matrx_scraper.parser): 8-stage HTML pipeline — normalize → NoiseRemover → ScrapeFilter → ElementExtractor → LinkExtractor → metadata (extruct) → hashing (MinHash/SimHash) → markdown conversion. Entry points: parse_html(html, **opts) and ParserOrchestrator.
  • Crawling (matrx_scraper.crawler): crawl_site(base_url), SiteCrawler — async BFS site traversal with an explicit robots.txt switch. The authenticated direct marketing transport persists only to canonical Supabase web.*; see web_crawl/FEATURE.md.
  • Search (matrx_scraper.search): BraveSearchClient.
  • Caching (matrx_scraper.cache): CacheBackend with MemoryCache and TwoTierCache (memory + Postgres, via the optional server extras).
  • Per-URL / per-domain config (matrx_scraper.domain_config): DomainConfigBackend — default is static, Postgres-backed variant available via the optional extras.
  • Browser automation (optional): PlaywrightBrowserPool.
  • FastAPI server (optional): matrx-scraper CLI at server/__main__.py; routers under api/.

Usage

One-off scrape

from matrx_scraper import scrape

result = await scrape("https://example.com/article")
print(result.title)
print(result.ai_content)           # clean, AI-ready markdown
print(result.links)                # categorized links
print(result.tables)               # parsed tables
print(result.organized_data)       # structured JSON of the page

ScrapeResult is a rich dataclass with ~20 fields: url, success, content_type, title, ai_content, ai_research_content, markdown_renderable, organized_data, tables, code_blocks, links, hashes, and more.

Parse raw HTML (no HTTP)

from matrx_scraper import parse_html

parsed = parse_html(open("page.html").read())
print(parsed.main_content)

Crawl a full site

from matrx_scraper import crawl_site

async for page in crawl_site("https://example.com", max_pages=100):
    print(page.url, page.title)

Brave Search

from matrx_scraper.search import BraveSearchClient

client = BraveSearchClient(api_key=settings.BRAVE_API_KEY)
results = await client.search("matrx-scraper python")

Integration with a Matrx host

When used inside a host that has matrx-connect available, you can stream scrape progress as typed events:

import matrx_scraper

matrx_scraper.configure_ext(
    info_payload_cls=InfoPayload,
    warning_payload_cls=WarningPayload,
    # … other Matrx event types
)

After this, scrape_many_stream and ScrapeService will emit matrx-connect event payloads. If configure_ext is not called, the package still works — it just doesn't emit stream events.

Local development (aidream monorepo)

Browser automation, homepage previews, and screenshots run in a separate scraper-service container (Chromium / Playwright is not installed in the aidream venv). The Matrix frontend calls the canonical crawler commands on this service directly; AI Dream is not an intermediary. Older dashboard features may still proxy unrelated preview routes via MATRX_SCRAPER_URL.

Quick start (from monorepo root — in VS Code: Terminal → New Terminal, Ctrl+` / Cmd+`):

  1. Start Docker Desktop.
  2. ./scripts/scraper-local.sh up — builds if needed, listens on http://localhost:8001.
  3. In aidream .env: 
    MATRX_SCRAPER_URL=http://localhost:8001
MATRX_SCRAPER_TOKEN=<token>
  4. Run aidream (uv run run.py) + dashboard; refresh the scraper tab.
Command Purpose
./scripts/scraper-local.sh status Container + health probe
./scripts/scraper-local.sh logs Follow logs (Ctrl+C detaches)
./scripts/scraper-local.sh down Stop
./scripts/scraper-local.sh restart Restart after code changes
./scripts/scraper-local.sh rebuild Full image rebuild (slow)

After a machine reboot, run up again. Production: set MATRX_SCRAPER_URL on the aidream container to the deployed scraper-service URL (see DEVELOPER_GUIDE.md).

Dependency posture

Core dependencies are a small set of well-known libraries (httpx, beautifulsoup4, selectolax, markdownify, tldextract, tabulate, python-dotenv) plus matrx-utils. All heavier dependencies (Playwright, PyMuPDF, Tesseract, FastAPI) live behind optional extras so lean installs stay lean.

Documentation

Doc Purpose
DEVELOPER_GUIDE.md Production server setup, API contract, scraper-postgres env vars, retry queue — hand this to external devs
SCHEMA.md Supabase web-crawler schema (scraper.* tables)
MIGRATION_GUIDE.md /api/v1 → /api/scraper API migration
STANDALONE_USAGE.md Embed or run as microservice
matrx_scraper/web_crawl/FEATURE.md Direct command/stream contract and canonical web.* persistence
../../docs/archive/2026/scraper__README.md Old monorepo scraper doc index (archived)

When deployment or API behavior changes, update DEVELOPER_GUIDE.md in the same PR.

Migration notes

This package replaces the legacy root-level scraper/ folder in the aidream monorepo and parts of research/. Internal docs (MIGRATION_STATUS.md, GAPS_TO_FIX.md, LEGACY_AUDIT.md, MIGRATION_GUIDE.md) track what has been ported and what hasn't.

Contributing

See CLAUDE.md for package-specific rules. This package lives in the aidream monorepo at github.com/AI-Matrix-Engine/aidream-current.

License

MIT.

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.2.152)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for armanisadeghi from gravatar.com armanisadeghi

Credits

Author/Maintainer: Matrx

License

MIT License (MIT)

Requires

Python >=3.12

Provides Extra

all browser connect db dedup durable graph metadata ocr pdf server

Tags

crawler html matrx parsing scraping search web-scraping

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

matrx_scraper-0.1.94.tar.gz (474.3 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

matrx_scraper-0.1.94-py3-none-any.whl (382.8 kB view details)

Uploaded Python 3

File details

Details for the file matrx_scraper-0.1.94.tar.gz.

File metadata

  • Download URL: matrx_scraper-0.1.94.tar.gz
  • Upload date:
  • Size: 474.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for matrx_scraper-0.1.94.tar.gz
Algorithm Hash digest
SHA256 98017f307816f94c0c47e4760877fece59e6595a40ad31e3f3ff9a808d5e8f13
MD5 21de73c0656ee37b5b72661a4855396c
BLAKE2b-256 698086ec06d1733ea287c8dd007ca7c52ef5cfdcab2e2b8fec60ef5a1b7eb5fb

See more details on using hashes here.

Provenance

The following attestation bundles were made for matrx_scraper-0.1.94.tar.gz:

Publisher: publish-package.yml on AI-Matrix-Engine/aidream

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file matrx_scraper-0.1.94-py3-none-any.whl.

File metadata

  • Download URL: matrx_scraper-0.1.94-py3-none-any.whl
  • Upload date:
  • Size: 382.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for matrx_scraper-0.1.94-py3-none-any.whl
Algorithm Hash digest
SHA256 3cb92cef2ca7df3d93aa30c02f4569efa6579433bac26fd0bf85fb6a828c1650
MD5 bfa32549be4ffd34cc0f805611e54083
BLAKE2b-256 c1d1f025fe07801399a836becf1d3bd84d032dc75641fa68986099eec0f1369d

See more details on using hashes here.

Provenance

The following attestation bundles were made for matrx_scraper-0.1.94-py3-none-any.whl:

Publisher: publish-package.yml on AI-Matrix-Engine/aidream

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.2.152

2 files

0.2.151

2 files

0.2.150

2 files

0.2.149

2 files

0.2.148

2 files

0.2.147

2 files

0.2.146

2 files

0.2.145

2 files

0.2.144

2 files

0.2.143

2 files

0.2.142

2 files

0.2.141

2 files

0.2.140

2 files

0.2.139

2 files

0.2.138

2 files

0.2.137

2 files

0.2.136

2 files

0.2.135

2 files

0.2.134

2 files

0.2.133

2 files

0.2.132

2 files

0.2.131

2 files

0.2.130

2 files

0.2.129

2 files

0.2.128

2 files

0.2.127

2 files

0.2.126

2 files

0.2.125

2 files

0.2.124

2 files

0.2.123

2 files

0.2.122

2 files

0.2.121

2 files

0.2.120

2 files

0.2.119

2 files

0.2.118

2 files

0.2.117

2 files

0.2.116

2 files

0.2.115

2 files

0.2.114

2 files

0.2.113

2 files

0.2.112

2 files

0.2.111

2 files

0.2.110

2 files

0.2.109

2 files

0.2.108

2 files

0.2.107

2 files

0.2.106

2 files

0.2.105

2 files

0.2.104

2 files

0.2.103

2 files

0.2.102

2 files

0.2.101

2 files

0.2.100

2 files

0.2.99

2 files

0.2.98

2 files

0.2.97

2 files

0.2.96

2 files

0.2.95

2 files

0.2.94

2 files

0.2.93

2 files

0.2.92

2 files

0.2.91

2 files

0.2.90

2 files

0.2.89

2 files

0.2.88

2 files

0.2.87

2 files

0.2.86

2 files

0.2.85

2 files

0.2.84

2 files

0.2.83

2 files

0.2.82

2 files

0.2.81

2 files

0.2.80

2 files

0.2.79

2 files

0.2.78

2 files

0.2.77

2 files

0.2.76

2 files

0.2.75

2 files

0.2.74

2 files

0.2.73

2 files

0.2.72

2 files

0.2.71

2 files

0.2.70

2 files

0.2.69

2 files

0.2.68

2 files

0.2.67

2 files

0.2.66

2 files

0.2.65

2 files

0.2.64

2 files

0.2.63

2 files

0.2.62

2 files

0.2.61

2 files

0.2.60

2 files

0.2.59

2 files

0.2.58

2 files

0.2.57

2 files

0.2.56

2 files

0.2.55

2 files

0.2.54

2 files

0.2.53

2 files

0.2.52

2 files

0.2.51

2 files

0.2.50

2 files

0.2.49

2 files

0.2.48

2 files

0.2.47

2 files

0.2.46

2 files

0.2.45

2 files

0.2.44

2 files

0.2.43

2 files

0.2.42

2 files

0.2.41

2 files

0.2.40

2 files

0.2.39

2 files

0.2.38

2 files

0.2.37

2 files

0.2.36

2 files

0.2.35

2 files

0.2.34

2 files

0.2.33

2 files

0.2.32

2 files

0.2.31

2 files

0.2.30

2 files

0.2.29

2 files

0.2.28

2 files

0.2.27

2 files

0.2.26

2 files

0.2.25

2 files

0.2.24

2 files

0.2.23

2 files

0.2.22

2 files

0.2.21

2 files

0.2.20

2 files

0.2.19

2 files

0.2.18

2 files

0.2.17

2 files

0.2.16

2 files

0.2.15

2 files

0.2.14

2 files

0.2.13

2 files

0.2.12

2 files

0.2.11

2 files

0.2.10

2 files

0.2.9

2 files

0.2.8

2 files

0.2.7

2 files

0.2.6

2 files

0.2.5

2 files

0.2.4

2 files

0.2.3

2 files

0.2.2

2 files

0.2.1

2 files

0.2.0

2 files

0.1.113

2 files

0.1.112

2 files

0.1.111

2 files

0.1.110

2 files

0.1.109

2 files

0.1.108

2 files

0.1.107

2 files

0.1.106

2 files

0.1.105

2 files

0.1.104

2 files

0.1.103

2 files

0.1.102

2 files

0.1.101

2 files

0.1.100

2 files

0.1.99

2 files

0.1.98

2 files

0.1.97

2 files

0.1.96

2 files

0.1.95

2 files

This release

0.1.94 This release

2 files

0.1.93

2 files

0.1.92

2 files

0.1.91

2 files

0.1.90

2 files

0.1.89

2 files

0.1.88

2 files

0.1.87

2 files

0.1.86

2 files

0.1.85

2 files

0.1.84

2 files

0.1.83

2 files

0.1.82

2 files

0.1.81

2 files

0.1.80

2 files

0.1.79

2 files

0.1.78

2 files

0.1.77

2 files

0.1.76

2 files

0.1.75

2 files

0.1.74

2 files

0.1.73

2 files

0.1.72

2 files

0.1.71

2 files

0.1.70

2 files

0.1.69

2 files

0.1.68

2 files

0.1.67

2 files

0.1.66

2 files

0.1.65

2 files

0.1.64

2 files

0.1.63

2 files

0.1.62

2 files

0.1.61

2 files

0.1.60

2 files

0.1.59

2 files

0.1.58

2 files

0.1.57

2 files

0.1.56

2 files

0.1.55

2 files

0.1.54

2 files

0.1.53

2 files

0.1.52

2 files

0.1.51

2 files

0.1.50

2 files

0.1.49

2 files

0.1.48

2 files

0.1.47

2 files

0.1.46

2 files

0.1.45

2 files

0.1.44

2 files

0.1.43

2 files

0.1.42

2 files

0.1.41

2 files

0.1.40

2 files

0.1.39

2 files

0.1.38

2 files

0.1.37

2 files

0.1.36

2 files

0.1.35

2 files

0.1.34

2 files

0.1.33

2 files

0.1.32

2 files

0.1.31

2 files

0.1.30

2 files

0.1.29

2 files

0.1.28

2 files

0.1.27

2 files

0.1.26

2 files

0.1.25

2 files

0.1.24

2 files

0.1.23

2 files

0.1.22

2 files

0.1.21

2 files

0.1.20

2 files

0.1.19

2 files

0.1.18

2 files

0.1.17

2 files

0.1.16

2 files

0.1.15

2 files

0.1.14

2 files

0.1.13

2 files

0.1.12

2 files

0.1.11

2 files

0.1.10

2 files

0.1.9

2 files

0.1.8

2 files

0.1.7

2 files

0.1.6

2 files

0.1.5

2 files

0.1.4

2 files

0.1.3

2 files

0.1.2

2 files

0.1.1

2 files

0.1.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page