matrx-scraper

Web scraping + HTML parsing + site crawling + search client for Python. An 8-stage parser pipeline turns raw HTML into clean, AI-ready content plus structured extractions (tables, code blocks, links by category, metadata). Designed to work standalone with just httpx, with optional extras for headless browsing, PDF extraction, OCR, and a FastAPI server front-end.

Install

pip install matrx-scraper                  # core: HTTP fetch + parse + crawl + Brave Search
pip install "matrx-scraper[browser]"       # + Playwright / curl_cffi for JS-rendered pages
pip install "matrx-scraper[pdf]"           # + PyMuPDF for PDF extraction
pip install "matrx-scraper[ocr]"           # + Tesseract OCR
pip install "matrx-scraper[connect]"       # + matrx-connect (stream events to a Matrx app)
pip install "matrx-scraper[server]"        # + FastAPI server + uvicorn + asyncpg
pip install "matrx-scraper[all]"           # everything

Python 3.12+ required. Depends on matrx-utils; matrx-connect is optional.

What's in the box

  • Scraping (matrx_scraper.scraper, matrx_scraper.orchestrator): scrape(url, **opts), scrape_many(urls), scrape_many_stream(urls), ScrapeResult, ScrapeOptions, ScrapeService.
  • Parser pipeline (matrx_scraper.parser): 8-stage HTML pipeline — normalize → NoiseRemover → ScrapeFilter → ElementExtractor → LinkExtractor → metadata (extruct) → hashing (MinHash/SimHash) → markdown conversion. Entry points: parse_html(html, **opts) and ParserOrchestrator.
  • Crawling (matrx_scraper.crawler): crawl_site(base_url), SiteCrawler — async BFS site traversal with an explicit robots.txt switch. The authenticated direct marketing transport persists only to canonical Supabase web.*; see web_crawl/FEATURE.md.
  • Search (matrx_scraper.search): BraveSearchClient.
  • Caching (matrx_scraper.cache): CacheBackend with MemoryCache and TwoTierCache (memory + Postgres, via the optional server extras).
  • Per-URL / per-domain config (matrx_scraper.domain_config): DomainConfigBackend — default is static, Postgres-backed variant available via the optional extras.
  • Browser automation (optional): PlaywrightBrowserPool.
  • FastAPI server (optional): matrx-scraper CLI at server/__main__.py; routers under api/.

Usage

One-off scrape

from matrx_scraper import scrape

result = await scrape("https://example.com/article")
print(result.title)
print(result.ai_content)           # clean, AI-ready markdown
print(result.links)                # categorized links
print(result.tables)               # parsed tables
print(result.organized_data)       # structured JSON of the page

ScrapeResult is a rich dataclass with ~20 fields: url, success, content_type, title, ai_content, ai_research_content, markdown_renderable, organized_data, tables, code_blocks, links, hashes, and more.

Parse raw HTML (no HTTP)

from matrx_scraper import parse_html

parsed = parse_html(open("page.html").read())
print(parsed.main_content)

Crawl a full site

from matrx_scraper import crawl_site

async for page in crawl_site("https://example.com", max_pages=100):
    print(page.url, page.title)

Brave Search

from matrx_scraper.search import BraveSearchClient

client = BraveSearchClient(api_key=settings.BRAVE_API_KEY)
results = await client.search("matrx-scraper python")

Integration with a Matrx host

When used inside a host that has matrx-connect available, you can stream scrape progress as typed events:

import matrx_scraper

matrx_scraper.configure_ext(
    info_payload_cls=InfoPayload,
    warning_payload_cls=WarningPayload,
    # … other Matrx event types
)

After this, scrape_many_stream and ScrapeService will emit matrx-connect event payloads. If configure_ext is not called, the package still works — it just doesn't emit stream events.

Local development (aidream monorepo)

Browser automation, homepage previews, and screenshots run in a separate scraper-service container (Chromium / Playwright is not installed in the aidream venv). The Matrix frontend calls the canonical crawler commands on this service directly; AI Dream is not an intermediary. Older dashboard features may still proxy unrelated preview routes via MATRX_SCRAPER_URL.

Quick start (from monorepo root — in VS Code: Terminal → New Terminal, Ctrl+` / Cmd+`):

  1. Start Docker Desktop.
  2. ./scripts/scraper-local.sh up — builds if needed, listens on http://localhost:8001.
  3. In aidream .env: 
    MATRX_SCRAPER_URL=http://localhost:8001
MATRX_SCRAPER_TOKEN=<token>
  4. Run aidream (uv run run.py) + dashboard; refresh the scraper tab.
Command Purpose
./scripts/scraper-local.sh status Container + health probe
./scripts/scraper-local.sh logs Follow logs (Ctrl+C detaches)
./scripts/scraper-local.sh down Stop
./scripts/scraper-local.sh restart Restart after code changes
./scripts/scraper-local.sh rebuild Full image rebuild (slow)

After a machine reboot, run up again. Production: set MATRX_SCRAPER_URL on the aidream container to the deployed scraper-service URL (see DEVELOPER_GUIDE.md).

Dependency posture

Core dependencies are a small set of well-known libraries (httpx, beautifulsoup4, selectolax, markdownify, tldextract, tabulate, python-dotenv) plus matrx-utils. All heavier dependencies (Playwright, PyMuPDF, Tesseract, FastAPI) live behind optional extras so lean installs stay lean.

Documentation

Doc Purpose
DEVELOPER_GUIDE.md Production server setup, API contract, scraper-postgres env vars, retry queue — hand this to external devs
SCHEMA.md Supabase web-crawler schema (scraper.* tables)
MIGRATION_GUIDE.md /api/v1 → /api/scraper API migration
STANDALONE_USAGE.md Embed or run as microservice
matrx_scraper/web_crawl/FEATURE.md Direct command/stream contract and canonical web.* persistence
../../docs/archive/2026/scraper__README.md Old monorepo scraper doc index (archived)

When deployment or API behavior changes, update DEVELOPER_GUIDE.md in the same PR.

Migration notes

This package replaces the legacy root-level scraper/ folder in the aidream monorepo and parts of research/. Internal docs (MIGRATION_STATUS.md, GAPS_TO_FIX.md, LEGACY_AUDIT.md, MIGRATION_GUIDE.md) track what has been ported and what hasn't.

Contributing

See CLAUDE.md for package-specific rules. This package lives in the aidream monorepo at github.com/AI-Matrix-Engine/aidream-current.

License

MIT.

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.2.152)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for armanisadeghi from gravatar.com armanisadeghi

Credits

Author/Maintainer: Matrx

License

MIT License (MIT)

Requires

Python >=3.12

Provides Extra

all browser connect db dedup durable graph metadata ocr pdf server

Tags

crawler html matrx parsing scraping search web-scraping

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

matrx_scraper-0.1.87.tar.gz (419.7 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

matrx_scraper-0.1.87-py3-none-any.whl (349.7 kB view details)

Uploaded Python 3

File details

Details for the file matrx_scraper-0.1.87.tar.gz.

File metadata

  • Download URL: matrx_scraper-0.1.87.tar.gz
  • Upload date:
  • Size: 419.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for matrx_scraper-0.1.87.tar.gz
Algorithm Hash digest
SHA256 fbf0ba7373030ed822c8d247c78eefe95c15b282c78639cc0bc2f2d8cd828914
MD5 ceb57fe0a904368f61385ffaaa0a391a
BLAKE2b-256 5099b936bf3b8530dc2b4163134facb6870a7bc83aa524d5e61a503db94f155e

See more details on using hashes here.

Provenance

The following attestation bundles were made for matrx_scraper-0.1.87.tar.gz:

Publisher: publish-package.yml on AI-Matrix-Engine/aidream

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file matrx_scraper-0.1.87-py3-none-any.whl.

File metadata

  • Download URL: matrx_scraper-0.1.87-py3-none-any.whl
  • Upload date:
  • Size: 349.7 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for matrx_scraper-0.1.87-py3-none-any.whl
Algorithm Hash digest
SHA256 4f024fe31187bb94f1c1fc6876a2c6d05c4836977e3ac2141a4a6c5d0eee82d0
MD5 be96e1b1c9562e52b4a0867072f17c5e
BLAKE2b-256 41e9b86bc5759de258dd15d5d3778bb77d2a88f754db17fef9c9b030bd029209

See more details on using hashes here.

Provenance

The following attestation bundles were made for matrx_scraper-0.1.87-py3-none-any.whl:

Publisher: publish-package.yml on AI-Matrix-Engine/aidream

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.2.152

2 files

0.2.151

2 files

0.2.150

2 files

0.2.149

2 files

0.2.148

2 files

0.2.147

2 files

0.2.146

2 files

0.2.145

2 files

0.2.144

2 files

0.2.143

2 files

0.2.142

2 files

0.2.141

2 files

0.2.140

2 files

0.2.139

2 files

0.2.138

2 files

0.2.137

2 files

0.2.136

2 files

0.2.135

2 files

0.2.134

2 files

0.2.133

2 files

0.2.132

2 files

0.2.131

2 files

0.2.130

2 files

0.2.129

2 files

0.2.128

2 files

0.2.127

2 files

0.2.126

2 files

0.2.125

2 files

0.2.124

2 files

0.2.123

2 files

0.2.122

2 files

0.2.121

2 files

0.2.120

2 files

0.2.119

2 files

0.2.118

2 files

0.2.117

2 files

0.2.116

2 files

0.2.115

2 files

0.2.114

2 files

0.2.113

2 files

0.2.112

2 files

0.2.111

2 files

0.2.110

2 files

0.2.109

2 files

0.2.108

2 files

0.2.107

2 files

0.2.106

2 files

0.2.105

2 files

0.2.104

2 files

0.2.103

2 files

0.2.102

2 files

0.2.101

2 files

0.2.100

2 files

0.2.99

2 files

0.2.98

2 files

0.2.97

2 files

0.2.96

2 files

0.2.95

2 files

0.2.94

2 files

0.2.93

2 files

0.2.92

2 files

0.2.91

2 files

0.2.90

2 files

0.2.89

2 files

0.2.88

2 files

0.2.87

2 files

0.2.86

2 files

0.2.85

2 files

0.2.84

2 files

0.2.83

2 files

0.2.82

2 files

0.2.81

2 files

0.2.80

2 files

0.2.79

2 files

0.2.78

2 files

0.2.77

2 files

0.2.76

2 files

0.2.75

2 files

0.2.74

2 files

0.2.73

2 files

0.2.72

2 files

0.2.71

2 files

0.2.70

2 files

0.2.69

2 files

0.2.68

2 files

0.2.67

2 files

0.2.66

2 files

0.2.65

2 files

0.2.64

2 files

0.2.63

2 files

0.2.62

2 files

0.2.61

2 files

0.2.60

2 files

0.2.59

2 files

0.2.58

2 files

0.2.57

2 files

0.2.56

2 files

0.2.55

2 files

0.2.54

2 files

0.2.53

2 files

0.2.52

2 files

0.2.51

2 files

0.2.50

2 files

0.2.49

2 files

0.2.48

2 files

0.2.47

2 files

0.2.46

2 files

0.2.45

2 files

0.2.44

2 files

0.2.43

2 files

0.2.42

2 files

0.2.41

2 files

0.2.40

2 files

0.2.39

2 files

0.2.38

2 files

0.2.37

2 files

0.2.36

2 files

0.2.35

2 files

0.2.34

2 files

0.2.33

2 files

0.2.32

2 files

0.2.31

2 files

0.2.30

2 files

0.2.29

2 files

0.2.28

2 files

0.2.27

2 files

0.2.26

2 files

0.2.25

2 files

0.2.24

2 files

0.2.23

2 files

0.2.22

2 files

0.2.21

2 files

0.2.20

2 files

0.2.19

2 files

0.2.18

2 files

0.2.17

2 files

0.2.16

2 files

0.2.15

2 files

0.2.14

2 files

0.2.13

2 files

0.2.12

2 files

0.2.11

2 files

0.2.10

2 files

0.2.9

2 files

0.2.8

2 files

0.2.7

2 files

0.2.6

2 files

0.2.5

2 files

0.2.4

2 files

0.2.3

2 files

0.2.2

2 files

0.2.1

2 files

0.2.0

2 files

0.1.113

2 files

0.1.112

2 files

0.1.111

2 files

0.1.110

2 files

0.1.109

2 files

0.1.108

2 files

0.1.107

2 files

0.1.106

2 files

0.1.105

2 files

0.1.104

2 files

0.1.103

2 files

0.1.102

2 files

0.1.101

2 files

0.1.100

2 files

0.1.99

2 files

0.1.98

2 files

0.1.97

2 files

0.1.96

2 files

0.1.95

2 files

0.1.94

2 files

0.1.93

2 files

0.1.92

2 files

0.1.91

2 files

0.1.90

2 files

0.1.89

2 files

0.1.88

2 files

This release

0.1.87 This release

2 files

0.1.86

2 files

0.1.85

2 files

0.1.84

2 files

0.1.83

2 files

0.1.82

2 files

0.1.81

2 files

0.1.80

2 files

0.1.79

2 files

0.1.78

2 files

0.1.77

2 files

0.1.76

2 files

0.1.75

2 files

0.1.74

2 files

0.1.73

2 files

0.1.72

2 files

0.1.71

2 files

0.1.70

2 files

0.1.69

2 files

0.1.68

2 files

0.1.67

2 files

0.1.66

2 files

0.1.65

2 files

0.1.64

2 files

0.1.63

2 files

0.1.62

2 files

0.1.61

2 files

0.1.60

2 files

0.1.59

2 files

0.1.58

2 files

0.1.57

2 files

0.1.56

2 files

0.1.55

2 files

0.1.54

2 files

0.1.53

2 files

0.1.52

2 files

0.1.51

2 files

0.1.50

2 files

0.1.49

2 files

0.1.48

2 files

0.1.47

2 files

0.1.46

2 files

0.1.45

2 files

0.1.44

2 files

0.1.43

2 files

0.1.42

2 files

0.1.41

2 files

0.1.40

2 files

0.1.39

2 files

0.1.38

2 files

0.1.37

2 files

0.1.36

2 files

0.1.35

2 files

0.1.34

2 files

0.1.33

2 files

0.1.32

2 files

0.1.31

2 files

0.1.30

2 files

0.1.29

2 files

0.1.28

2 files

0.1.27

2 files

0.1.26

2 files

0.1.25

2 files

0.1.24

2 files

0.1.23

2 files

0.1.22

2 files

0.1.21

2 files

0.1.20

2 files

0.1.19

2 files

0.1.18

2 files

0.1.17

2 files

0.1.16

2 files

0.1.15

2 files

0.1.14

2 files

0.1.13

2 files

0.1.12

2 files

0.1.11

2 files

0.1.10

2 files

0.1.9

2 files

0.1.8

2 files

0.1.7

2 files

0.1.6

2 files

0.1.5

2 files

0.1.4

2 files

0.1.3

2 files

0.1.2

2 files

0.1.1

2 files

0.1.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page