matrx-scraper

Web scraping + HTML parsing + site crawling + search client for Python. An 8-stage parser pipeline turns raw HTML into clean, AI-ready content plus structured extractions (tables, code blocks, links by category, metadata). Designed to work standalone with just httpx, with optional extras for headless browsing, PDF extraction, OCR, and a FastAPI server front-end.

Install

pip install matrx-scraper                  # core: HTTP fetch + parse + crawl + Brave Search
pip install "matrx-scraper[browser]"       # + Playwright / curl_cffi for JS-rendered pages
pip install "matrx-scraper[pdf]"           # + PyMuPDF for PDF extraction
pip install "matrx-scraper[ocr]"           # + Tesseract OCR
pip install "matrx-scraper[connect]"       # + matrx-connect (stream events to a Matrx app)
pip install "matrx-scraper[server]"        # + FastAPI server + uvicorn + asyncpg
pip install "matrx-scraper[all]"           # everything

Python 3.12+ required. Depends on matrx-utils; matrx-connect is optional.

What's in the box

  • Scraping (matrx_scraper.scraper, matrx_scraper.orchestrator): scrape(url, **opts), scrape_many(urls), scrape_many_stream(urls), ScrapeResult, ScrapeOptions, ScrapeService.
  • Parser pipeline (matrx_scraper.parser): 8-stage HTML pipeline — normalize → NoiseRemover → ScrapeFilter → ElementExtractor → LinkExtractor → metadata (extruct) → hashing (MinHash/SimHash) → markdown conversion. Entry points: parse_html(html, **opts) and ParserOrchestrator.
  • Crawling (matrx_scraper.crawler): crawl_site(base_url), SiteCrawler — async BFS site traversal with an explicit robots.txt switch. The authenticated direct marketing transport persists only to canonical Supabase web.*; see web_crawl/FEATURE.md.
  • Search (matrx_scraper.search): BraveSearchClient.
  • Caching (matrx_scraper.cache): CacheBackend with MemoryCache and TwoTierCache (memory + Postgres, via the optional server extras).
  • Per-URL / per-domain config (matrx_scraper.domain_config): DomainConfigBackend — default is static, Postgres-backed variant available via the optional extras.
  • Browser automation (optional): PlaywrightBrowserPool.
  • FastAPI server (optional): matrx-scraper CLI at server/__main__.py; routers under api/.

Usage

One-off scrape

from matrx_scraper import scrape

result = await scrape("https://example.com/article")
print(result.title)
print(result.ai_content)           # clean, AI-ready markdown
print(result.links)                # categorized links
print(result.tables)               # parsed tables
print(result.organized_data)       # structured JSON of the page

ScrapeResult is a rich dataclass with ~20 fields: url, success, content_type, title, ai_content, ai_research_content, markdown_renderable, organized_data, tables, code_blocks, links, hashes, and more.

Parse raw HTML (no HTTP)

from matrx_scraper import parse_html

parsed = parse_html(open("page.html").read())
print(parsed.main_content)

Crawl a full site

from matrx_scraper import crawl_site

async for page in crawl_site("https://example.com", max_pages=100):
    print(page.url, page.title)

Brave Search

from matrx_scraper.search import BraveSearchClient

client = BraveSearchClient(api_key=settings.BRAVE_API_KEY)
results = await client.search("matrx-scraper python")

Integration with a Matrx host

When used inside a host that has matrx-connect available, you can stream scrape progress as typed events:

import matrx_scraper

matrx_scraper.configure_ext(
    info_payload_cls=InfoPayload,
    warning_payload_cls=WarningPayload,
    # … other Matrx event types
)

After this, scrape_many_stream and ScrapeService will emit matrx-connect event payloads. If configure_ext is not called, the package still works — it just doesn't emit stream events.

Local development (aidream monorepo)

Browser automation, homepage previews, and screenshots run in a separate scraper-service container (Chromium / Playwright is not installed in the aidream venv). The Matrix frontend calls the canonical crawler commands on this service directly; AI Dream is not an intermediary. Older dashboard features may still proxy unrelated preview routes via MATRX_SCRAPER_URL.

Quick start (from monorepo root — in VS Code: Terminal → New Terminal, Ctrl+` / Cmd+`):

  1. Start Docker Desktop.
  2. ./scripts/scraper-local.sh up — builds if needed, listens on http://localhost:8001.
  3. In aidream .env: 
    MATRX_SCRAPER_URL=http://localhost:8001
MATRX_SCRAPER_TOKEN=<token>
  4. Run aidream (uv run run.py) + dashboard; refresh the scraper tab.
Command Purpose
./scripts/scraper-local.sh status Container + health probe
./scripts/scraper-local.sh logs Follow logs (Ctrl+C detaches)
./scripts/scraper-local.sh down Stop
./scripts/scraper-local.sh restart Restart after code changes
./scripts/scraper-local.sh rebuild Full image rebuild (slow)

After a machine reboot, run up again. Production: set MATRX_SCRAPER_URL on the aidream container to the deployed scraper-service URL (see DEVELOPER_GUIDE.md).

Dependency posture

Core dependencies are a small set of well-known libraries (httpx, beautifulsoup4, selectolax, markdownify, tldextract, tabulate, python-dotenv) plus matrx-utils. All heavier dependencies (Playwright, PyMuPDF, Tesseract, FastAPI) live behind optional extras so lean installs stay lean.

Documentation

Doc Purpose
DEVELOPER_GUIDE.md Production server setup, API contract, scraper-postgres env vars, retry queue — hand this to external devs
SCHEMA.md Supabase web-crawler schema (scraper.* tables)
MIGRATION_GUIDE.md /api/v1 → /api/scraper API migration
STANDALONE_USAGE.md Embed or run as microservice
matrx_scraper/web_crawl/FEATURE.md Direct command/stream contract and canonical web.* persistence
../../docs/archive/2026/scraper__README.md Old monorepo scraper doc index (archived)

When deployment or API behavior changes, update DEVELOPER_GUIDE.md in the same PR.

Migration notes

This package replaces the legacy root-level scraper/ folder in the aidream monorepo and parts of research/. Internal docs (MIGRATION_STATUS.md, GAPS_TO_FIX.md, LEGACY_AUDIT.md, MIGRATION_GUIDE.md) track what has been ported and what hasn't.

Contributing

See CLAUDE.md for package-specific rules. This package lives in the aidream monorepo at github.com/AI-Matrix-Engine/aidream-current.

License

MIT.

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.2.152)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for armanisadeghi from gravatar.com armanisadeghi

Credits

Author/Maintainer: Matrx

License

MIT License (MIT)

Requires

Python >=3.12

Provides Extra

all browser connect db dedup durable graph metadata ocr pdf server

Tags

crawler html matrx parsing scraping search web-scraping

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

matrx_scraper-0.1.90.tar.gz (459.2 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

matrx_scraper-0.1.90-py3-none-any.whl (373.8 kB view details)

Uploaded Python 3

File details

Details for the file matrx_scraper-0.1.90.tar.gz.

File metadata

  • Download URL: matrx_scraper-0.1.90.tar.gz
  • Upload date:
  • Size: 459.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for matrx_scraper-0.1.90.tar.gz
Algorithm Hash digest
SHA256 83ff29d5c020fa572e5bccd546c7bf4995956a0df6de316bd73b610fabde929f
MD5 b435462446e4351c729fbb2581ecf920
BLAKE2b-256 dc530f9ea779be37f08b4374332637931ea9546e4b93d277ed1485103581ce83

See more details on using hashes here.

Provenance

The following attestation bundles were made for matrx_scraper-0.1.90.tar.gz:

Publisher: publish-package.yml on AI-Matrix-Engine/aidream

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file matrx_scraper-0.1.90-py3-none-any.whl.

File metadata

  • Download URL: matrx_scraper-0.1.90-py3-none-any.whl
  • Upload date:
  • Size: 373.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for matrx_scraper-0.1.90-py3-none-any.whl
Algorithm Hash digest
SHA256 bac6fcb5b65d060f8d2a155a3414bf223a2c439e3f4ee1e909858e35efed92dc
MD5 cf12c8a57b582271d90dc98f06af3897
BLAKE2b-256 cccdc18ef5d543b4c969f5ae98b7b48e801d7c5aa4577a06660bfcb4fd9d4867

See more details on using hashes here.

Provenance

The following attestation bundles were made for matrx_scraper-0.1.90-py3-none-any.whl:

Publisher: publish-package.yml on AI-Matrix-Engine/aidream

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.2.152

2 files

0.2.151

2 files

0.2.150

2 files

0.2.149

2 files

0.2.148

2 files

0.2.147

2 files

0.2.146

2 files

0.2.145

2 files

0.2.144

2 files

0.2.143

2 files

0.2.142

2 files

0.2.141

2 files

0.2.140

2 files

0.2.139

2 files

0.2.138

2 files

0.2.137

2 files

0.2.136

2 files

0.2.135

2 files

0.2.134

2 files

0.2.133

2 files

0.2.132

2 files

0.2.131

2 files

0.2.130

2 files

0.2.129

2 files

0.2.128

2 files

0.2.127

2 files

0.2.126

2 files

0.2.125

2 files

0.2.124

2 files

0.2.123

2 files

0.2.122

2 files

0.2.121

2 files

0.2.120

2 files

0.2.119

2 files

0.2.118

2 files

0.2.117

2 files

0.2.116

2 files

0.2.115

2 files

0.2.114

2 files

0.2.113

2 files

0.2.112

2 files

0.2.111

2 files

0.2.110

2 files

0.2.109

2 files

0.2.108

2 files

0.2.107

2 files

0.2.106

2 files

0.2.105

2 files

0.2.104

2 files

0.2.103

2 files

0.2.102

2 files

0.2.101

2 files

0.2.100

2 files

0.2.99

2 files

0.2.98

2 files

0.2.97

2 files

0.2.96

2 files

0.2.95

2 files

0.2.94

2 files

0.2.93

2 files

0.2.92

2 files

0.2.91

2 files

0.2.90

2 files

0.2.89

2 files

0.2.88

2 files

0.2.87

2 files

0.2.86

2 files

0.2.85

2 files

0.2.84

2 files

0.2.83

2 files

0.2.82

2 files

0.2.81

2 files

0.2.80

2 files

0.2.79

2 files

0.2.78

2 files

0.2.77

2 files

0.2.76

2 files

0.2.75

2 files

0.2.74

2 files

0.2.73

2 files

0.2.72

2 files

0.2.71

2 files

0.2.70

2 files

0.2.69

2 files

0.2.68

2 files

0.2.67

2 files

0.2.66

2 files

0.2.65

2 files

0.2.64

2 files

0.2.63

2 files

0.2.62

2 files

0.2.61

2 files

0.2.60

2 files

0.2.59

2 files

0.2.58

2 files

0.2.57

2 files

0.2.56

2 files

0.2.55

2 files

0.2.54

2 files

0.2.53

2 files

0.2.52

2 files

0.2.51

2 files

0.2.50

2 files

0.2.49

2 files

0.2.48

2 files

0.2.47

2 files

0.2.46

2 files

0.2.45

2 files

0.2.44

2 files

0.2.43

2 files

0.2.42

2 files

0.2.41

2 files

0.2.40

2 files

0.2.39

2 files

0.2.38

2 files

0.2.37

2 files

0.2.36

2 files

0.2.35

2 files

0.2.34

2 files

0.2.33

2 files

0.2.32

2 files

0.2.31

2 files

0.2.30

2 files

0.2.29

2 files

0.2.28

2 files

0.2.27

2 files

0.2.26

2 files

0.2.25

2 files

0.2.24

2 files

0.2.23

2 files

0.2.22

2 files

0.2.21

2 files

0.2.20

2 files

0.2.19

2 files

0.2.18

2 files

0.2.17

2 files

0.2.16

2 files

0.2.15

2 files

0.2.14

2 files

0.2.13

2 files

0.2.12

2 files

0.2.11

2 files

0.2.10

2 files

0.2.9

2 files

0.2.8

2 files

0.2.7

2 files

0.2.6

2 files

0.2.5

2 files

0.2.4

2 files

0.2.3

2 files

0.2.2

2 files

0.2.1

2 files

0.2.0

2 files

0.1.113

2 files

0.1.112

2 files

0.1.111

2 files

0.1.110

2 files

0.1.109

2 files

0.1.108

2 files

0.1.107

2 files

0.1.106

2 files

0.1.105

2 files

0.1.104

2 files

0.1.103

2 files

0.1.102

2 files

0.1.101

2 files

0.1.100

2 files

0.1.99

2 files

0.1.98

2 files

0.1.97

2 files

0.1.96

2 files

0.1.95

2 files

0.1.94

2 files

0.1.93

2 files

0.1.92

2 files

0.1.91

2 files

This release

0.1.90 This release

2 files

0.1.89

2 files

0.1.88

2 files

0.1.87

2 files

0.1.86

2 files

0.1.85

2 files

0.1.84

2 files

0.1.83

2 files

0.1.82

2 files

0.1.81

2 files

0.1.80

2 files

0.1.79

2 files

0.1.78

2 files

0.1.77

2 files

0.1.76

2 files

0.1.75

2 files

0.1.74

2 files

0.1.73

2 files

0.1.72

2 files

0.1.71

2 files

0.1.70

2 files

0.1.69

2 files

0.1.68

2 files

0.1.67

2 files

0.1.66

2 files

0.1.65

2 files

0.1.64

2 files

0.1.63

2 files

0.1.62

2 files

0.1.61

2 files

0.1.60

2 files

0.1.59

2 files

0.1.58

2 files

0.1.57

2 files

0.1.56

2 files

0.1.55

2 files

0.1.54

2 files

0.1.53

2 files

0.1.52

2 files

0.1.51

2 files

0.1.50

2 files

0.1.49

2 files

0.1.48

2 files

0.1.47

2 files

0.1.46

2 files

0.1.45

2 files

0.1.44

2 files

0.1.43

2 files

0.1.42

2 files

0.1.41

2 files

0.1.40

2 files

0.1.39

2 files

0.1.38

2 files

0.1.37

2 files

0.1.36

2 files

0.1.35

2 files

0.1.34

2 files

0.1.33

2 files

0.1.32

2 files

0.1.31

2 files

0.1.30

2 files

0.1.29

2 files

0.1.28

2 files

0.1.27

2 files

0.1.26

2 files

0.1.25

2 files

0.1.24

2 files

0.1.23

2 files

0.1.22

2 files

0.1.21

2 files

0.1.20

2 files

0.1.19

2 files

0.1.18

2 files

0.1.17

2 files

0.1.16

2 files

0.1.15

2 files

0.1.14

2 files

0.1.13

2 files

0.1.12

2 files

0.1.11

2 files

0.1.10

2 files

0.1.9

2 files

0.1.8

2 files

0.1.7

2 files

0.1.6

2 files

0.1.5

2 files

0.1.4

2 files

0.1.3

2 files

0.1.2

2 files

0.1.1

2 files

0.1.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page