matrx-scraper
Web scraping, HTML parsing, site crawling, headless-browser automation, and search for Python. An 8-stage parser turns raw HTML into clean AI-ready content plus structured extractions (tables, code blocks, categorized links, metadata, schema.org). It runs standalone with no database, and scales up to a full FastAPI microservice with a durable crash-safe crawl frontier.
Install
pip install matrx-scraper # core: HTTP fetch + parse + crawl + Brave Search
pip install "matrx-scraper[browser]" # + Playwright / curl_cffi for JS-rendered pages
pip install "matrx-scraper[metadata]" # + extruct (JSON-LD, microdata, RDFa, OpenGraph)
pip install "matrx-scraper[dedup]" # + MinHash / SimHash near-duplicate detection
pip install "matrx-scraper[pdf]" # + PyMuPDF
pip install "matrx-scraper[ocr]" # + Tesseract
pip install "matrx-scraper[db]" # + matrx-orm persistence
pip install "matrx-scraper[durable]" # + matrx-runtime crash-safe crawl frontier
pip install "matrx-scraper[server]" # + FastAPI microservice (includes browser, db, metadata, durable)
pip install "matrx-scraper[all]" # everything
Python 3.12+. Hard deps are a small set of well-known libraries (httpx,
beautifulsoup4, selectolax, markdownify, tldextract, tabulate,
croniter, python-dotenv) plus matrx-utils and matrx-files. Everything
heavy sits behind an extra so lean installs stay lean.
Quickstart
from matrx_scraper import scrape, scrape_many_stream, parse_html, audit_html
result = await scrape("https://example.com/article")
result.success # bool — outcome
result.failure_reason # str | None
result.title
result.ai_content # clean, AI-ready markdown
result.links # links by category
result.tables # parsed tables
result.organized_data # structured JSON of the page
# Many URLs — yields each page as it finishes, never a buffered batch.
async for page in scrape_many_stream(urls, concurrency=5):
print(page.url, page.success)
parsed = parse_html(open("page.html").read()) # no network
seo = audit_html(html, url) # full page evidence in one parse
Crawl a site
from matrx_scraper import crawl_site
async for page in crawl_site("https://example.com", max_pages=100):
print(page.url, page.title)
Drive a browser
from matrx_scraper.ai_browser import navigate, type_text, wait_for, screenshot, close_session
nav = await navigate("https://example.com/login", extract_text=True)
await type_text(nav.session_id, "input[name=email]", "user@example.com")
await wait_for(nav.session_id, selector="[data-ready]", timeout_ms=15_000)
shot = await screenshot(nav.session_id, full_page=True)
await close_session(nav.session_id)
Every action returns a Pydantic result with success: bool plus
error_type / error_message, so an agent loop recovers from timeouts and
selector misses without exception handling.
Other clients
BraveSearchClient (search), PsiClient (PageSpeed Insights v5), GscClient
(Google Search Console), compute_link_scores (PageRank over a link graph),
CustomExtractor (per-host CSS/XPath/regex/JSON-LD rules), quick_preview
(robots + homepage audit + screenshot). All pure — they return typed values and
persist nothing.
AI tools and MCP
matrx_scraper.ai_tools.ALL_TOOLS is 22 ToolSpec descriptors (14 browser, 4
scrape, 4 crawl), each with a full JSON Schema and an async handler — drop them
straight into an OpenAI/Anthropic tool call or any agent registry.
The bundled MCP server exposes the same list:
python -m matrx_scraper.mcp # stdio (canonical MCP transport)
python -m matrx_scraper.mcp --list # print registered tools and exit
python -m matrx_scraper.mcp --groups browser,scrape
stdio is the only transport. The MCP server has no network listener by design — one would hand a JS-executing browser to anyone who can reach the port. Network callers use the authenticated microservice below.
Running the microservice
# Docker (build context is the monorepo root, so sibling packages resolve)
cd packages/matrx-scraper && cp .env.example .env && docker compose up -d
# Or directly
pip install "matrx-scraper[server]" && playwright install chromium
matrx-scraper --port 8000
| Variable | Required | Purpose |
|---|---|---|
SUPABASE_MATRIX_{HOST,PORT,DATABASE_NAME,USER,PASSWORD} |
yes | The one connection, required. Both scraper.* (cache / domain config / retry queue) and the canonical web.* crawler bind here. Point these five at your own Postgres to run the scraper on its own database; there is never a second variable and never a fallback chain. |
SUPABASE_MATRIX_URL |
yes | Project URL; also derives the JWT JWKS URL |
SUPABASE_JWT_SECRET |
yes | HS256 compatibility (ES256 verifies via JWKS) |
DATACENTER_PROXIES |
for proxied fetches | Comma-separated pool. Missing or exhausted fails loudly — never a silent direct request that exposes the host IP. |
ADMIN_API_TOKEN |
for server-to-server | Shared secret for the approved-server identity |
BRAVE_SEARCH_API_KEY_PRO_AI |
for search | The one platform Brave key; explicit api_key injection supports user-supplied keys. |
AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_REGION / AWS_S3_DEFAULT_BUCKET |
for the canonical crawler | Artifact bytes via matrx-files |
PORT / WORKERS / LOG_LEVEL |
no | Defaults 8000 / 1 / info |
Everything mounts under /api/scraper/: quick-scrape · batch · search ·
search-and-scrape · browser-fetch · browser/* (16 session endpoints) ·
preview · content/save · queue/* · config/domains · crawler/*.
GET /health (reports the installed version) and GET /health/ready are public;
everything else takes Authorization: Bearer <supabase-jwt>.
Embed it in another FastAPI app instead:
from matrx_scraper.server import create_app, ServerConfig
app = create_app(ServerConfig.from_env()) # DB comes from SUPABASE_MATRIX_*, never a config URL
Extending it in a host
The crawler takes four injection points; defaults are in-memory / no-op / discard, so nothing is required to get started:
| Protocol | Default | Swap in |
|---|---|---|
CrawlEventSink |
NoopEventSink |
your durable event writer |
QueueBackend |
InMemoryQueueBackend |
the [durable] runtime frontier — survives restarts |
BodyPersister |
discard | your storage |
RecipeBackend |
StaticRecipeBackend(DEFAULT_RECIPES) |
your per-host Playwright playbooks |
Host objects (cache, domain config, browser pool, file manager, work-queue
factory) are handed over with matrx_scraper.configure_ext(...); DB models bind
to an existing matrx-orm pool with matrx_scraper.configure_db("<db name>").
Documentation
| Doc | For |
|---|---|
matrx_scraper/FEATURE.md |
Engine contract: SSRF gates, proxy rules, browser runtimes, tool surface, auth, persistence |
matrx_scraper/web_crawl/FEATURE.md |
The canonical site crawler |
CLAUDE.md |
Contributing to the package |
License
MIT. Developed in the aidream monorepo.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file matrx_scraper-0.2.120.tar.gz.
File metadata
- Download URL: matrx_scraper-0.2.120.tar.gz
- Upload date:
- Size: 1.2 MB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
a3c891e2f4130f334a4e6cd287b329c3c55b7ed753164da68b86b36899450837
|
|
| MD5 |
e6c1fe9d333001f5b6cb0dfaaf8bdb76
|
|
| BLAKE2b-256 |
a23823e0b73a7e99ae0465eac060626ad715ec2c791fc92591f72f923957ec89
|
Provenance
The following attestation bundles were made for matrx_scraper-0.2.120.tar.gz:
Publisher:
publish-package.yml on AI-Matrix-Engine/aidream
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
matrx_scraper-0.2.120.tar.gz -
Subject digest:
a3c891e2f4130f334a4e6cd287b329c3c55b7ed753164da68b86b36899450837 - Sigstore transparency entry: 2599351195
- Sigstore integration time:
-
Permalink:
AI-Matrix-Engine/aidream@5f2c7c75f648ddc6ad4fbb1cf72301e9a1332fca -
Branch / Tag:
refs/tags/matrx-scraper/v0.2.120 - Owner: https://github.com/AI-Matrix-Engine
-
Access:
private
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
publish-package.yml@5f2c7c75f648ddc6ad4fbb1cf72301e9a1332fca -
Trigger Event:
push
-
Statement type:
File details
Details for the file matrx_scraper-0.2.120-py3-none-any.whl.
File metadata
- Download URL: matrx_scraper-0.2.120-py3-none-any.whl
- Upload date:
- Size: 833.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
b4a0a8af4c9a214342e4096a6bd22f732a949b921c3ad88c47ea38fa3bc7e110
|
|
| MD5 |
3d8f5a5b699f4a68621eb221cbc40bf9
|
|
| BLAKE2b-256 |
0cb003a24f58b1378ffb6692b88055e8badd2be61ce429a3f36126ef62c90246
|
Provenance
The following attestation bundles were made for matrx_scraper-0.2.120-py3-none-any.whl:
Publisher:
publish-package.yml on AI-Matrix-Engine/aidream
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
matrx_scraper-0.2.120-py3-none-any.whl -
Subject digest:
b4a0a8af4c9a214342e4096a6bd22f732a949b921c3ad88c47ea38fa3bc7e110 - Sigstore transparency entry: 2599351655
- Sigstore integration time:
-
Permalink:
AI-Matrix-Engine/aidream@5f2c7c75f648ddc6ad4fbb1cf72301e9a1332fca -
Branch / Tag:
refs/tags/matrx-scraper/v0.2.120 - Owner: https://github.com/AI-Matrix-Engine
-
Access:
private
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
publish-package.yml@5f2c7c75f648ddc6ad4fbb1cf72301e9a1332fca -
Trigger Event:
push
-
Statement type:
