matrx-scraper
Web scraping, HTML parsing, site crawling, headless-browser automation, and search for Python. An 8-stage parser turns raw HTML into clean AI-ready content plus structured extractions (tables, code blocks, categorized links, metadata, schema.org). It runs standalone with no database, and scales up to a full FastAPI microservice with a durable crash-safe crawl frontier.
Install
pip install matrx-scraper # core: HTTP fetch + parse + crawl + Brave Search
pip install "matrx-scraper[browser]" # + Playwright / curl_cffi for JS-rendered pages
pip install "matrx-scraper[metadata]" # + extruct (JSON-LD, microdata, RDFa, OpenGraph)
pip install "matrx-scraper[dedup]" # + MinHash / SimHash near-duplicate detection
pip install "matrx-scraper[pdf]" # + PyMuPDF
pip install "matrx-scraper[ocr]" # + Tesseract
pip install "matrx-scraper[db]" # + matrx-orm persistence
pip install "matrx-scraper[durable]" # + matrx-runtime crash-safe crawl frontier
pip install "matrx-scraper[server]" # + FastAPI microservice (includes browser, db, metadata, durable)
pip install "matrx-scraper[all]" # everything
Python 3.12+. Hard deps are a small set of well-known libraries (httpx,
beautifulsoup4, selectolax, markdownify, tldextract, tabulate,
croniter, python-dotenv) plus matrx-utils and matrx-files. Everything
heavy sits behind an extra so lean installs stay lean.
Quickstart
from matrx_scraper import scrape, scrape_many_stream, parse_html, audit_html
result = await scrape("https://example.com/article")
result.success # bool — outcome
result.failure_reason # str | None
result.title
result.ai_content # clean, AI-ready markdown
result.links # links by category
result.tables # parsed tables
result.organized_data # structured JSON of the page
# Many URLs — yields each page as it finishes, never a buffered batch.
async for page in scrape_many_stream(urls, concurrency=5):
print(page.url, page.success)
parsed = parse_html(open("page.html").read()) # no network
seo = audit_html(html, url) # full page evidence in one parse
Crawl a site
from matrx_scraper import crawl_site
async for page in crawl_site("https://example.com", max_pages=100):
print(page.url, page.title)
Drive a browser
from matrx_scraper.ai_browser import navigate, type_text, wait_for, screenshot, close_session
nav = await navigate("https://example.com/login", extract_text=True)
await type_text(nav.session_id, "input[name=email]", "user@example.com")
await wait_for(nav.session_id, selector="[data-ready]", timeout_ms=15_000)
shot = await screenshot(nav.session_id, full_page=True)
await close_session(nav.session_id)
Every action returns a Pydantic result with success: bool plus
error_type / error_message, so an agent loop recovers from timeouts and
selector misses without exception handling.
Other clients
BraveSearchClient (search), PsiClient (PageSpeed Insights v5), GscClient
(Google Search Console), compute_link_scores (PageRank over a link graph),
CustomExtractor (per-host CSS/XPath/regex/JSON-LD rules), quick_preview
(robots + homepage audit + screenshot). All pure — they return typed values and
persist nothing.
AI tools and MCP
matrx_scraper.ai_tools.ALL_TOOLS is 22 ToolSpec descriptors (14 browser, 4
scrape, 4 crawl), each with a full JSON Schema and an async handler — drop them
straight into an OpenAI/Anthropic tool call or any agent registry.
The bundled MCP server exposes the same list:
python -m matrx_scraper.mcp # stdio (canonical MCP transport)
python -m matrx_scraper.mcp --list # print registered tools and exit
python -m matrx_scraper.mcp --groups browser,scrape
stdio is the only transport. The MCP server has no network listener by design — one would hand a JS-executing browser to anyone who can reach the port. Network callers use the authenticated microservice below.
Running the microservice
# Docker (build context is the monorepo root, so sibling packages resolve)
cd packages/matrx-scraper && cp .env.example .env && docker compose up -d
# Or directly
pip install "matrx-scraper[server]" && playwright install chromium
matrx-scraper --port 8000
| Variable | Required | Purpose |
|---|---|---|
SUPABASE_MATRIX_{HOST,PORT,DATABASE_NAME,USER,PASSWORD} |
yes | The one connection, required. Both scraper.* (cache / domain config / retry queue) and the canonical web.* crawler bind here. Point these five at your own Postgres to run the scraper on its own database; there is never a second variable and never a fallback chain. |
SUPABASE_MATRIX_URL |
yes | Project URL; also derives the JWT JWKS URL |
SUPABASE_JWT_SECRET |
yes | HS256 compatibility (ES256 verifies via JWKS) |
DATACENTER_PROXIES |
for proxied fetches | Comma-separated pool. Missing or exhausted fails loudly — never a silent direct request that exposes the host IP. |
ADMIN_API_TOKEN |
for server-to-server | Shared secret for the approved-server identity |
BRAVE_SEARCH_API_KEY_PRO_AI |
for search | The one platform Brave key; explicit api_key injection supports user-supplied keys. |
AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_REGION / AWS_S3_DEFAULT_BUCKET |
for the canonical crawler | Artifact bytes via matrx-files |
PORT / WORKERS / LOG_LEVEL |
no | Defaults 8000 / 1 / info |
Everything mounts under /api/scraper/: quick-scrape · batch · search ·
search-and-scrape · browser-fetch · browser/* (16 session endpoints) ·
preview · content/save · queue/* · config/domains · crawler/*.
GET /health (reports the installed version) and GET /health/ready are public;
everything else takes Authorization: Bearer <supabase-jwt>.
Embed it in another FastAPI app instead:
from matrx_scraper.server import create_app, ServerConfig
app = create_app(ServerConfig.from_env()) # DB comes from SUPABASE_MATRIX_*, never a config URL
Extending it in a host
The crawler takes four injection points; defaults are in-memory / no-op / discard, so nothing is required to get started:
| Protocol | Default | Swap in |
|---|---|---|
CrawlEventSink |
NoopEventSink |
your durable event writer |
QueueBackend |
InMemoryQueueBackend |
the [durable] runtime frontier — survives restarts |
BodyPersister |
discard | your storage |
RecipeBackend |
StaticRecipeBackend(DEFAULT_RECIPES) |
your per-host Playwright playbooks |
Host objects (cache, domain config, browser pool, file manager, work-queue
factory) are handed over with matrx_scraper.configure_ext(...); DB models bind
to an existing matrx-orm pool with matrx_scraper.configure_db("<db name>").
Documentation
| Doc | For |
|---|---|
matrx_scraper/FEATURE.md |
Engine contract: SSRF gates, proxy rules, browser runtimes, tool surface, auth, persistence |
matrx_scraper/web_crawl/FEATURE.md |
The canonical site crawler |
CLAUDE.md |
Contributing to the package |
License
MIT. Developed in the aidream monorepo.
1 maintainer
Release files for matrx-scraper 0.2.225
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| matrx_scraper-0.2.225.tar.gz | 1.3 MB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| matrx_scraper-0.2.225-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 2.1 MB
Release files / matrx_scraper-0.2.225.tar.gz
| Download URL | matrx_scraper-0.2.225.tar.gz |
|---|---|
| Size | 1.3 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
af983fda0f29aa98963d6a4f41e10d8e03146efdb1207dda28d84be57e4b347b
|
|
BLAKE2b-256 checksum How to use checksums |
e9c5e3a95002d7913686e2b1e4e41287a2177cfa7284f823be6600ee22aa8ef9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 17, 2026.
Transparency logRelease files / matrx_scraper-0.2.225-py3-none-any.whl
| Download URL | matrx_scraper-0.2.225-py3-none-any.whl |
|---|---|
| Size | 865.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b3132a1639df1125e959a96b7965b3acfed6b4b0ac3c4373e4bd73b9717b03b4
|
|
BLAKE2b-256 checksum How to use checksums |
9511e9c5fc0a9e9ac27125658f336306aa4bfe937ca85eb1f3bdca413f333223
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 17, 2026.
Transparency log
