Meshagent Scrapy
Spider a website with Scrapy and import page content into a Meshagent room dataset.
from meshagent.scrapy import import_domain_with_scrapy
result = await import_domain_with_scrapy(
room,
url="https://example.com",
table="pages",
namespace=["crawls"],
limit=100,
concurrency=5,
)
To test it through meshagent room connect:
meshagent room connect --room=my-room --identity=scrapy -- \
python meshagent-sdk/meshagent-scrapy/examples/crawl.py \
https://www.meshagent.com --table=sample --namespace=crawls --limit=100 --concurrency=5
The sample command writes progress to stderr while it imports. TTY output uses a
single updating line; redirected output uses plain log lines. Pass --silent to
suppress progress output.
Pass --concurrency or concurrency= to tune Scrapy's maximum concurrent
requests.
Pass --batch-size or batch_size= to cap how many page records are merged
into the content table at once. The crawler also flushes content batches by
estimated payload size with --max-batch-bytes or max_batch_bytes=, which
defaults to 16 MiB, and by elapsed time with --max-batch-delay or
max_batch_delay=, which defaults to 60 seconds. The row-count cap defaults to
100. Raw HTML rows can be large, so prefer lowering the byte limit before
lowering the row count if the room server reports Lance/DataFusion merge memory
exhaustion while importing full HTML pages.
The crawler sends a browser-like User-Agent by default. Pass --user-agent or
user_agent= to override it for a specific crawl.
Pass --include-sitemap or include_sitemap=True to seed the crawl with URLs
from Sitemap: entries in /robots.txt and from /sitemap.xml. Sitemap URLs
are filtered to the same domain and by url_filter before the crawler requests
them; normal page link discovery remains enabled.
The default extractor writes page content as markdown in the text column. Use
--format=html to keep HTML, --format=text to strip markup to plain text, or
pass content_format= from library code.
By default, the crawler runs Trafilatura cleanup before converting markdown/text
content, which strips common navigation, footer, sidebar, and ad boilerplate.
For --format=html, the default is to strip scripts and inline image data URLs
while preserving the rest of the HTML. Use --strip= with comma-separated values
like scripts, css, whitespace, image-data-urls, or clean to choose the
HTML stripping steps, or --strip=none to process the raw response body.
The CLI persists crawl frontier state by default in <table>__frontier, so a
limited run can be resumed by running the same command again:
meshagent room connect --room=my-room --identity=scrapy -- \
python meshagent-sdk/meshagent-scrapy/examples/crawl.py \
https://www.meshagent.com --table=sample --namespace=crawls --limit=100
Pass --frontier-table to choose a different state table, or --no-resume to
run without frontier persistence. Library callers can opt in with
resume=True. Frontier updates are buffered before they are written; tune that
with --frontier-batch-size or the library frontier_batch_size= argument.
Failed URLs are not retried on resume unless you pass --retry-failed or
retry_failed=True.
The crawler creates indexes by default: a BTREE index on the page table primary
key, plus BTREE url and BITMAP status indexes on the frontier table. Pass
--index=text or index_columns=("text",) to also create an INVERTED index on
text. Pass --no-indexes or create_indexes=False to skip all automatic
index creation. It also runs dataset optimization periodically while importing
and shows optimizing/optimized in progress output. Tune that with
--optimize-every or optimize_every=, and use 0 on the CLI or None in
library code to disable automatic optimization.
By default, the crawler imports textual responses only, based on Content-Type
values containing text/, html, xml, or json. Pass --response-filter
or response_filter= to replace that default with a JMESPath expression over
url, status, headers, content_type, and content_type_lower. Header
names are lower-cased, so an HTML-only crawl can use:
--response-filter "contains(headers.\"content-type\", 'text/html')"
By default, records are merged on url with the columns url, date,
content_type, text, and images. text is markdown unless another content
format is selected. images is a struct array with src and alt only, and
inline image data URLs are excluded.
Pass an async extract= callback to derive custom columns from the Scrapy
response and content bytes. Return None from the callback to skip the record.
Pass an async progress= callback to observe import progress from library code.
1 maintainer
Metadata
1 maintainer
Release files for meshagent-scrapy 0.53.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| meshagent_scrapy-0.53.0.tar.gz | 31.0 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| meshagent_scrapy-0.53.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 53.2 kB
Release files / meshagent_scrapy-0.53.0.tar.gz
| Download URL | meshagent_scrapy-0.53.0.tar.gz |
|---|---|
| Size | 31.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
f2f4288ee72b24331406775f3c629fd7e9d09a80a68e4a7ca303f5cffbadd05e
|
|
BLAKE2b-256 checksum How to use checksums |
308e52b10075d535e9999dce293054242f6ab79e718fac2d1869d67f416de884
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / meshagent_scrapy-0.53.0-py3-none-any.whl
| Download URL | meshagent_scrapy-0.53.0-py3-none-any.whl |
|---|---|
| Size | 22.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
84b862bce6ca737d287569e005c0418c496d79a3807110e2b2ca41e8a53be103
|
|
BLAKE2b-256 checksum How to use checksums |
1b6a23949694ee14213e657cf9c5f3e77d6bec46c8c78550da06be6bf2dd008b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
