Meshagent Common Crawl
Import Common Crawl captures into a Meshagent room dataset.
from meshagent.commoncrawl import import_domain_from_commoncrawl
result = await import_domain_from_commoncrawl(
room,
index="CC-MAIN-2025-08",
domain="example.com",
table="pages",
url_filter=r"https?://(www\.)?example\.com/docs/.*",
)
To test it through meshagent room connect:
meshagent room connect --room=my-room --identity=commoncrawl -- \
python meshagent-sdk/meshagent-commoncrawl/examples/crawl.py \
http://www.meshagent.com --table=sample --namespace=crawls --limit=10
The example defaults to --scope=host, so https://www.example.com imports
only captures from www.example.com. Use --scope=domain when you explicitly
want sibling subdomains too, for example when a large site stores useful content
outside www.
The sample command writes progress to stderr while it imports. TTY output uses a
single updating line; redirected output uses plain log lines. Pass --silent to
suppress progress output. Columnar scans emit periodic heartbeat updates while
waiting for DataFusion batches. WARC reads run concurrently by default and report
queued records, downloaded bytes, and request counts; use --scan-partitions to
tune DataFusion scan parallelism and --concurrency, --warc-retries, and
--warc-retry-delay to tune object reads.
The importer uses Common Crawl's columnar index by default through DataFusion.
Basic imports generate a SQL query that selects one latest HTML capture per URL
from the requested host or domain, excluding robots.txt. Advanced callers can
pass columnar_sql= to control the URL selection directly; the query must return
url plus WARC pointer columns (filename/offset/length or the Common Crawl
names warc_filename/warc_record_offset/warc_record_length). The example CLI
exposes this as --sql.
Common Crawl's CDX API is rate limited and not a good fit for broad filtering.
The SDK still contains the polite CDX reader for compatibility, using
https://index.commoncrawl.org, a Meshagent User-Agent, serialized/paced
requests, and clearer HTTP 503 guidance.
By default, records are merged on url with the columns url, date,
content_type, and text. Pass an async extract= callback to derive custom
columns from the WARC record and decoded content bytes. Return None from the
callback to skip the record. Pass an async progress= callback to observe import
progress from library code.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file meshagent_commoncrawl-0.49.0.tar.gz.
File metadata
- Download URL: meshagent_commoncrawl-0.49.0.tar.gz
- Upload date:
- Size: 22.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
7464218267af27c7a371b7a2a392077411e815d91a7ebb84d07f4685d9c3b136
|
|
| MD5 |
df0eb8d6984445e7b0273211a9f87045
|
|
| BLAKE2b-256 |
b000692558c5cbba09ec6c26ccfdc5bb42f5a13204630124e7742ca8854d02bd
|
File details
Details for the file meshagent_commoncrawl-0.49.0-py3-none-any.whl.
File metadata
- Download URL: meshagent_commoncrawl-0.49.0-py3-none-any.whl
- Upload date:
- Size: 17.0 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
462ab0be57f501138accc4142a4a32f647cd8c922d4f9c321f31be787cbafc8c
|
|
| MD5 |
079ef8c0101feaaa1b499611903885ff
|
|
| BLAKE2b-256 |
32dde6783e5cb1bc22c6516e00c10ae4921606df33291fb88492745b95a4f9bb
|
