Meshagent Common Crawl
Import Common Crawl captures into a Meshagent room dataset.
from meshagent.commoncrawl import import_domain_from_commoncrawl
result = await import_domain_from_commoncrawl(
room,
index="CC-MAIN-2025-08",
domain="example.com",
table="pages",
url_filter=r"https?://(www\.)?example\.com/docs/.*",
)
To test it through meshagent room connect:
meshagent room connect --room=my-room --identity=commoncrawl -- \
python meshagent-sdk/meshagent-commoncrawl/examples/crawl.py \
http://www.meshagent.com --table=sample --namespace=crawls --limit=10
The example defaults to --scope=host, so https://www.example.com imports
only captures from www.example.com. Use --scope=domain when you explicitly
want sibling subdomains too, for example when a large site stores useful content
outside www.
The sample command writes progress to stderr while it imports. TTY output uses a
single updating line; redirected output uses plain log lines. Pass --silent to
suppress progress output. Columnar scans emit periodic heartbeat updates while
waiting for DataFusion batches. WARC reads run concurrently by default and report
queued records, downloaded bytes, and request counts; use --scan-partitions to
tune DataFusion scan parallelism and --concurrency, --warc-retries, and
--warc-retry-delay to tune object reads.
The importer uses Common Crawl's columnar index by default through DataFusion.
Basic imports generate a SQL query that selects one latest HTML capture per URL
from the requested host or domain, excluding robots.txt. Advanced callers can
pass columnar_sql= to control the URL selection directly; the query must return
url plus WARC pointer columns (filename/offset/length or the Common Crawl
names warc_filename/warc_record_offset/warc_record_length). The example CLI
exposes this as --sql.
Common Crawl's CDX API is rate limited and not a good fit for broad filtering.
The SDK still contains the polite CDX reader for compatibility, using
https://index.commoncrawl.org, a Meshagent User-Agent, serialized/paced
requests, and clearer HTTP 503 guidance.
By default, records are merged on url with the columns url, date,
content_type, and text. Pass an async extract= callback to derive custom
columns from the WARC record and decoded content bytes. Return None from the
callback to skip the record. Pass an async progress= callback to observe import
progress from library code.
1 maintainer
Metadata
1 maintainer
Release files for meshagent-commoncrawl 0.54.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| meshagent_commoncrawl-0.54.0.tar.gz | 22.6 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| meshagent_commoncrawl-0.54.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 39.5 kB
Release files / meshagent_commoncrawl-0.54.0.tar.gz
| Download URL | meshagent_commoncrawl-0.54.0.tar.gz |
|---|---|
| Size | 22.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
a98ed4b1fc14439360bd8aa970508a383a35d7850e65689e497b007bbe6989a2
|
|
BLAKE2b-256 checksum How to use checksums |
5ef0ff89c58936871448a79a76ce03c886b20134b3747266128650137b09337e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / meshagent_commoncrawl-0.54.0-py3-none-any.whl
| Download URL | meshagent_commoncrawl-0.54.0-py3-none-any.whl |
|---|---|
| Size | 17.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e67b32cbdfbc1b8e59998ad205504611b9d2a0d2a0c7b648a4e611bd3fcd6309
|
|
BLAKE2b-256 checksum How to use checksums |
2efb24ac20719046dcb47a36b4b79679c74aab1e22861552013b18310412df4b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Security for meshagent-commoncrawl 0.54.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
