commoncrawl-fsspec
An
fsspecfilesystem plugin that turns the Common Crawl web archive into a browseable virtual filesystem.
✨ Features
- Browse the Common Crawl archive like a filesystem — crawls, segments, and WARC files
- Read individual WARC records via HTTP byte-range requests
- Download files to disk with standard
fsspecoperations - Interactive CLI with a rich terminal browser for exploration
📦 Installation
pip install commoncrawl-fsspec
Optional extras
| Extra | Description |
|---|---|
examples |
Interactive CLI (click, rich) |
pip install commoncrawl-fsspec[examples]
🚀 Quick Start
Python API
import fsspec
# Create the filesystem
fs = fsspec.filesystem("cc")
# List top-level directories
fs.ls("/")
# ['/crawls']
# Browse available crawls
crawls = fs.ls("/crawls")
# ['/crawls/CC-MAIN-2026-25', '/crawls/CC-MAIN-2026-21', ...]
# Navigate into a crawl
segments = fs.ls("/crawls/CC-MAIN-2026-25/segments")
# ['/crawls/CC-MAIN-2026-25/segments/1780687572080.85', ...]
# List WARC files in a segment
files = fs.ls("/crawls/CC-MAIN-2026-25/segments/1780687572080.85/warc")
# ['/crawls/CC-MAIN-2026-25/segments/1780687572080.85/warc/CC-MAIN-20260605214811-00000.warc.gz', ...]
# Get file metadata
info = fs.info("/crawls/CC-MAIN-2026-25/segments/1780687572080.85/warc/CC-MAIN-20260605214811-00000.warc.gz")
# {'name': '...', 'type': 'file', 'size': 940246254, 'mtime': ...}
Interactive CLI
# Launch the interactive browser
python -m examples.cc interactive
# List a path
python -m examples.cc ls /crawls
📂 Virtual Filesystem
The plugin exposes a virtual filesystem for navigating the archive:
Archive Browser (/crawls)
Navigate the Common Crawl archive structure:
/
└── /crawls/ → List of crawls (from collinfo.json)
└── /crawls/{crawl-id}/
└── /crawls/{crawl-id}/segments/
└── /crawls/{crawl-id}/segments/{segment-id}/
├── /crawls/.../warc/ → WARC files
├── /crawls/.../wet/ → WET files
└── /crawls/.../wat/ → WAT files
Note: the legacy 2008–2012 crawls (
CC-MAIN-2008-2009,CC-MAIN-2009-2010,CC-MAIN-2012) are listed in collinfo.json but omitted from/crawls, because Common Crawl doesn't currently serve browsable data for them.
📖 API Reference
CommonCrawlFileSystem
| Method | Description |
|---|---|
ls(path, detail=True) |
List directory contents |
info(path) |
Get file/directory metadata |
glob(pattern, detail=True) |
Glob for files matching pattern |
open(path, mode="rb") |
Open a file for reading |
cat_file(path, start, end) |
Read bytes from a file |
get_file(rpath, lpath) |
Download a file to disk |
Constructor Options
| Parameter | Default | Description |
|---|---|---|
cache_ttl |
86400 |
Cache TTL in seconds (24 hours) |
🏗️ Architecture
commoncrawl-fsspec/
├── filesystem.py # CommonCrawlFileSystem (fsspec orchestrator)
├── paths.py # Virtual path parsing and building
├── models.py # Data models (CrawlInfo, WarcFileInfo)
├── caching.py # TTL cache for crawl list
├── clients/
│ ├── http_client.py # HTTP client with retries
│ ├── crawl_index_client.py # Crawl discovery (collinfo.json)
│ ├── s3_listing_client.py # Crawl manifest browsing
│ └── warc_fetcher.py # WARC record byte-range fetching
🧪 Development
# Install dependencies
uv sync
# Run tests
uv run pytest
# Type checking
uv run mypy src
# Linting
uv run ruff check src
📝 License
MIT
Metadata
Release files for commoncrawl-fsspec 0.1.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| commoncrawl_fsspec-0.1.2.tar.gz | 8.8 kB | With provenance | Details for commoncrawl_fsspec-0.1.2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| commoncrawl_fsspec-0.1.2-py3-none-any.whl | Python 3 | none | any | Details for commoncrawl_fsspec-0.1.2-py3-none-any.whl |
Total release size: 22.3 kB
Release files / commoncrawl_fsspec-0.1.2.tar.gz
| Download URL | commoncrawl_fsspec-0.1.2.tar.gz |
|---|---|
| Size | 8.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
8a48ae566b7c58221999296546a6a28962b586c8249a0138b66602ebe4a99496
|
|
BLAKE2b-256 checksum How to use checksums |
fad3fd1ff6cc123074ec957367ddffd8be02d0e02854a1b933a2eb145de4940b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 1, 2026.
Transparency logRelease files / commoncrawl_fsspec-0.1.2-py3-none-any.whl
| Download URL | commoncrawl_fsspec-0.1.2-py3-none-any.whl |
|---|---|
| Size | 13.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
84a9bc48abb7d81d393e59b3cbd992e23aa5515c743e1e0a6e336b6f4e1b88a6
|
|
BLAKE2b-256 checksum How to use checksums |
4147c1f15ad81b8032c23ae1a71824966faa1a35e2b4f2d2f681169d0429b78f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 1, 2026.
Transparency logSecurity for commoncrawl-fsspec 0.1.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for commoncrawl_fsspec-0.1.2.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 1, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.