cc-downloader

Python bindings for cc-downloader, a polite and user-friendly client for downloading Common Crawl data.

The library is implemented in Rust and exposes a synchronous Python API. All downloads run inside a multi-threaded Tokio runtime, so concurrent file fetching works exactly as in the CLI — the threads parameter controls how many files are downloaded in parallel.

Requirements

• Python ≥ 3.12

Installation

pip install cc-downloader

uv add cc-downloader

Quick start

The workflow has two steps: fetch a paths index for a crawl snapshot, then download every file it lists.

import cc_downloader

# Step 1: download the paths index for WET files.
cc_downloader.download_paths(
    snapshot="CC-MAIN-2024-46",
    data_type="wet",
    dst="./output",
)

# Step 2: download every file listed in the index.
cc_downloader.download(
    paths="./output/wet.paths.gz",
    dst="./output",
    threads=10,
    progress=True,
)

API reference

download_paths

cc_downloader.download_paths(
    snapshot: str,
    data_type: str,
    dst: str,
    subsets: list[str] | None = None,
    max_retries: int = 1000,
) -> None

Downloads the gzip-compressed paths index file for a given crawl snapshot and data type to dst.

Example — cc-index-table with subset filtering:

cc_downloader.download_paths(
    snapshot="CC-MAIN-2024-46",
    data_type="cc-index-table",
    dst="./output",
    subsets=["warc", "robotstxt"],
)

download_contrib_paths

cc_downloader.download_contrib_paths(
    url: str,
    dst: str,
    max_retries: int = 1000,
) -> None

Downloads a paths index file directly from a contributor dataset URL.

Example:

cc_downloader.download_contrib_paths(
    url="https://data.commoncrawl.org/contrib/my-dataset/paths.gz",
    dst="./output",
)

download

cc_downloader.download(
    paths: str,
    dst: str,
    threads: int = 10,
    max_retries: int = 1000,
    numbered: bool = False,
    files_only: bool = False,
    progress: bool = False,
) -> None

Downloads every file listed in a .paths.gz index. Files are fetched concurrently up to threads at a time, with exponential-backoff retries on transient failures.

Example:

cc_downloader.download(
    paths="./output/wet.paths.gz",
    dst="./output",
    threads=10,
    progress=True,
)

Politeness and retries

Every HTTP request uses an exponential-backoff retry policy (bounds: 1 s – 1 h, bounded jitter, base 2). The default of 1 000 retries is intentionally high: Common Crawl is a shared public resource and transient failures are common under load. This tool is intended for use outside of AWS. You can monitor Common Crawl infrastructure traffic on the Infrastructure Status Webpage.

Avoid setting threads much higher than the default of 10. Too many concurrent requests in a short period can trigger 403 errors, which are unrecoverable and cannot be retried.

Contributing

Contributions are welcome. Please see CONTRIBUTING.md for guidelines.

License

Licensed under either of Apache License, Version 2.0 or MIT license at your option.