cc-downloader

Python bindings for cc-downloader, a polite and user-friendly client for downloading Common Crawl data.

The library is implemented in Rust and exposes a synchronous Python API. All downloads run inside a multi-threaded Tokio runtime, so concurrent file fetching works exactly as in the CLI — the threads parameter controls how many files are downloaded in parallel.

Requirements

Python ≥ 3.12

Installation

pip install cc-downloader

uv add cc-downloader

Quick start

The workflow has two steps: fetch a paths index for a crawl snapshot, then download every file it lists.

import cc_downloader

# Step 1: download the paths index for WET files.
cc_downloader.download_paths(
    snapshot="CC-MAIN-2024-46",
    data_type="wet",
    dst="./output",
)

# Step 2: download every file listed in the index.
cc_downloader.download(
    paths="./output/wet.paths.gz",
    dst="./output",
    threads=10,
    progress=True,
)

API reference

`download_paths`

cc_downloader.download_paths(
    snapshot: str,
    data_type: str,
    dst: str,
    subsets: list[str] | None = None,
    max_retries: int = 1000,
) -> None

Downloads the gzip-compressed paths index file for a given crawl snapshot and data type to dst.

Parameter	Description
`snapshot`	Crawl identifier, e.g. `"CC-MAIN-2024-46"` or `"CC-NEWS-2025-01"`
`data_type`	Data type: `"warc"`, `"wet"`, `"wat"`, `"robotstxt"`, `"non200responses"`, `"cc-index"`, `"cc-index-table"`
`dst`	Destination directory
`subsets`	For `cc-index-table` only: list of subsets to keep. Valid values: `"crawldiagnostics"`, `"robotstxt"`, `"warc"`. Defaults to all three.
`max_retries`	Maximum retry attempts per request (default `1000`)

Example — cc-index-table with subset filtering:

cc_downloader.download_paths(
    snapshot="CC-MAIN-2024-46",
    data_type="cc-index-table",
    dst="./output",
    subsets=["warc", "robotstxt"],
)

`download_contrib_paths`

cc_downloader.download_contrib_paths(
    url: str,
    dst: str,
    max_retries: int = 1000,
) -> None

Downloads a paths index file directly from a contributor dataset URL.

Parameter	Description
`url`	Direct URL to the paths file. Must start with `https://data.commoncrawl.org/contrib/`
`dst`	Destination directory
`max_retries`	Maximum retry attempts per request (default `1000`)

Example:

cc_downloader.download_contrib_paths(
    url="https://data.commoncrawl.org/contrib/my-dataset/paths.gz",
    dst="./output",
)

`download`

cc_downloader.download(
    paths: str,
    dst: str,
    threads: int = 10,
    max_retries: int = 1000,
    numbered: bool = False,
    files_only: bool = False,
    progress: bool = False,
) -> None

Downloads every file listed in a .paths.gz index. Files are fetched concurrently up to threads at a time, with exponential-backoff retries on transient failures.

Parameter	Description
`paths`	Path to a `.paths.gz` index file
`dst`	Destination directory
`threads`	Maximum concurrent downloads (default `10`)
`max_retries`	Maximum retry attempts per file (default `1000`)
`numbered`	Rename output files sequentially (`0.txt.gz`, `1.txt.gz`, …). Only for WET files. Mutually exclusive with `files_only`.
`files_only`	Write files flat into `dst` with no subdirectory structure. Only for WARC/WET/WAT files. Mutually exclusive with `numbered`.
`progress`	Show a per-file progress bar

Example:

cc_downloader.download(
    paths="./output/wet.paths.gz",
    dst="./output",
    threads=10,
    progress=True,
)

Politeness and retries

Every HTTP request uses an exponential-backoff retry policy (bounds: 1 s – 1 h, bounded jitter, base 2). The default of 1 000 retries is intentionally high: Common Crawl is a shared public resource and transient failures are common under load. This tool is intended for use outside of AWS. You can monitor Common Crawl infrastructure traffic on the Infrastructure Status Webpage.

Avoid setting threads much higher than the default of 10. Too many concurrent requests in a short period can trigger 403 errors, which are unrecoverable and cannot be retried.

Contributing

Contributions are welcome. Please see CONTRIBUTING.md for guidelines.

License

Licensed under either of Apache License, Version 2.0 or MIT license at your option.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

cc_downloader-1.0.1.tar.gz (50.3 kB view details)

Uploaded Aug 5, 2026 Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

cc_downloader-1.0.1-cp312-abi3-win_amd64.whl (3.1 MB view details)

Uploaded Aug 5, 2026 CPython 3.12+Windows x86-64

cc_downloader-1.0.1-cp312-abi3-manylinux_2_28_x86_64.whl (4.0 MB view details)

Uploaded Aug 5, 2026 CPython 3.12+manylinux: glibc 2.28+ x86-64

cc_downloader-1.0.1-cp312-abi3-manylinux_2_28_aarch64.whl (3.9 MB view details)

Uploaded Aug 5, 2026 CPython 3.12+manylinux: glibc 2.28+ ARM64

cc_downloader-1.0.1-cp312-abi3-macosx_11_0_arm64.whl (3.5 MB view details)

Uploaded Aug 5, 2026 CPython 3.12+macOS 11.0+ ARM64

cc_downloader-1.0.1-cp312-abi3-macosx_10_12_x86_64.whl (3.7 MB view details)

Uploaded Aug 5, 2026 CPython 3.12+macOS 10.12+ x86-64

File details

Details for the file cc_downloader-1.0.1.tar.gz.

File metadata

Download URL: cc_downloader-1.0.1.tar.gz
Upload date: Aug 5, 2026
Size: 50.3 kB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for cc_downloader-1.0.1.tar.gz
Algorithm	Hash digest
SHA256	`26a691345429bd3c0c4747443bba2bc7afb0bbc864143ba6455c5c4bd100ccda`
MD5	`b27b03298a26ffd933593d7257bfcf13`
BLAKE2b-256	`57b7d00b633f3f7ad58e8cb5c32be574ae974bb32bddfc8352cc02f64b3e3177`

See more details on using hashes here.

Provenance

The following attestation bundles were made for cc_downloader-1.0.1.tar.gz:

Publisher: release.yml on commoncrawl/cc-downloader

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: cc_downloader-1.0.1.tar.gz
- Subject digest: 26a691345429bd3c0c4747443bba2bc7afb0bbc864143ba6455c5c4bd100ccda
- Sigstore transparency entry: 2345355787
- Sigstore integration time: Aug 5, 2026
Source repository:
- Permalink: commoncrawl/cc-downloader@17884754287380c6f8f6b26c149fb14e7bfa9112
- Branch / Tag: refs/tags/v1.0.1
- Owner: https://github.com/commoncrawl
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@17884754287380c6f8f6b26c149fb14e7bfa9112
- Trigger Event: push

File details

Details for the file cc_downloader-1.0.1-cp312-abi3-win_amd64.whl.

File metadata

Download URL: cc_downloader-1.0.1-cp312-abi3-win_amd64.whl
Upload date: Aug 5, 2026
Size: 3.1 MB
Tags: CPython 3.12+, Windows x86-64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for cc_downloader-1.0.1-cp312-abi3-win_amd64.whl
Algorithm	Hash digest
SHA256	`667518f03ea42a1d9f726910950dd1384b95b2ceefeefdb01fbcab4d218af77d`
MD5	`7bb01dce222d1697c2124daf7e5de194`
BLAKE2b-256	`8c186afac01758f10269be7f139aff8bfb57e33d7ba7a935ab6a519bb76beca1`

See more details on using hashes here.

Provenance

The following attestation bundles were made for cc_downloader-1.0.1-cp312-abi3-win_amd64.whl:

Publisher: release.yml on commoncrawl/cc-downloader

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: cc_downloader-1.0.1-cp312-abi3-win_amd64.whl
- Subject digest: 667518f03ea42a1d9f726910950dd1384b95b2ceefeefdb01fbcab4d218af77d
- Sigstore transparency entry: 2345356035
- Sigstore integration time: Aug 5, 2026
Source repository:
- Permalink: commoncrawl/cc-downloader@17884754287380c6f8f6b26c149fb14e7bfa9112
- Branch / Tag: refs/tags/v1.0.1
- Owner: https://github.com/commoncrawl
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@17884754287380c6f8f6b26c149fb14e7bfa9112
- Trigger Event: push

File details

Details for the file cc_downloader-1.0.1-cp312-abi3-manylinux_2_28_x86_64.whl.

File metadata

Download URL: cc_downloader-1.0.1-cp312-abi3-manylinux_2_28_x86_64.whl
Upload date: Aug 5, 2026
Size: 4.0 MB
Tags: CPython 3.12+, manylinux: glibc 2.28+ x86-64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for cc_downloader-1.0.1-cp312-abi3-manylinux_2_28_x86_64.whl
Algorithm	Hash digest
SHA256	`97c5d20504c2bb54f447a4097787a555c3168c02ab41ecf5fd4fe6108ae40912`
MD5	`e41f6c7c28f75c9badc791a426fe2321`
BLAKE2b-256	`a609fa2fd00866df05769180057c8942c06b47f09cf777fa0191a61da4e564c0`

See more details on using hashes here.

Provenance

The following attestation bundles were made for cc_downloader-1.0.1-cp312-abi3-manylinux_2_28_x86_64.whl:

Publisher: release.yml on commoncrawl/cc-downloader

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: cc_downloader-1.0.1-cp312-abi3-manylinux_2_28_x86_64.whl
- Subject digest: 97c5d20504c2bb54f447a4097787a555c3168c02ab41ecf5fd4fe6108ae40912
- Sigstore transparency entry: 2345355928
- Sigstore integration time: Aug 5, 2026
Source repository:
- Permalink: commoncrawl/cc-downloader@17884754287380c6f8f6b26c149fb14e7bfa9112
- Branch / Tag: refs/tags/v1.0.1
- Owner: https://github.com/commoncrawl
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@17884754287380c6f8f6b26c149fb14e7bfa9112
- Trigger Event: push

File details

Details for the file cc_downloader-1.0.1-cp312-abi3-manylinux_2_28_aarch64.whl.

File metadata

Download URL: cc_downloader-1.0.1-cp312-abi3-manylinux_2_28_aarch64.whl
Upload date: Aug 5, 2026
Size: 3.9 MB
Tags: CPython 3.12+, manylinux: glibc 2.28+ ARM64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for cc_downloader-1.0.1-cp312-abi3-manylinux_2_28_aarch64.whl
Algorithm	Hash digest
SHA256	`a98f07d5e8ff4d4e7d6f31a96042b83879f3d77c71f0057b6c8bf538060275c9`
MD5	`aee67908804132b88461d255cb06fb22`
BLAKE2b-256	`028ce5b4020d0b78ba9c13412bbcbaf66cb0d9e838dad154c15e7e54eabcd7bd`

See more details on using hashes here.

Provenance

The following attestation bundles were made for cc_downloader-1.0.1-cp312-abi3-manylinux_2_28_aarch64.whl:

Publisher: release.yml on commoncrawl/cc-downloader

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: cc_downloader-1.0.1-cp312-abi3-manylinux_2_28_aarch64.whl
- Subject digest: a98f07d5e8ff4d4e7d6f31a96042b83879f3d77c71f0057b6c8bf538060275c9
- Sigstore transparency entry: 2345355977
- Sigstore integration time: Aug 5, 2026
Source repository:
- Permalink: commoncrawl/cc-downloader@17884754287380c6f8f6b26c149fb14e7bfa9112
- Branch / Tag: refs/tags/v1.0.1
- Owner: https://github.com/commoncrawl
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@17884754287380c6f8f6b26c149fb14e7bfa9112
- Trigger Event: push

File details

Details for the file cc_downloader-1.0.1-cp312-abi3-macosx_11_0_arm64.whl.

File metadata

Download URL: cc_downloader-1.0.1-cp312-abi3-macosx_11_0_arm64.whl
Upload date: Aug 5, 2026
Size: 3.5 MB
Tags: CPython 3.12+, macOS 11.0+ ARM64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for cc_downloader-1.0.1-cp312-abi3-macosx_11_0_arm64.whl
Algorithm	Hash digest
SHA256	`1a963edc13544d4a6aabd78eedf67fcd72f2294d6764224082a73c883e1c311d`
MD5	`1481bc0c04488199bb11e4d0685ed20d`
BLAKE2b-256	`864d20106f22b2af7b54c08366acc92a3e2bd8260d08a8caf842b37e83efc5ea`

See more details on using hashes here.

Provenance

The following attestation bundles were made for cc_downloader-1.0.1-cp312-abi3-macosx_11_0_arm64.whl:

Publisher: release.yml on commoncrawl/cc-downloader

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: cc_downloader-1.0.1-cp312-abi3-macosx_11_0_arm64.whl
- Subject digest: 1a963edc13544d4a6aabd78eedf67fcd72f2294d6764224082a73c883e1c311d
- Sigstore transparency entry: 2345355825
- Sigstore integration time: Aug 5, 2026
Source repository:
- Permalink: commoncrawl/cc-downloader@17884754287380c6f8f6b26c149fb14e7bfa9112
- Branch / Tag: refs/tags/v1.0.1
- Owner: https://github.com/commoncrawl
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@17884754287380c6f8f6b26c149fb14e7bfa9112
- Trigger Event: push

File details

Details for the file cc_downloader-1.0.1-cp312-abi3-macosx_10_12_x86_64.whl.

File metadata

Download URL: cc_downloader-1.0.1-cp312-abi3-macosx_10_12_x86_64.whl
Upload date: Aug 5, 2026
Size: 3.7 MB
Tags: CPython 3.12+, macOS 10.12+ x86-64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for cc_downloader-1.0.1-cp312-abi3-macosx_10_12_x86_64.whl
Algorithm	Hash digest
SHA256	`35a9b92c946edb2f968b454aa93ee844355bfe8a12e20067fc91ac87c9c95737`
MD5	`fa165c235cc94e34072cbbd195f25f22`
BLAKE2b-256	`d254e3c25d220312ef44ce44584eb861c267cc819d0e0d68d061f072b530e44a`

See more details on using hashes here.

Provenance

The following attestation bundles were made for cc_downloader-1.0.1-cp312-abi3-macosx_10_12_x86_64.whl:

Publisher: release.yml on commoncrawl/cc-downloader

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: cc_downloader-1.0.1-cp312-abi3-macosx_10_12_x86_64.whl
- Subject digest: 35a9b92c946edb2f968b454aa93ee844355bfe8a12e20067fc91ac87c9c95737
- Sigstore transparency entry: 2345355870
- Sigstore integration time: Aug 5, 2026
Source repository:
- Permalink: commoncrawl/cc-downloader@17884754287380c6f8f6b26c149fb14e7bfa9112
- Branch / Tag: refs/tags/v1.0.1
- Owner: https://github.com/commoncrawl
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@17884754287380c6f8f6b26c149fb14e7bfa9112
- Trigger Event: push

Release history Release notifications | RSS feed

This release

1.0.1 This release

Aug 5, 2026 6 files

1.0.0

Jul 13, 2026 6 files

cc-downloader 1.0.1

cc-downloader

Requirements

Installation

Quick start

API reference

download_paths

download_contrib_paths

download

Politeness and retries

Contributing

License

Key dates

Owner

License

Requires

Download files

Source Distribution

Built Distributions

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

Release history Release notifications | RSS feed

`download_paths`

`download_contrib_paths`

`download`