6 projects
cc-downloader
Python bindings for the Common Crawl downloader
cc-pyspark
Common Crawl data processing examples for PySpark.
commonlid
Evaluate language identification models on CommonLID and other benchmarks.
url-is-in
A Python package for efficiently checking if a URL is part of large whitelist or blacklist of URLs and domain names.
cdx-toolkit
A toolkit for working with CDX indices
warc2zip
Convert WARC files into zip-of-files plus metadata