Skip to main content

pyburrow - low-level web crawling library (Python 3)

Project description

pyburrow is a Python 3 library for crawling websites: capturing, archiving and processing their resources. It is different from all other known crawlers by being very low level: the HTTP response body is stored as raw unencoded bytes, and further the HTTP response headers and status codes are stored as well. The resulting rich data set is a trove for data mining and analysis.

Project details


Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page