Skip to main content

pyburrow - low-level web crawling library (Python 3)

Project description

pyburrow is a Python 3 library for crawling websites: capturing, archiving and processing their resources. It is different from all other known crawlers by being very low level: the HTTP response body is stored as raw unencoded bytes, and further the HTTP response headers and status codes are stored as well. The resulting rich data set is a trove for data mining and analysis.

Project details


Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page