pyburrow - low-level web crawling library (Python 3)
Project description
pyburrow is a Python 3 library for crawling websites: capturing, archiving and processing their resources. It is different from all other known crawlers by being very low level: the HTTP response body is stored as raw unencoded bytes, and further the HTTP response headers and status codes are stored as well. The resulting rich data set is a trove for data mining and analysis.
