pyburrow - low-level web crawling library (Python 3)
pyburrow is a Python 3 library for crawling websites: capturing,
archiving and processing their resources. It is different from all
other known crawlers by being very low level: the HTTP response body
is stored as raw unencoded bytes, and further the HTTP response
headers and status codes are stored as well. The resulting rich data
set is a trove for data mining and analysis.