Distributed web crawling with browsers
rethinkdb python library
Rudimentary rethinkdb python library with some smarts, perhaps some dumbs
Sort-friendly URI Reordering Transform (SURT) python package.
url canonicalization library for python and java
WARC writing MITM HTTP/S proxy
Command line tools and libraries for handling and manipulating WARC files (and HTTP contents)