Multithreaded crawling, reporting, and mirroring for Web and FTP
Project description
This module provides multithreaded crawling, reporting, and mirroring for Web and FTP in one convenient library. Crawling depth, maximum number of URLs to crawl, and maximum number of threads are user-configurable. Reports can be generated on external URLS, internal redirects to outside URLs, unparsable HTML, non-HTTP/FTP URLs, and broken links.
