HarvestMan - Multithreaded Offline Browser/Web Crawler
Project description
HarvestMan is a multithreaded off-line browser.It has many features for customizing offline browsing through URL filters, depth-fetching, fetch levels, domain filters, file limits, thread limits, download depth, directory checking, and robot exclusion protocol. It is useful to download an entire Web site or certain files from a Web site to the hard disk for offline browsing later.
