This module provides FTP and Web spiders and mirroring utilities in one convenient script. FTP sites are crawled by walking a remote directory tree. Websites are crawled by visiting URLs extracted from HTML pages. Downloads during mirroring are multi-threaded for supported protocols. Other features included listing bad URLs and the URL containing them and finding horrifically malformed HTML. Requires Python 2.3

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for lrees from gravatar.com lrees

Credits

Author/Maintainer: L. C. Rees

License

BSD

Tags

spider robot crawler ftp crawler ftp robot ftp spider web crawler web robot web spider web-bot link checker bad link finder

Classifiers

Report project as malware

Release history Release notifications | RSS feed

This release

0.44 This release

0.5

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page