Skip to main content

Efficient asyncio based web crawler

Project description


  • Asynchronous downloading using aiohttp
  • All downloads cached locally in sqlite
  • Continue an interrupted crawl


>>> import ws


Install from pypi:

pip install ws

Or checkout latest version from repository:

hg clone

Project details

Release history Release notifications | RSS feed

This version


Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page