Skip to main content

Efficient asyncio based web crawler

Project description

Features

  • Asynchronous downloading using aiohttp
  • All downloads cached locally in sqlite
  • Continue an interrupted crawl

Example

>>> import ws
...

Install

Install from pypi:

pip install ws

Or checkout latest version from repository:

hg clone https://bitbucket.org/richardpenman/ws

Project details


Release history Release notifications | RSS feed

This version

0.1

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page