Skip to main content

Efficient asyncio based web crawler

Project description

Features

  • Asynchronous downloading using aiohttp
  • All downloads cached locally in sqlite
  • Continue an interrupted crawl

Example

>>> import ws
...

Install

Install from pypi:

pip install ws

Or checkout latest version from repository:

hg clone https://bitbucket.org/richardpenman/ws

Project details


Release history Release notifications | RSS feed

This version

0.1

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page