Skip to main content
Join the official 2019 Python Developers SurveyStart the survey!

Efficient asyncio based web crawler

Project description

Features

  • Asynchronous downloading using aiohttp
  • All downloads cached locally in sqlite
  • Continue an interrupted crawl

Example

>>> import ws
...

Install

Install from pypi:

pip install ws

Or checkout latest version from repository:

hg clone https://bitbucket.org/richardpenman/ws

Project details


Release history Release notifications

This version

0.1

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page