在scrapy的基础上进行修改,使用了异步请求,加快采集速度,并与rabbitmq结合,实现了断点续爬,与防止漏采
Project description
Scrapy_Rabbit
Overview
将Scrapy与RabbitMQ进行结合,解决断点续爬,采集进度可视化并且提高代码复用性
Requirements
- Python 3.6+
- Works on Linux, Windows, macOS, BSD
Install
The quick way::
pip install scrapy-rabbit
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
scrapy_rabbit-0.2.1.tar.gz
(36.5 kB
view hashes)