Scrapy pipeline extension for qiniu.com
Project description
scrapy_qiniu
=====
Scrapy中的media pipeline机制,可以方便地将静态资源资源(文件、图像等)下载到本地,然后进行处理。scrapy-qiniu扩展了这一机制,可以将资源存储到七牛云存储上面。并且,实现了以下几个特性:
* 支持缓存,可以避免静态资源的重复下载
* 采用fetch模式,让七牛服务器代为下载,而不用像默认的FilesPipeline那样,先下载到爬虫所在
主机,然后再上传到七牛服务器
=====
Scrapy中的media pipeline机制,可以方便地将静态资源资源(文件、图像等)下载到本地,然后进行处理。scrapy-qiniu扩展了这一机制,可以将资源存储到七牛云存储上面。并且,实现了以下几个特性:
* 支持缓存,可以避免静态资源的重复下载
* 采用fetch模式,让七牛服务器代为下载,而不用像默认的FilesPipeline那样,先下载到爬虫所在
主机,然后再上传到七牛服务器
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
scrapy_qiniu-0.1.2.tar.gz
(3.9 kB
view hashes)
Built Distribution
Close
Hashes for scrapy_qiniu-0.1.2-py2-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ddb30516e0c094fa373d5f150da50f4fcdd43df862035f647a9cb87ef1753d4f |
|
MD5 | 1a0185f834890ab3411c1f3a588d72cb |
|
BLAKE2b-256 | 6ff3dba632b0eafcd6f0d2f74589318360a267f82f81615bc539d8aea12e971c |