Skip to main content
Join the official 2019 Python Developers SurveyStart the survey!

Scrapy pipeline extension for qiniu.com

Project description

scrapy_qiniu
=====

Scrapy中的media pipeline机制,可以方便地将静态资源资源(文件、图像等)下载到本地,然后进行处理。scrapy-qiniu扩展了这一机制,可以将资源存储到七牛云存储上面。并且,实现了以下几个特性:

* 支持缓存,可以避免静态资源的重复下载
* 采用fetch模式,让七牛服务器代为下载,而不用像默认的FilesPipeline那样,先下载到爬虫所在
主机,然后再上传到七牛服务器

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for scrapy-qiniu, version 0.1.2
Filename, size File type Python version Upload date Hashes
Filename, size scrapy_qiniu-0.1.2-py2-none-any.whl (5.7 kB) File type Wheel Python version py2 Upload date Hashes View hashes
Filename, size scrapy_qiniu-0.1.2.tar.gz (3.9 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page