Scrapy pipeline extension for qiniu.com
Project description
scrapy_qiniu
=====
Scrapy中的media pipeline机制，可以方便地将静态资源资源（文件、图像等）下载到本地，然后进行处理。scrapy-qiniu扩展了这一机制，可以将资源存储到七牛云存储上面。并且，实现了以下几个特性：
* 支持缓存，可以避免静态资源的重复下载
* 采用fetch模式，让七牛服务器代为下载，而不用像默认的FilesPipeline那样，先下载到爬虫所在
主机，然后再上传到七牛服务器
=====
Scrapy中的media pipeline机制，可以方便地将静态资源资源（文件、图像等）下载到本地，然后进行处理。scrapy-qiniu扩展了这一机制，可以将资源存储到七牛云存储上面。并且，实现了以下几个特性：
* 支持缓存，可以避免静态资源的重复下载
* 采用fetch模式，让七牛服务器代为下载，而不用像默认的FilesPipeline那样，先下载到爬虫所在
主机，然后再上传到七牛服务器
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
scrapy_qiniu-0.1.2.tar.gz
(3.9 kB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file scrapy_qiniu-0.1.2.tar.gz.
File metadata
- Download URL: scrapy_qiniu-0.1.2.tar.gz
- Upload date:
- Size: 3.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
250c3848a503d60051893552b0eee88af4cec991d564beebccddc3078bb47cb7
|
|
| MD5 |
131d0acf3163fabb218706f85cb577a4
|
|
| BLAKE2b-256 |
913ff2c8715621104d9341fe933ad899f9a31dba756df8b956e35acc33b6f0cd
|
File details
Details for the file scrapy_qiniu-0.1.2-py2-none-any.whl.
File metadata
- Download URL: scrapy_qiniu-0.1.2-py2-none-any.whl
- Upload date:
- Size: 5.7 kB
- Tags: Python 2
- Uploaded using Trusted Publishing? No
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
ddb30516e0c094fa373d5f150da50f4fcdd43df862035f647a9cb87ef1753d4f
|
|
| MD5 |
1a0185f834890ab3411c1f3a588d72cb
|
|
| BLAKE2b-256 |
6ff3dba632b0eafcd6f0d2f74589318360a267f82f81615bc539d8aea12e971c
|
