Pipeline to save scrapy item field attachment to qiniu
Installation
Install scrapy-save-to-qiniu using pip::
$ pip install scrapy-save-to-qiniu
Configuration
- Add the
settings.pyof your Scrapy project like this:
QINIU_AK = ''
QINIU_SK = ''
QINIU_BUCKET = ''
QINIU_DOMAIN = ''
QINIU_DEL_SRC = True
QINIU_FIELDS = [
'pdf_url',
]
- Enable the pipeline by adding it to
ITEM_PIPELINESin yoursettings.pyfile and changing HttpCompressionMiddleware priority:
ITEM_PIPELINES = {
'scrapy_save_to_qiniu.pipelines.SaveToQiniuPipeline': -1,
}
The order should before your persist pipeline such as save to database and after your preprocess pipeline.
Usage
no need to change your code
Getting help
Please use github issue
Contributing
PRs are always welcomed.
Changes
0.1.0 (2020-12-26)
Initial release
1 maintainer
Release files for scrapy-save-to-qiniu 0.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| scrapy-save-to-qiniu-0.2.0.tar.gz | 2.8 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| scrapy_save_to_qiniu-0.2.0-py2.py3-none-any.whl | Python 3, Python 2 | none | any | Details |
Total release size:5.8 kB
Release files / scrapy-save-to-qiniu-0.2.0.tar.gz
| Download URL | scrapy-save-to-qiniu-0.2.0.tar.gz |
|---|---|
| Size | 2.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
a6afcb75382514b94d6ee0a149fddf2a5a152b43ea30810b566b31d68dde2df4
|
|
BLAKE2b-256 checksum How to use checksums |
846e1624f900aa5cb8445a7873fb19a37a7ed278b7fbf38a7370866fb85ad1c5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.2.0 pkginfo/1.6.1 requests/2.22.0 setuptools/45.2.0 requests-toolbelt/0.9.1 tqdm/4.51.0 CPython/3.8.5
|
Release files / scrapy_save_to_qiniu-0.2.0-py2.py3-none-any.whl
| Download URL | scrapy_save_to_qiniu-0.2.0-py2.py3-none-any.whl |
|---|---|
| Size | 3.0 kB |
| Tags | Python 2 Python 3 |
|
SHA-256 checksum How to use checksums |
8bf24b86ebe41a509994586aa506313e03fa98ab03724a14721ed3db46fb453a
|
|
BLAKE2b-256 checksum How to use checksums |
f82dd9cca811410f668ffc3d2c29b95d47c2d0f8f897fa06aa8ed56c10c73c33
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.2.0 pkginfo/1.6.1 requests/2.22.0 setuptools/45.2.0 requests-toolbelt/0.9.1 tqdm/4.51.0 CPython/3.8.5
|
