Pipeline to post scrapy item to your url
Installation
Install scrapy-sink using pip::
$ pip install scrapy-sink
Configuration
- Add the
settings.pyof your Scrapy project like this:
SINK_ADDR = 'http://127.0.0.1:8000/data_receiver.php'
You can also only set it in your environment variable
- Enable the pipeline by adding it to
ITEM_PIPELINESin yoursettings.pyfile and changing HttpCompressionMiddleware priority:
ITEM_PIPELINES = {
'scrapy_sink.pipelines.SinkPipeline': 9999,
}
The order should after your persist pipeline such as save to database and after your preprocess pipeline.
Usage
no need to change your code
Getting help
Please use github issue
Contributing
PRs are always welcomed.
Changes
0.1.0 (2020-12-26)
Initial release
1 maintainer
Release files for scrapy-sink 0.1.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| scrapy-sink-0.1.2.tar.gz | 2.8 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| scrapy_sink-0.1.2-py2.py3-none-any.whl | Python 3, Python 2 | none | any | Details |
Total release size:6.0 kB
Release files / scrapy-sink-0.1.2.tar.gz
| Download URL | scrapy-sink-0.1.2.tar.gz |
|---|---|
| Size | 2.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
349e002c5dc199eb7f480af3c10568a0cbbe5d08f9cb313b9280431123b6d836
|
|
BLAKE2b-256 checksum How to use checksums |
468c9b759fbd7860caec5a58f7c626f4b5c919933cda54528904508efac10e5f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.2.0 pkginfo/1.6.1 requests/2.22.0 setuptools/45.2.0 requests-toolbelt/0.9.1 tqdm/4.51.0 CPython/3.8.5
|
Release files / scrapy_sink-0.1.2-py2.py3-none-any.whl
| Download URL | scrapy_sink-0.1.2-py2.py3-none-any.whl |
|---|---|
| Size | 3.2 kB |
| Tags | Python 2 Python 3 |
|
SHA-256 checksum How to use checksums |
ca337c9df45a6139dcca7f46ffe69cdebc60ea9df6ab6d503e2a38700bb4c4d7
|
|
BLAKE2b-256 checksum How to use checksums |
190fb6ed7567659b19410db708272e7c129beb2d6adf612949f579d70b43f919
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.2.0 pkginfo/1.6.1 requests/2.22.0 setuptools/45.2.0 requests-toolbelt/0.9.1 tqdm/4.51.0 CPython/3.8.5
|
