Scrapy HTTP POST items pipeline
Project description
scrapy-http-pipeline
Just a simple Scrapy HTTP pipeline to POST your items to your server.
Usage
Install
pip install scrapy-http-pipeline
Configure scrapy settings.py
ITEM_PIPELINES = {
'scrapyhttppipeline.scrapyhttppipeline.HttpPostPipeline': 500
}
# Url to your server, which accepts POST requests
HTTP_POST_PIPELINE_URL = 'http://localhost:8080/items'
# Any custom headers you want to add, e.g. authentication
HTTP_POST_PIPELINE_HEADERS = {
'X-Authorization': 'xxx'
}
# If you want to send more items at once (and have less HTTP POST requests incoming.)
# If True items will be send as [{key1:val1},{key1:val1}] instead of {key1:val1}
HTTP_POST_PIPELINE_BUFFERED = False
HTTP_POST_PIPELINE_BUFFER_SIZE = 100
Developing
Package requirements are handled using pip. To install them do
pip install -r requirements.txt
Tests
Testing is set up using pytest and coverage is handled with the pytest-cov plugin.
Run your tests with py.test
in the root directory.
Coverage is ran by default and is set in the pytest.ini
file.
To see an html output of coverage open htmlcov/index.html
after running the tests.
Travis CI
There is a .travis.yml
file that is set up to run your tests for python 2.7
and python 3.3-3.7, should you choose to use it.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for scrapy-http-pipeline-0.2.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | f348fdb7c734be64b95e1cdcff0fcbf0eb832becaac8107576b47455f65fb897 |
|
MD5 | 689f1ae493660a40c66dbebe82ebff6f |
|
BLAKE2b-256 | 8f6100bc5a724d79407e5bc0787864ed6b4322398a1738e388edc03e901a6f62 |
Close
Hashes for scrapy_http_pipeline-0.2.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f3b74c57c6b771dd234692d6c5f3a3d16ccd96c80309cc8de9573e8afeac716b |
|
MD5 | a9d0600b8a5ede9b5920f8a7f5ce77ec |
|
BLAKE2b-256 | 849193f44db99846747333328d292d9fca150e32f4056872a54d73d0d8112ee4 |