No project description provided
Project description
spider-brew-kit
A library for scrapy tools, including but not limited to the usual pipelines, middlewares, etc.
install
pip install spider-brew-kit
usage
pipelines
mongo pipeline
A pipeline saved into MongoDB asynchronously with txmongo
use database db.createUser( { user: "username", pwd: "password", roles: [ { role: "readWrite", db: "database" } ] } )
how use:
- add to settings.py
ITEM_PIPELINES = {
'scrapy_kit.pipelines.MongoPipeline': 300,
}
- add mongo config to settings.py
MONGO_URI = "mongodb://username:password@host:port"
MONGO_DATABASE_NAME = "database"
MONGO_COLLECTION_NAME = "collection"
middlewares
proxy connection close middleware
Proxy close connection multiplexing middleware
Tunnel Proxy Dynamic Edition request found that the number of requests in the Personal Centre Tunnel Proxy Usage Statistics is very small, which is seriously inconsistent with the real number of requests. Moreover, there is no IP change when using Tunnel Broker Dynamic Edition. The reason for this is that the tunnel sends requests that reuse previously established connections. You need to add Connection: close to the header.
How to use it:
- Add in settings.py:
DOWNLOADER_MIDDLEWARES = {
'scrapy_kit.middlewares.ProxyConnectionCloseMiddleware': 543,
}
development
git clone
cd spider-brew-kit
poetry install
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for spider_brew_kit-0.1.6-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ca0fb973348b35c104b5c45875730fe1fa5565ad31b8a1534c201ecdb4647026 |
|
MD5 | ae446f162f306c0a7352517c0b875fca |
|
BLAKE2b-256 | fed8bc8b17f5995b386aa44de21dac4827f9ed83a010ac4af90fd27cebb6199a |