High aviariable proxy pool client for crawlers.
Project description
在原先SpiderClub/haipproxy cli 的基础上,稍微修改了下
- 可以通过scrapy中的settings配置来覆盖默认配置,
- 编写了scrapy download middleware ,自动配置proxy,前提是你通过haipproxy配置好了redis proxy pool
todo
- delete proxy 逻辑,删除连接失败的代理
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
haipproxy2-0.1.4.tar.gz
(13.3 kB
view hashes)
Built Distribution
haipproxy2-0.1.4-py3-none-any.whl
(16.1 kB
view hashes)
Close
Hashes for haipproxy2-0.1.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f95b9b2cf9fa494abd216a5e0e1501b5131d9ad77a8573afee2c96c3474c47c1 |
|
MD5 | be68106f30800b9b5f63052efb2f7891 |
|
BLAKE2b-256 | 72ce565f73004a29664a0a2e363b5ac39e05343572e018b0ab27090800a9c3b6 |