爬虫代理IP池
Project description
proxy_pool_redis 使用redis作为后端存储的ip代理池
能够根据对应付费ip的请求Url加载对应的Proxy ip到池中,并能够提供多个应用使用,对应每个应用需要传入特定的应用名字做标示,可对特定的应用使用的Ip报告被ban,质量差的ip
使用方法
from proxy_pool_redis import XunProxyPool
pool = XunProxyPool(api_url='xxx',name='crawl_name',redis_host='xx',redis_port=xx,redis_password='xx',report_num=10)
pool.start()
# 获取一个ip
ip = pool.get_ip()
# 报告ip已经被当前应用ban了
pool.report_ban_ip(ip)
# 报告当前ip的质量不好
# 如果报告的次数达到了report_num的值,则将该ip ban掉
pool.report_bad_ip(ip)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
proxy_pool_redis-0.0.8.tar.gz
(6.7 kB
view hashes)
Built Distribution
Close
Hashes for proxy_pool_redis-0.0.8-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3c59fe04a37f76ab44ca8b455755b7adb52b1c11448f0444366bdd8ba510c710 |
|
MD5 | d6d0bae35ac9a83f8f70f9029ed22412 |
|
BLAKE2b-256 | a905810e9e7edd4b4f397d6740c4b69655806e0c7673853d4fd8253e027c8307 |