simple proxy pool + proxy validation
Project description
simple-proxy-pool
简单IP代理池 simple_pp 是个 异步并发IP代理验证工具,速度很快,一千个代理半分钟左右可完成。
安装
pip install -U simple-proxy-pool
或下载 repo (e.g., git clone https://github.com/ffreemt/simple-proxy-pool.git
换到 simple-proxy-pool目录执行 python setup.py develop
原理
通过过IP代理访问 www.baidu.com, 能成功获取百度返回的头则代理有效。再检查头里面时候含'via', 不含'via'即为匿名代理。参考 aio_headers.py。
用法
命令行
python -m simple_pp
贴入需验证的IP代理(格式 ip:端口, 以空格、回车非数字字母或中文隔开)。或:
python -m simple_pp file1 file2 ...
文件内含以上格式的IP代理
也可以用pipe,例如
curl "https://www.freeip.top/?page=1" | python -m simple_pp
python 程序内调用
from simple_pp import simple_pp
from pprint import pprint
ip_list = [ip1, ip2, ...]
res = simple_pp(ip_list)
pprint(res)
输出 res 里格式为: res[0] = ip_list[0] +(是否有效,是否匿名,响应时间秒)
可参看tests 里面的文件。有疑问或反馈可发 Issues。
例如
import asyncio
import httpx
from simple_pp import simple_pp
simple_pp(['113.53.230.167:80', '36.25.243.51:80'])
# 输出-> [('113.53.230.167:80', True, False, 0.31),
# ('36.25.243.51:80', True, True, 0.51)]
# 第一个代理为透明代理,第二个代理为匿名代理
也可以直接将网页结果送给 simple_pp, 例如
import re
import asyncio
import httpx
from pprint import pprint
from simple_pp import simple_pp
arun = lambda x: asyncio.get_event_loop().run_until_complete(x)
_ = [elm for elm in simple_pp([':'.join(elm) if elm[1] else elm[0] for elm in re.findall(r'(?:https?://)?(\d{1,3}(?:\.\d{1,3}){3})(?:[\s\t:\'",]+(\d{1,4}))?', arun(httpx.get('https://www.freeip.top/?page=1')).text)]) if elm[-3] is True]
pprint(_)
# 可能拿到将近 10 个代理
# 或
_ = [elm for elm in simple_pp(arun(httpx.get('https://www.freeip.top/?page=1')).text) if elm[-3] is True]
pprint(_)
# ditto
Acknowledgments
- 含 jhao 的 proxypool 项目里的几个文件, 但并未使用,可能将来会用。先感谢jhao开源。
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
simple-proxypool-0.0.1.tar.gz
(25.8 kB
view details)
File details
Details for the file simple-proxypool-0.0.1.tar.gz
.
File metadata
- Download URL: simple-proxypool-0.0.1.tar.gz
- Upload date:
- Size: 25.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.12.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.0 requests-toolbelt/0.8.0 tqdm/4.32.1 CPython/3.6.7
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 07f851a8bcf5c4605b0c3c070b7bfd491700b9d44dcdaf3eec1cb152f44f6898 |
|
MD5 | 78833d34c448a2befc5e8f9535b073f6 |
|
BLAKE2b-256 | a65ac8c2ccc3099870d92aa7d72c72c0bf3dbe817078b92941b2b830c4c04fcc |