This module provide asynchronously scraping a bunch of http, https, socks4, socks5 proxies from websites such as:
Just now proxy-master scraped 23265 proxies from websites above. Count of unique by ip:port is 18383,
but some servers support different protocols.
I add more websites to scrap from time to time, BUT looking for help and advices from people with experience
Dependencies
python3.10 and higher. Pattern matching using in project
Will be installed automatically with pip
aiohttp bs4 lxml pycountry aiohttp_socks
Installation
pip install proxy-master
Usage
- This will create
proxy_master.jsonin you home directory and return list of proxies.
import proxy_master as pm
proxies = pm.get_proxies(
protocol='http',
do_prints=True
)
Then you can test proxies:
working_proxies = pm.test_proxies(
proxies,
proxy_protocol: 'http',
website_protocol: 'http'
)
Features
- Scrap different type of proxies include
https,socks4,socks5 - Recursively scraping. Use already collected proxies to scrap another website
- test_public_ip(...)
socks4,socks5using aiohttp-proxy
In plans
1 maintainer
Metadata
1 maintainer
Release files for proxy-master 2.0.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| proxy_master-2.0.1.tar.gz | 7.9 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| proxy_master-2.0.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 16.2 kB
Release files / proxy_master-2.0.1.tar.gz
| Download URL | proxy_master-2.0.1.tar.gz |
|---|---|
| Size | 7.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
bc5eb789d0e1496b28f4adc6265cd5ec3a9b94c719b93ee1b9c129b8f1895f8d
|
|
BLAKE2b-256 checksum How to use checksums |
b90b7691d9467a63db7bdecfcfc7a9cfb02cd581bcdaf6f4b02f0bda4be25664
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.11.1
|
Release files / proxy_master-2.0.1-py3-none-any.whl
| Download URL | proxy_master-2.0.1-py3-none-any.whl |
|---|---|
| Size | 8.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
9e455848998ac9ffb1f758ba36454cce6150b4f78509ac397f33d90b45356959
|
|
BLAKE2b-256 checksum How to use checksums |
40097481e898908911c0191fb53d46450ddb2f0cf37ab5e5ca09a8e3aeaabcd0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.11.1
|
