Intelligent proxy pool for Humans™
An intelligent proxy pool for humanities, only supports Python 3.6. Key features:
- Automatic proxy ip crawling and validation
- Easy-to-use JSON API
- Simple but beautiful web-based user interface (eg. geographical distribution of proxies)
- Get started with only 1 command minimally
- Simple HTTP Forward proxy server
- Scrapy and requests integration with only 1 line of code minimally
- Headless browser crawling
Please read the Documentation.
Install with Docker (highly recommended)
docker run -d -p 8899:8899 -p 8081:8081 -v /var/www/scylla:/var/www/scylla --name scylla wildcat/scylla:latest
Install directly via pip
pip install scylla scylla --help scylla # Run the crawler and web server for JSON API
Install from source
git clone https://github.com/imWildCat/scylla.git cd scylla pip install -r requirements.txt npm install # or yarn install make assets-build python -m scylla
For Windows user who fails at installing sanic due to uvloop does not support Windows at the moment:
export SANIC_NO_UVLOOP=true export SANIC_NO_UJSON=true pip3 install sanic
If this also fails, yoi will need to manual install sanic from source.
Please see Projects.
Development and Contribution
git clone https://github.com/imWildCat/scylla.git cd scylla pip install -r requirements.txt npm install # or `yarn install` make assets-build
If you wish to run tests locally, the commands are shown below:
pip install -r tests/requirements-test.txt pytest tests/
You are welcomed to add more test cases to this project, increasing the robustness of this project.
Naming of This Project
If you find this project useful, could you please donate some money to it?
No matter how much the money is, Your donation will inspire the author to develop new features continuously! 🎉 Thank you!
The ways for donation are shown below:
Apache License 2.0. For more details, please read the LICENSE file.
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size gd-scylla-10.0.0.tar.gz (31.7 kB)||File type Source||Python version None||Upload date||Hashes View|