Intelligent proxy pool for Humans™
Project description
An intelligent proxy pool for humanities, only supports Python 3.6. Key features:
Automatic proxy ip crawling and validation
Easy-to-use JSON API
Simple but beautiful web-based user interface (eg. geographical distribution of proxies)
Get started with only 1 command minimally
Straightforward programmable API
(Additional) Headless browser crawling
Get started
Install with Docker (highly recommended)
docker run -it --name scylla wildcat/scylla
Install directly via pip
pip install scylla
scylla # Run the cralwer and web server for JSON API
Use from source
git clone https://github.com/imWildCat/scylla.git
cd scylla
pip install -r requirements.txt
npm install # or yarn install
npm build # or yarn build
python -m scylla
Programmatically
For more details, please read the API Documentation.
Roadmap
[x] Basic RESTFul API
[x] Advanced RESTFul API (Sorting, paging, conditions)
[ ] Programmable API
[x] Docker integration
[ ] Smart web proxy server
[ ] More statistical views of proxies
[ ] Quality statistics for different providers
[x] Over 10 data sources
Development and Contribution
git clone https://github.com/imWildCat/scylla.git
cd scylla
pip install -r requirements.txt
npm install # or yarn install
Testing
pip install -r tests/requirements-test.txt
pytest -n 10 tests
License
Apache License 2.0. For more details, please read the LICENSE file.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.