Intelligent proxy pool for Humans™
Project description
An intelligent proxy pool for humanities, only supports Python 3.6. Key features:
Automatic proxy ip crawling and validation
Easy-to-use JSON API
Simple but beautiful web-based user interface (eg. geographical distribution of proxies)
Get started with only 1 command minimally
Straightforward programmable API
(Additional) Headless browser crawling
Get started
Install with Docker (highly recommended)
docker run -it --name scylla wildcat/scylla
Install directly via pip
pip install scylla
scylla # Run the cralwer and web server for JSON API
Use from source
git clone https://github.com/imWildCat/scylla.git
cd scylla
pip install -r requirements.txt
npm install # or yarn install
npm build # or yarn build
python -m scylla
Programmatically
For more details, please read the API Documentation.
Roadmap
[x] Basic RESTFul API
[x] Advanced RESTFul API (Sorting, paging, conditions)
[ ] Programmable API
[x] Docker integration
[ ] Smart web proxy server
[ ] More statistical views of proxies
[ ] Quality statistics for different providers
[x] Over 10 data sources
Development and Contribution
git clone https://github.com/imWildCat/scylla.git
cd scylla
pip install -r requirements.txt
make build-assets
Testing
pip install -r tests/requirements-test.txt
pytest -n 10 tests
License
Apache License 2.0. For more details, please read the LICENSE file.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for scylla-0.2.3-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 328455b2f81ac9639a630e657d4c3763fc1d1026838108cafe9e099f7dfc45c0 |
|
MD5 | 3c2d0189bd08cb3b872293ba9445413b |
|
BLAKE2b-256 | e62d48bfc71e60ea4fe057d3b211e5aad64eaa7db7c28aa95ab6c99bdbbd5a22 |