scylla 0.1.7

Intelligent proxy pool for Humans™

An intelligent proxy pool for humanities, only supports Python 3.6. Key features:

• Automatic proxy ip crawling and validation

• Easy-to-use JSON API

• Simple but beautiful web-based user interface (eg. geographical distribution of proxies)

• Get started with only 1 command minimally

• Straightforward programmable API

• (Additional) Headless browser crawling

Get started

Install with Docker (highly recommended)

docker run -it --name scylla wildcat/scylla

Install directly via pip

pip install scylla
scylla # Run the cralwer and web server for JSON API

Use from source

git clone https://github.com/imWildCat/scylla.git
cd scylla

pip install -r requirements.txt

npm install # or yarn install
npm build # or yarn build

python -m scylla

Programmatically

For more details, please read the API Documentation.

Roadmap

• [x] Basic RESTFul API

• [x] Advanced RESTFul API (Sorting, paging, conditions)

• [ ] Programmable API

• [x] Docker integration

• [ ] Smart web proxy server

• [ ] More statistical views of proxies

• [ ] Quality statistics for different providers

• [x] Over 10 data sources

Development and Contribution

git clone https://github.com/imWildCat/scylla.git
cd scylla

pip install -r requirements.txt

npm install # or yarn install

Testing

pip install -r tests/requirements-test.txt
pytest -n 10 tests

License

Apache License 2.0. For more details, please read the LICENSE file.