Intelligent proxy pool for Humans™
Project description
An intelligent proxy pool for humanities, only supports Python 3.6. Key features:
Automatic proxy ip crawling and validation
Easy-to-use JSON API
Simple but beautiful web-based user interface (eg. geographical distribution of proxies)
Get started with only 1 command minimally
Simple HTTP Forward proxy server
Scrapy and requests integration with only 1 line of code minimally
Headless browser crawling
对于偏好中文的用户,请阅读 中文文档。For those who prefer to use Chinese, please read the Chinese Documentation
Documentation
Please read the Documentation.
Quick start
Installation
Install with Docker (highly recommended)
docker run -d -p 8899:8899 -p 8081:8081 -v /var/www/scylla:/var/www/scylla --name scylla wildcat/scylla:latest
Install directly via pip
pip install scylla
scylla --help
scylla # Run the crawler and web server for JSON API
Install from source
git clone https://github.com/imWildCat/scylla.git
cd scylla
pip install -r requirements.txt
npm install # or yarn install
make assets-build
python -m scylla
For Windows user who fails at installing sanic due to uvloop does not support Windows at the moment:
export SANIC_NO_UVLOOP=true
export SANIC_NO_UJSON=true
pip3 install sanic
If this also fails, yoi will need to manual install sanic from source.
Roadmap
Please see Projects.
Development and Contribution
git clone https://github.com/imWildCat/scylla.git
cd scylla
pip install -r requirements.txt
npm install # or `yarn install`
make assets-build
Testing
If you wish to run tests locally, the commands are shown below:
pip install -r tests/requirements-test.txt
pytest tests/
You are welcomed to add more test cases to this project, increasing the robustness of this project.
Naming of This Project
Scylla is derived from the name of a group of memory chips in the American TV series, Prison Break. This project was named after this American TV series to pay tribute to it.
Donation
If you find this project useful, could you please donate some money to it?
No matter how much the money is, Your donation will inspire the author to develop new features continuously! 🎉 Thank you!
The ways for donation are shown below:
PayPal
Alipay or WeChat Pay
License
Apache License 2.0. For more details, please read the LICENSE file.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for scylla-1.1.7-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d1183f417d7dd2b57275f773ae0d0d70d46b2694e88401f41f4a8b22bcc1cfb8 |
|
MD5 | ff025531e9b4a37f9faa6c341697c64c |
|
BLAKE2b-256 | b26512bde3e6196a1bfe641172c8639a53ec3d134c45cb5fa3dc1c3c6c9f17ab |