An Easy-to-use and Fast Python Spider Framework
Project description
Distributed - Asynchronous - Easy-to-use
An Easy-to-use and Fast Python Spider Framework
Overview
- AirSpider is a high-performance asynchronous crawler framework for developers
- Based on Redis: task distribution, task deduplication, and distributed ☁️
Requirements
- Python 3.6
- Works on Linux, Windows, macOS
Features
- Quick to Start
- Low Coupling
- High Cohesion
- Easy Expansion
- Orderly Workflow
Installation
# For Linux && MacOS
pip3 install airspider
# For Windows
pip3 install airspider
Documents
- Introduction:Introduction to AirSpider
- Tutorials: Quick to program a Spider By AirSpider
- Plugins:Program extensions for AirSpider
Topics
- Item:Define Target in HTML
- Selector:Select Target from HTML
- Request:Request Part
- Response:Response Part
- Middleware:Make Airspider Support to Extensions
- Spider:Entrance for AirSpider
TODO
- Complete Plugins of Redis
- Complete Distributed Architecture
Contributing
AirSpider🕷️ is still under Developing
Feel free to open issues💬 and pull requests
- Report or Fix bugs
- Build Powerful plugins
- Make documentation Better
- Add Examples of Spiders
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
AirSpider-2.0.4.tar.gz
(16.0 kB
view hashes)
Built Distribution
Close
Hashes for AirSpider-2.0.4-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f0d31596f3582e6103284b48fb35daec75f691037c66f2b167bbadfc5935cf88 |
|
MD5 | 733c9fc9262cf1394911a1446632ef9d |
|
BLAKE2b-256 | d844c6cfccfaf0d21d662a2fc356c4394573703379ef55da31d3225e1ba65218 |