An Easy-to-use and Fast Python Spider Framework
Project description
Distributed - Asynchronous - Easy-to-use
An Easy-to-use and Fast Python Spider Framework
Overview
- AirSpider is a high-performance asynchronous crawler framework for developers
- Based on Redis: task distribution, task deduplication, and distributed ☁️
Requirements
- Python 3.6
- Works on Linux, Windows, macOS
Features
- Quick to Start
- Low Coupling
- High Cohesion
- Easy Expansion
- Orderly Workflow
Installation
# For Linux && MacOS
pip3 install airspider
# For Windows
pip3 install airspider
Documents
- Introduction：Introduction to AirSpider
- Tutorials： Quick to program a Spider By AirSpider
- Plugins：Program extensions for AirSpider
Topics
- Item：Define Target in HTML
- Selector：Select Target from HTML
- Request：Request Part
- Response：Response Part
- Middleware：Make Airspider Support to Extensions
- Spider：Entrance for AirSpider
TODO
- Complete Plugins of Redis
- Complete Distributed Architecture
Contributing
AirSpider🕷️ is still under Developing
Feel free to open issues💬 and pull requests
- Report or Fix bugs
- Build Powerful plugins
- Make documentation Better
- Add Examples of Spiders
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
AirSpider-2.0.4.tar.gz
(16.0 kB
view details)
Built Distribution
File details
Details for the file AirSpider-2.0.4.tar.gz.
File metadata
- Download URL: AirSpider-2.0.4.tar.gz
- Upload date:
- Size: 16.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.6.0 requests-toolbelt/0.9.1 tqdm/4.48.2 CPython/3.7.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | fbce74e8d154e1136260db7bd5f674feddc1e50327d9e5888ec570feaf76e8d7 |
|
| MD5 | a324879ef98ea2967b8e1e122f9a7cc8 |
|
| BLAKE2b-256 | 6da5e5cbbde0cc14fc5a0c8afd13541fa796d0d76b657295b5615afa75e9eda6 |
File details
Details for the file AirSpider-2.0.4-py2.py3-none-any.whl.
File metadata
- Download URL: AirSpider-2.0.4-py2.py3-none-any.whl
- Upload date:
- Size: 18.8 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.6.0 requests-toolbelt/0.9.1 tqdm/4.48.2 CPython/3.7.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | f0d31596f3582e6103284b48fb35daec75f691037c66f2b167bbadfc5935cf88 |
|
| MD5 | 733c9fc9262cf1394911a1446632ef9d |
|
| BLAKE2b-256 | d844c6cfccfaf0d21d662a2fc356c4394573703379ef55da31d3225e1ba65218 |
