Distributed - Asynchronous - Easy-to-use
An Easy-to-use and Fast Python Spider Framework
Overview
- AirSpider is a high-performance asynchronous crawler framework for developers
- Based on Redis: task distribution, task deduplication, and distributed ☁️
Requirements
- Python 3.6
- Works on Linux, Windows, macOS
Features
- Quick to Start
- Low Coupling
- High Cohesion
- Easy Expansion
- Orderly Workflow
Installation
# For Linux && MacOS
pip3 install airspider
# For Windows
pip3 install airspider
Documents
- Introduction：Introduction to AirSpider
- Tutorials： Quick to program a Spider By AirSpider
- Plugins：Program extensions for AirSpider
Topics
- Item：Define Target in HTML
- Selector：Select Target from HTML
- Request：Request Part
- Response：Response Part
- Middleware：Make Airspider Support to Extensions
- Spider：Entrance for AirSpider
TODO
- Complete Plugins of Redis
- Complete Distributed Architecture
Contributing
AirSpider🕷️ is still under Developing
Feel free to open issues💬 and pull requests
- Report or Fix bugs
- Build Powerful plugins
- Make documentation Better
- Add Examples of Spiders
1 maintainer
Release files for AirSpider 2.0.4
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| AirSpider-2.0.4.tar.gz | 16.0 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| AirSpider-2.0.4-py2.py3-none-any.whl | Python 3, Python 2 | none | any | Details |
Total release size: 34.8 kB
Release files / AirSpider-2.0.4.tar.gz
| Download URL | AirSpider-2.0.4.tar.gz |
|---|---|
| Size | 16.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
fbce74e8d154e1136260db7bd5f674feddc1e50327d9e5888ec570feaf76e8d7
|
|
BLAKE2b-256 checksum How to use checksums |
6da5e5cbbde0cc14fc5a0c8afd13541fa796d0d76b657295b5615afa75e9eda6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.6.0 requests-toolbelt/0.9.1 tqdm/4.48.2 CPython/3.7.3
|
Release files / AirSpider-2.0.4-py2.py3-none-any.whl
| Download URL | AirSpider-2.0.4-py2.py3-none-any.whl |
|---|---|
| Size | 18.8 kB |
| Tags | Python 2 Python 3 |
|
SHA-256 checksum How to use checksums |
f0d31596f3582e6103284b48fb35daec75f691037c66f2b167bbadfc5935cf88
|
|
BLAKE2b-256 checksum How to use checksums |
d844c6cfccfaf0d21d662a2fc356c4394573703379ef55da31d3225e1ba65218
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.6.0 requests-toolbelt/0.9.1 tqdm/4.48.2 CPython/3.7.3
|
