AirSpider 2.0.1

A light and fast python Web crawler framework based on asyncio.

Distributed🌍 - Asynchronous🏃 - Light☁️ - Fast⚡️ - Easy😄

AirSpider🕷️, a Light and Fast Python Web Crawler Framework Based on Redis🕷️

---

Overview👀

• AirSpider is a high-performance asynchronous crawler framework for developers 🚀
• Based on Redis: task distribution, task deduplication, and distributed ☁️

Requirements☁️

• Python 3.6➕
• Works on Linux, Windows, macOS🍎

Features🌲

• Quick to Start ☑️
• Low Coupling ☑️
• High Cohesion ☑️
• Easy Expansion ☑️
• Orderly Workflow ☑️

Installation🔨

---------------------------

# For Linux && MacOS🔥
pip3 install airspider

---------------------------

# For Windows🔥
pip3 install airspider

---------------------------

• Documents🔥

  • Introduction：介绍airspider
  • Tutorials：使用airspider快速编写一个稳健的爬虫
  • Plugins：编写airspider扩展

  Topics

  • Item：定义爬虫的目标字段
  • Selector：从HTML中提取出目标字段
  • Request：请求并抓取目标网站资源
  • Response：进一步封装响应内容
  • Middleware：使爬虫支持第三方扩展
  • Spider：爬虫程序的入口

TODO✈️

• Complete Plugins of Redis🔥
• Complete Distributed Architecture☁️

Contributing👬

AirSpider🕷️ is still under Developing🔨

Feel free to open issues💬 and pull requests💗

• Report or Fix bugs🌈
• Build Powerful plugins🔥
• Make documentation Better📖
• Add Examples of Crawling 🕷️