Asynchronous crawler micro-framework based on python.
Project description
hoopa
简介
hoopa 是一个基于异步的快速、轻量、分布式爬虫框架
- 支持基于内存、redis、rabbitmq的优先级队列
- 支持aiohttp、httpx
- 支持断点续传
环境要求:
- Python 3.7.0+
- Works on Linux, Windows, macOS
安装
# For Linux & Mac
pip install -U hoopa[uvloop]
# For Windows
pip install -U hoopa
小试一下
创建爬虫
hoopa create -s first_spider
创建后的爬虫代码如下:
import hoopa
class FirstSpider(hoopa.Spider):
async def start_requests(self):
yield hoopa.Request("https://www.baidu.com")
async def parse(self, request, response):
print(response)
if __name__ == "__main__":
FirstSpider().start()
TODO
- 用于debug的缓存爬虫
感谢
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
No source distribution files available for this release.See tutorial on generating distribution archives.
Built Distribution
hoopa-0.0.2-py3-none-any.whl
(44.8 kB
view hashes)