gspider

https://img.shields.io/badge/license-Apache2-blue.svg

A web spider framework based on gevent and requests.

Spider Example

以下是我们的一个爬虫类示例，其作用为爬取百度新闻的热点要闻:

from gspider import Spider, HttpRequest, run_spider, Selector


class BaiduNewsSpider(Spider):
    def start_requests(self):
        yield HttpRequest("http://news.baidu.com/")

    def parse(self, response):
        selector = Selector(response.text)
        hot = selector.css("div.hotnews a").text
        self.log("Hot News:")
        for i in range(len(hot)):
            self.log("%s: %s", i + 1, hot[i])


if __name__ == '__main__':
    run_spider(BaiduNewsSpider)

在爬虫类中我们定义了一些方法：

start_requests: 返回爬虫初始请求。
parse: 处理请求得到的页面，这里借助 Selector 及CSS Selector语法提取到了我们所需的数据。

Documentation

http://gspider.readthedocs.io/

Metadata

Release files for gspider 0.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for gspider 0.1.1
File	Size	Uploaded
gspider-0.1.1.tar.gz	26.5 kB	Apr 1, 2020	Details

Release files / gspider-0.1.1.tar.gz

Download URL	gspider-0.1.1.tar.gz
Size	26.5 kB
Tags	Source
SHA-256 checksum How to use checksums	`da0be69cf6759a404e3b0d79acac52615113776ac550a3dbc0a1b6f13fb901eb`
BLAKE2b-256 checksum How to use checksums	`abfe5d13fbd53689938baafdeabc2f0f4c6eca2ed1714a54dc82d0158beaf9d6`
Upload date	Apr 1, 2020
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`Python-urllib/3.6`