simplified-scrapy 0.6.71

A Simple Distributed Web Crawle

simplified-scrapy

simplified scrapy, A Simple Web Crawle

Requirements

• Python 2.7, 3.0+
• Works on Linux, Windows, Mac OSX, BSD

run

from simplified_scrapy.simplified_main import SimplifiedMain
SimplifiedMain.startThread()

Demo

Custom crawler class needs to extend Spider class

from core.spider import Spider 
class DemoSpider(Spider):

Here is an example of collecting data

from simplified_scrapy.spider import Spider, SimplifiedDoc
class DemoSpider(Spider):
  name = 'demo-spider'
  start_urls = ['http://www.scrapyd.cn/']
  allowed_domains = ['www.scrapyd.cn']
  def extract(self, url, html, models, modelNames):
    doc = SimplifiedDoc(html)
    lstA = doc.listA(url=url["url"])
    return [{"Urls": lstA, "Data": None}]

from simplified_scrapy.simplified_main import SimplifiedMain
SimplifiedMain.startThread(DemoSpider())

pip install

pip install simplified-scrapy

Examples