spydy 0.1.25

light-weight high-level web-crawling framework

Spydy

spydy is a light-weight high-level web-crawling framework for fast-devlopment and high performance, which is inspired by unix pipeline.

---

Code

Document

---

Install

pip install spydy

How to use

There are two ways of running spydy:

• one way is to prepare a configuration file, and run spydy from cmd:

spydy myconfig.cfg

myconfig.cfg may looks like below:

[Globals]
run_mode = async_forever
nworkers = 4

[PipeLine]
url = DummyUrls
request = AsyncHttpRequest
parser = DmozParser
log = MessageLog
store = CsvStore

[url]
url = https://dmoz-odp.org
repeat = 10

[store]
file_name = result.csv

• or run it from a python file(e.g. spider.py):

from spydy.engine import Engine
from spydy.utils import check_configs
from spydy import urls, request, parsers, logs, store

myconfig = {
    "Globals":{
        "run_mode": "async_forever",
        "nworkers": "4"
    },
    "PipeLine":[urls.DummyUrls(url="https://dmoz-odp.org", repeat=10),
                request.AsyncHttpRequest(), parsers.DmozParser(), logs.MessageLog(), store.CsvStore(file_name=FILE_NAME)]
    }

check_configs(myconfig)
spider = Engine.from_dict(myconfig)
spider.run()

then run it :

$ python spider.py