Spydy

spydy is a light-weight high-level web-crawling framework for fast-devlopment and high performance, which is inspired by unix pipeline.

Code

Document

Install

pip install spydy

How to use

There are two ways of running spydy:

one way is to prepare a configuration file, and run spydy from cmd:

spydy myconfig.cfg

myconfig.cfg may looks like below:

[Globals]
run_mode = async_forever
nworkers = 4

[PipeLine]
url = DummyUrls
request = AsyncHttpRequest
parser = DmozParser
log = MessageLog
store = CsvStore

[url]
url = https://dmoz-odp.org
repeat = 10

[store]
file_name = result.csv

or run it from a python file(e.g. spider.py):

from spydy.engine import Engine
from spydy.utils import check_configs
from spydy import urls, request, parsers, logs, store

myconfig = {
    "Globals":{
        "run_mode": "async_forever",
        "nworkers": "4"
    },
    "PipeLine":[urls.DummyUrls(url="https://dmoz-odp.org", repeat=10),
                request.AsyncHttpRequest(), parsers.DmozParser(), logs.MessageLog(), store.CsvStore(file_name=FILE_NAME)]
    }

check_configs(myconfig)
spider = Engine.from_dict(myconfig)
spider.run()

then run it :

$ python spider.py

Metadata

Release files for spydy 0.1.25

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for spydy 0.1.25
File	Size	Uploaded
spydy-0.1.25.tar.gz	16.6 kB	May 2, 2021	Details

Release files / spydy-0.1.25.tar.gz

Download URL	spydy-0.1.25.tar.gz
Size	16.6 kB
Tags	Source
SHA-256 checksum How to use checksums	`bee78e8e9a278b758961f87c463869b6e7fafc76dbc865a7bf4b3f7bde45edf7`
BLAKE2b-256 checksum How to use checksums	`0b7577f9eb441d24d345511f500e14e7cd01e99ac8b00380e4e8d8422ab4ecd6`
Upload date	May 2, 2021
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/3.1.1 pkginfo/1.5.0.1 requests/2.21.0 setuptools/41.4.0 requests-toolbelt/0.9.1 tqdm/4.48.0 CPython/3.7.4`