teyespider-sdk 0.0.1

Python SDK

TeyeSpider SDK for Python

中文 | English

The SDK for Python contains two parts:

1. CLI Tool
2. Utility Tools

CLI Tool

The CLI Tool is mainly designed for those who are more comfortable using command line tools to interact with teye-spider.

The installation of the CLI Tool is simple:

pip install teyespider-sdk

Then, you can use the teye-spider command in the command prompt to action with teye-spider.

Check the help document below, or you can refer to the official documentation (Chinese).

teye-spider --help

Utility Tools

Utility tools mainly provide some helper methods to make it easier for you to integrate your spiders into teye-spider, e.g. saving results.

Below are integration methods of Scrapy and general Python spiders with teye-spider.

⚠️Note: make sure you have already installed teye-spider-sdk using pip.

Scrapy Integration

In settings.py in your Scrapy project, find the variable named ITEM_PIPELINES (a dict variable). Add content below.

ITEM_PIPELINES = {
    'teyespider.pipelines.teye-spiderMongoPipeline': 888,
}

Then, start the Scrapy spider. After it's done, you should be able to see scraped results in Task Detail -> Result

General Python Spider Integration

Please add below content to your spider files to save results.

# import result saving method
from teyespider import save_item

# this is a result record, must be dict type
result = {'name': 'teyespider'}

# call result saving method
save_item(result)

Then, start the spider. After it's done, you should be able to see scraped results in Task Detail -> Result