Python SDK for elcrawl
Project description
SDK for Python
The SDK for Python contains two parts:
- CLI Tool
- Utility Tools
CLI Tool
The CLI Tool is mainly designed for those who are more comfortable using command line tools.
The installation of the CLI Tool is:pip install elcrawlsdk
Then, you can use the elcrawl
command in the command prompt. elcrawl --help
Utility Tools
Utility tools mainly provide some helper
methods to make it easier for you to integrate your spiders into platform, e.g. saving results.
Scrapy Integration
In settings.py
in your Scrapy project, find the variable named ITEM_PIPELINES
(a dict
variable). Add content below.
ITEM_PIPELINES = {
'elcrawl.pipelines.ElcrawlMongoPipeline': 888,
}
Then, start the Scrapy spider. After it's done, you should be able to see scraped results in Task Detail -> Result
General Python Spider Integration
Please add below content to your spider files to save results.
# import result saving method
from elcrawl import save_item
# this is a result record, must be dict type
result = {'name': 'elcrawl'}
# call result saving method
save_item(result)
Then, start the spider. After it's done, you should be able to see scraped results in Task Detail -> Result
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for elcrawlsdk-1.0.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4112d1a693f43655eef80593d926e5d8f09d9c5f58af9d4da66cf18f7ac3cf05 |
|
MD5 | a770800028e4546f57fc95c10f4bf805 |
|
BLAKE2b-256 | f3a1f0b28044173714057aa9693aed0ea357f66a64a94c946b7f5447df922562 |