Crawlab SDK for Python
中文 | English
The SDK for Python contains two parts:
- CLI Tool
- Utility Tools
CLI Tool
The CLI Tool is mainly designed for those who are more comfortable using command line tools to interact with Crawlab.
The installation of the CLI Tool is simple:
pip install crawlab-sdk
Then, you can use the crawlab command in the command prompt to action with Crawlab.
Check the help document below, or you can refer to the official documentation (Chinese).
crawlab --help
Utility Tools
Utility tools mainly provide some helper methods to make it easier for you to integrate your spiders into Crawlab, e.g. saving results.
Below are integration methods of Scrapy and general Python spiders with Crawlab.
⚠️Note: make sure you have already installed crawlab-sdk using pip.
Scrapy Integration
In settings.py in your Scrapy project, find the variable named ITEM_PIPELINES (a dict variable). Add content below.
ITEM_PIPELINES = {
'crawlab.pipelines.CrawlabMongoPipeline': 888,
}
Then, start the Scrapy spider. After it's done, you should be able to see scraped results in Task Detail -> Result
General Python Spider Integration
Please add below content to your spider files to save results.
# import result saving method
from crawlab import save_item
# this is a result record, must be dict type
result = {'name': 'crawlab'}
# call result saving method
save_item(result)
Then, start the spider. After it's done, you should be able to see scraped results in Task Detail -> Result
1 maintainer
Metadata
1 maintainer
Release files for crawlab-sdk 0.6.0.post11
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| crawlab-sdk-0.6.0.post11.tar.gz | 22.4 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| crawlab_sdk-0.6.0.post11-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 71.0 kB
Release files / crawlab-sdk-0.6.0.post11.tar.gz
| Download URL | crawlab-sdk-0.6.0.post11.tar.gz |
|---|---|
| Size | 22.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
49e5a4addf7353c7a3074e4f2186e0e402d5c0daa1644c0ee2ad6c3b9f05a879
|
|
BLAKE2b-256 checksum How to use checksums |
f2d50cbbe7e4ccfe5833c5e56b3dd9d48b188d5bd080611753ce7f9d52f055c0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.8.15
|
Release files / crawlab_sdk-0.6.0.post11-py3-none-any.whl
| Download URL | crawlab_sdk-0.6.0.post11-py3-none-any.whl |
|---|---|
| Size | 48.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
8677f7c8743bfeb4cb49723ce0fa2feed0940e7bd5205883577a7cec1aa23802
|
|
BLAKE2b-256 checksum How to use checksums |
01810c1aaddea79f8c22a69ec06077ae5bc88b3209c8e88908725f5b68bb5677
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.8.15
|
