Skip to main content

Python SDK for Crawlab

Project description

Crawlab SDK for Python

中文 | English

The SDK for Python contains two parts:

  1. CLI Tool
  2. Utility Tools

CLI Tool

The CLI Tool is mainly designed for those who are more comfortable using command line tools to interact with Crawlab.

The installation of the CLI Tool is simple:

pip install crawlab-sdk

Then, you can use the crawlab command in the command prompt to action with Crawlab.

Check the help document below, or you can refer to the official documentation (Chinese).

crawlab --help

Utility Tools

Utility tools mainly provide some helper methods to make it easier for you to integrate your spiders into Crawlab, e.g. saving results.

Below are integration methods of Scrapy and general Python spiders with Crawlab.

⚠️Note: make sure you have already installed crawlab-sdk using pip.

Scrapy Integration

In settings.py in your Scrapy project, find the variable named ITEM_PIPELINES (a dict variable). Add content below.

ITEM_PIPELINES = {
    'crawlab.pipelines.CrawlabMongoPipeline': 888,
}

Then, start the Scrapy spider. After it's done, you should be able to see scraped results in Task Detail -> Result

General Python Spider Integration

Please add below content to your spider files to save results.

# import result saving method
from crawlab import save_item

# this is a result record, must be dict type
result = {'name': 'crawlab'}

# call result saving method
save_item(result)

Then, start the spider. After it's done, you should be able to see scraped results in Task Detail -> Result

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

crawlab-sdk-0.6.2.tar.gz (22.8 kB view details)

Uploaded Source

Built Distribution

crawlab_sdk-0.6.2-py3-none-any.whl (50.7 kB view details)

Uploaded Python 3

File details

Details for the file crawlab-sdk-0.6.2.tar.gz.

File metadata

  • Download URL: crawlab-sdk-0.6.2.tar.gz
  • Upload date:
  • Size: 22.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.10.12

File hashes

Hashes for crawlab-sdk-0.6.2.tar.gz
Algorithm Hash digest
SHA256 372d187ce4f6c3f7bbe881db6ea86940dec6684982c866a0297ca20724648e7b
MD5 8141de25dd4b6b0a6f4e1d0eafb10ff1
BLAKE2b-256 280a125ede64c2521ba098766adf208b7048c1098e0f24d1084e855fcc3f5c84

See more details on using hashes here.

File details

Details for the file crawlab_sdk-0.6.2-py3-none-any.whl.

File metadata

  • Download URL: crawlab_sdk-0.6.2-py3-none-any.whl
  • Upload date:
  • Size: 50.7 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.10.12

File hashes

Hashes for crawlab_sdk-0.6.2-py3-none-any.whl
Algorithm Hash digest
SHA256 febe5d42f670d7e3723094d4a02628d21840aaa679cc14c7243b238a3fbc2bf1
MD5 cf1b982307363e8e2002900e8a5e56ef
BLAKE2b-256 9adeac3e1b59bcad6339f93fcbdcf8ec8d58e9e08ad7f6788de59ba3ff7f32a5

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page