Crawlab AI SDK
This is the Python SDK for Crawlab AI, an AI-powered web scraping platform maintained by Crawlab.
Installation
pip install crawlab-ai
Pre-requisites
An API token is required to use this SDK. You can get the API token from the Crawlab official website.
Usage
Get data from a list page
from crawlab_ai import read_list
# Define the URL and fields
url = "https://example.com"
# Get the data without specifying fields
df = read_list(url=url)
print(df)
# You can also specify fields
fields = ["title", "content"]
df = read_list(url=url, fields=fields)
# You can also return a list of dictionaries instead of a DataFrame
data = read_list(url=url, as_dataframe=False)
print(data)
Usage with Scrapy
Create a Scrapy spider by extending ScrapyListSpider:
from crawlab_ai import ScrapyListSpider
class MySpider(ScrapyListSpider):
name = "my_spider"
start_urls = ["https://example.com"]
fields = ["title", "content"]
Then run the spider:
scrapy crawl my_spider
1 maintainer
Metadata
1 maintainer
Release files for crawlab-ai 0.0.10
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| crawlab-ai-0.0.10.tar.gz | 9.4 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| crawlab_ai-0.0.10-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 23.6 kB
Release files / crawlab-ai-0.0.10.tar.gz
| Download URL | crawlab-ai-0.0.10.tar.gz |
|---|---|
| Size | 9.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
b211f8d597bbcaa9e889448d569cbe9db08a6c879e30879742a96edb60826254
|
|
BLAKE2b-256 checksum How to use checksums |
2235ae028accd56b7dfc00d88eef4d4139fa31f2a506a2bd2040152aca7e1a76
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.0 CPython/3.10.14
|
Release files / crawlab_ai-0.0.10-py3-none-any.whl
| Download URL | crawlab_ai-0.0.10-py3-none-any.whl |
|---|---|
| Size | 14.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
d155b529dc9fd76d3de79046aab948e907760a602e3472d0fcd69b4f976eda06
|
|
BLAKE2b-256 checksum How to use checksums |
0c8acf2cc283d802ed8a51c36cb40d10204c1afa81e8bb0102ca5810ce81cee8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.0 CPython/3.10.14
|
