Python SDK for Spider Cloud API
Project description
Spider Cloud Python SDK
The Spider Cloud Python SDK offers a toolkit for straightforward website scraping, crawling at scale, and other utilities like extracting links and taking screenshots, enabling you to collect data formatted for compatibility with language models (LLMs). It features a user-friendly interface for seamless integration with the Spider Cloud API.
Installation
To install the Spider Cloud Python SDK, you can use pip:
pip install spider_client
Usage
- Get an API key from spider.cloud
- Set the API key as an environment variable named
SPIDER_API_KEYor pass it as a parameter to theSpiderclass.
Here's an example of how to use the SDK:
from spider import Spider
# Initialize the Spider with your API key
app = Spider(api_key='your_api_key')
# Scrape a single URL
url = 'https://spider.cloud'
scraped_data = app.scrape_url(url)
# Crawl a website
crawler_params = {
'limit': 1,
'proxy_enabled': True,
'metadata': False,
'request': 'http'
}
crawl_result = app.crawl_url(url, params=crawler_params)
Scraping a URL
To scrape data from a single URL:
url = 'https://example.com'
scraped_data = app.scrape_url(url)
Crawling a Website
To automate crawling a website:
url = 'https://example.com'
crawl_params = {
'limit': 200,
'request': 'smart_mode'
}
crawl_result = app.crawl_url(url, params=crawl_params)
Crawl Streaming
Stream crawl the website in chunks to scale.
def handle_json(json_obj: dict) -> None:
assert json_obj["url"] is not None
url = 'https://example.com'
crawl_params = {
'limit': 200,
}
response = app.crawl_url(
url,
params=params,
stream=True,
callback=handle_json,
)
Search
Perform a search for websites to crawl or gather search results:
query = 'a sports website'
crawl_params = {
'request': 'smart_mode',
'search_limit': 5,
'limit': 5,
'fetch_page_content': True
}
crawl_result = app.search(query, params=crawl_params)
Retrieving Links from a URL(s)
Extract all links from a specified URL:
url = 'https://example.com'
links = app.links(url)
Transform
Transform HTML to markdown or text lightning fast:
data = [ { 'html': '<html><body><h1>Hello world</h1></body></html>' } ]
params = {
'readability': False,
'return_format': 'markdown',
}
result = app.transform(data, params=params)
Taking Screenshots of a URL(s)
Capture a screenshot of a given URL:
url = 'https://example.com'
screenshot = app.screenshot(url)
Checking Available Credits
You can check the remaining credits on your account:
credits = app.get_credits()
Unblocker
Access blocked or protected content with anti-bot bypass:
url = 'https://protected-site.com'
result = app.unblocker(url)
Unblocker with AI Extraction
Unblock and extract structured data using AI:
url = 'https://protected-site.com/products'
result = app.unblocker(url, params={
'custom_prompt': 'Extract all product names and prices as JSON'
})
# Extracted data is available in result[0]['metadata']['extracted_data']
Unblocker with JSON Schema Extraction
Use JSON Schema for structured, validated extraction output:
url = 'https://protected-site.com/products'
result = app.unblocker(url, params={
'extraction_schema': {
'name': 'products',
'description': 'Product listing extraction',
'schema': '''{
"type": "object",
"properties": {
"products": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {"type": "string"},
"price": {"type": "number"}
},
"required": ["name", "price"]
}
}
}
}''',
'strict': True
}
})
# Extracted data conforms to the schema in result[0]['metadata']['extracted_data']
AI Studio Methods
AI Studio methods require an active AI Studio subscription. See spider.cloud/ai/pricing for plans.
AI Crawl
AI-guided crawling using natural language prompts:
result = app.ai_crawl(
url='https://example.com',
prompt='Find all blog posts and extract titles and summaries'
)
AI Scrape
AI-guided scraping using natural language prompts:
result = app.ai_scrape(
url='https://example.com/products',
prompt='Extract all product names, prices, and descriptions'
)
AI Search
AI-enhanced web search using natural language:
result = app.ai_search(prompt='Find the best Python web scraping libraries')
AI Browser
AI-guided browser automation:
result = app.ai_browser(
url='https://example.com/login',
prompt='Click the sign in button and fill the email field with test@example.com'
)
AI Links
AI-guided link extraction and filtering:
result = app.ai_links(
url='https://example.com',
prompt='Find all links to product pages and documentation'
)
Unlimited Methods
Unlimited methods require an active Unlimited subscription. See spider.cloud/pricing?plan=unlimited for plans. The plan is a flat monthly rate billed by purchased concurrency seats (the number of requests in flight at once) instead of per-request credits.
Unlimited Scrape
Scrape a single URL on the Unlimited plan:
url = 'https://example.com'
scraped_data = app.unlimited_scrape(url)
Unlimited Crawl
Crawl a website on the Unlimited plan:
url = 'https://example.com'
crawl_params = {
'limit': 200,
'request': 'smart_mode'
}
crawl_result = app.unlimited_crawl(url, params=crawl_params)
There is no queueing: when all purchased seats are in flight, the API returns an immediate 429 with a Retry-After header — retry with backoff. Every response carries X-Concurrency-Limit and X-Concurrency-Active headers. AI/LLM extraction params (prompt, custom_prompt, extraction_prompt, extraction_schema, model/vision params) are rejected with a 400; AI usage is billed separately via the ai_* methods. See the Unlimited API docs for details.
Streaming
If you need to stream the request use the third param:
url = 'https://example.com'
crawler_params = {
'limit': 1,
'proxy_enabled': True,
'metadata': False,
'request': 'http'
}
links = app.links(url, crawler_params, True)
Content-Type
The following Content-type headers are supported using the fourth param:
application/jsontext/csvapplication/xmlapplication/jsonl
url = 'https://example.com'
crawler_params = {
'limit': 1,
'proxy_enabled': True,
'metadata': False,
'request': 'http'
}
# stream json lines back to the client
links = app.crawl(url, crawler_params, True, "application/jsonl")
Error Handling
The SDK handles errors returned by the Spider Cloud API and raises appropriate exceptions. If an error occurs during a request, an exception will be raised with a descriptive error message.
Contributing
Contributions to the Spider Cloud Python SDK are welcome! If you find any issues or have suggestions for improvements, please open an issue or submit a pull request on the GitHub repository.
License
The Spider Cloud Python SDK is open-source and released under the MIT License.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file spider_client-0.1.91-py3-none-any.whl.
File metadata
- Download URL: spider_client-0.1.91-py3-none-any.whl
- Upload date:
- Size: 19.1 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.0.0 CPython/3.12.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
52dc48b3eaad054dafce384ab81191931a2fdb2810ec78c21a017aa889d42d04
|
|
| MD5 |
807169c1095585ea8e829c20dcf9ddf9
|
|
| BLAKE2b-256 |
c9d9e741ce0559674750c392def84fd26d1d014d1d4a6d1db438d7dba2d1b64e
|
