Python SDK for SpiderWebAI API
Project description
SpiderWebAI Python SDK
The SpiderWebAI Python SDK offers a toolkit for straightforward website scraping and crawling at scale, enabling you to collect data formatted for compatibility with language models (LLMs). It features a user-friendly interface for seamless integration with the SpiderWebAI API.
Installation
To install the spiderwebai Python SDK, you can use pip:
pip install spiderwebai-py
Usage
- Get an API key from spiderwebai.xyz
- Set the API key as an environment variable named
SPIDER_API_KEY
or pass it as a parameter to theSpiderWebAIApp
class.
Here's an example of how to use the SDK:
from spiderwebai import SpiderWebAIApp
# Initialize the SpiderWebAIApp with your API key
app = SpiderWebAIApp(api_key='your_api_key')
# Scrape a single URL
url = 'https://spiderwebai.xyz'
scraped_data = app.scrape_url(url)
# Crawl a website
crawl_url = 'https://spiderwebai.xyz'
params = {
'limit': 1,
'proxy_enabled': True,
'store_data': False,
'metadata': False,
'request': 'http'
}
crawl_result = app.crawl_url(crawl_url, params=params)
Scraping a URL
To scrape a single URL, use the scrape_url
method. It takes the URL as a parameter and returns the scraped data as a dictionary.
url = 'https://example.com'
scraped_data = app.scrape_url(url)
Crawling a Website
To crawl a website, use the crawl_url
method. It takes the starting URL and optional parameters as arguments. The params
argument allows you to specify additional options for the crawl, such as the maximum number of pages to crawl, allowed domains, and the output format.
crawl_url = 'https://example.com'
params = {
'limit': 200,
'request': 'smart_mode'
}
crawl_result = app.crawl_url(crawl_url, params=params)
Error Handling
The SDK handles errors returned by the SpiderWebAI API and raises appropriate exceptions. If an error occurs during a request, an exception will be raised with a descriptive error message.
Contributing
Contributions to the SpiderWebAI Python SDK are welcome! If you find any issues or have suggestions for improvements, please open an issue or submit a pull request on the GitHub repository.
License
The SpiderWebAI Python SDK is open-source and released under the MIT License.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for spiderwebai_py-0.0.8-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1f7b5f9ca4c93f6cd6e6120863895fe7db19d1cf38c13debbfe2140dfa014664 |
|
MD5 | 338250d819ee664450e466a079513fd2 |
|
BLAKE2b-256 | 068b91da906ee0d422208d32f46d83dc982caf37f99b17bd8c34cedaf3a299dd |