🌐 ScrapeGraph Python SDK
Official Python SDK for the ScrapeGraph AI API - Smart web scraping powered by AI.
🚀 Features
- ✨ Smart web scraping with AI
- 🔄 Both sync and async clients
- 📊 Structured output with Pydantic schemas
- 🔍 Detailed logging with emojis
- ⚡ Automatic retries and error handling
- 🔐 Secure API authentication
📦 Installation
Using pip
pip install scrapegraph-py
Using uv
We recommend using uv to install the dependencies and pre-commit hooks.
# Install uv if you haven't already
pip install uv
# Install dependencies
uv sync
# Install pre-commit hooks
uv run pre-commit install
🔧 Quick Start
[!NOTE] If you prefer, you can use the environment variables to configure the API key and load them using
load_dotenv()
from scrapegraph_py import SyncClient
from scrapegraph_py.logger import get_logger
# Enable debug logging
logger = get_logger(level="DEBUG")
# Initialize client
sgai_client = SyncClient(api_key="your-api-key-here")
# Make a request
response = sgai_client.smartscraper(
website_url="https://example.com",
user_prompt="Extract the main heading and description"
)
print(response["result"])
🎯 Examples
Async Usage
import asyncio
from scrapegraph_py import AsyncClient
async def main():
async with AsyncClient(api_key="your-api-key-here") as sgai_client:
response = await sgai_client.smartscraper(
website_url="https://example.com",
user_prompt="Summarize the main content"
)
print(response["result"])
asyncio.run(main())
With Output Schema
from pydantic import BaseModel, Field
from scrapegraph_py import SyncClient
class WebsiteData(BaseModel):
title: str = Field(description="The page title")
description: str = Field(description="The meta description")
sgai_client = SyncClient(api_key="your-api-key-here")
response = sgai_client.smartscraper(
website_url="https://example.com",
user_prompt="Extract the title and description",
output_schema=WebsiteData
)
print(response["result"])
📚 Documentation
For detailed documentation, visit docs.scrapegraphai.com
🛠️ Development
Setup
- Clone the repository:
git clone https://github.com/ScrapeGraphAI/scrapegraph-sdk.git
cd scrapegraph-sdk/scrapegraph-py
- Install dependencies:
uv sync
- Install pre-commit hooks:
uv run pre-commit install
Running Tests
# Run all tests
uv run pytest
# Run specific test file
poetry run pytest tests/test_client.py
📝 License
This project is licensed under the MIT License - see the LICENSE file for details.
🤝 Contributing
Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.
- Fork the repository
- Create your feature branch (
git checkout -b feature/AmazingFeature) - Commit your changes (
git commit -m 'Add some AmazingFeature') - Push to the branch (
git push origin feature/AmazingFeature) - Open a Pull Request
🔗 Links
💬 Support
- 📧 Email: support@scrapegraphai.com
- 💻 GitHub Issues: Create an issue
- 🌟 Feature Requests: Request a feature
Made with ❤️ by ScrapeGraph AI
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file scrapegraph_py-1.3.0.tar.gz.
File metadata
- Download URL: scrapegraph_py-1.3.0.tar.gz
- Upload date:
- Size: 104.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.10.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
4c09e1bb0a806f79874ced808582140a7578e7f8b044a8b493bb45026be8d554
|
|
| MD5 |
949d5225351127979a4d53d9c0e83a8a
|
|
| BLAKE2b-256 |
e819ecc2bb2bf698c5115ce1d85ab42806adafd2a95ab735f7b9f22f7f20e734
|
File details
Details for the file scrapegraph_py-1.3.0-py3-none-any.whl.
File metadata
- Download URL: scrapegraph_py-1.3.0-py3-none-any.whl
- Upload date:
- Size: 11.9 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.10.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
924a1c34797da6ad7b133c29763630b0d959976a03fb57d08b69f23e8985694d
|
|
| MD5 |
96534de022f4d55b24df2db1f61ff7fd
|
|
| BLAKE2b-256 |
a2f7f4483c9bd24f374ceda9aafbc66d65e10777415f8f79b18de7ddd8b403e3
|
