🌐 ScrapeGraph Python SDK
Official Python SDK for the ScrapeGraph AI API - Smart web scraping powered by AI.
🚀 Features
- ✨ Smart web scraping with AI
- 🔄 Both sync and async clients
- 📊 Structured output with Pydantic schemas
- 🔍 Detailed logging with emojis
- ⚡ Automatic retries and error handling
- 🔐 Secure API authentication
📦 Installation
Using pip
pip install scrapegraph-py
Using uv
We recommend using uv to install the dependencies and pre-commit hooks.
# Install uv if you haven't already
pip install uv
# Install dependencies
uv sync
# Install pre-commit hooks
uv run pre-commit install
🔧 Quick Start
[!NOTE] If you prefer, you can use the environment variables to configure the API key and load them using
load_dotenv()
from scrapegraph_py import SyncClient
from scrapegraph_py.logger import get_logger
# Enable debug logging
logger = get_logger(level="DEBUG")
# Initialize client
sgai_client = SyncClient(api_key="your-api-key-here")
# Make a request
response = sgai_client.smartscraper(
website_url="https://example.com",
user_prompt="Extract the main heading and description"
)
print(response["result"])
🎯 Examples
Async Usage
import asyncio
from scrapegraph_py import AsyncClient
async def main():
async with AsyncClient(api_key="your-api-key-here") as sgai_client:
response = await sgai_client.smartscraper(
website_url="https://example.com",
user_prompt="Summarize the main content"
)
print(response["result"])
asyncio.run(main())
With Output Schema
from pydantic import BaseModel, Field
from scrapegraph_py import SyncClient
class WebsiteData(BaseModel):
title: str = Field(description="The page title")
description: str = Field(description="The meta description")
sgai_client = SyncClient(api_key="your-api-key-here")
response = sgai_client.smartscraper(
website_url="https://example.com",
user_prompt="Extract the title and description",
output_schema=WebsiteData
)
print(response["result"])
📚 Documentation
For detailed documentation, visit docs.scrapegraphai.com
🛠️ Development
Setup
- Clone the repository:
git clone https://github.com/ScrapeGraphAI/scrapegraph-sdk.git
cd scrapegraph-sdk/scrapegraph-py
- Install dependencies:
uv sync
- Install pre-commit hooks:
uv run pre-commit install
Running Tests
# Run all tests
uv run pytest
# Run specific test file
poetry run pytest tests/test_client.py
📝 License
This project is licensed under the MIT License - see the LICENSE file for details.
🤝 Contributing
Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.
- Fork the repository
- Create your feature branch (
git checkout -b feature/AmazingFeature) - Commit your changes (
git commit -m 'Add some AmazingFeature') - Push to the branch (
git push origin feature/AmazingFeature) - Open a Pull Request
🔗 Links
💬 Support
- 📧 Email: support@scrapegraphai.com
- 💻 GitHub Issues: Create an issue
- 🌟 Feature Requests: Request a feature
Made with ❤️ by ScrapeGraph AI
1 maintainer
Metadata
1 maintainer
Release files for scrapegraph-py 1.3.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| scrapegraph_py-1.3.0.tar.gz | 104.7 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| scrapegraph_py-1.3.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 116.6 kB
Release files / scrapegraph_py-1.3.0.tar.gz
| Download URL | scrapegraph_py-1.3.0.tar.gz |
|---|---|
| Size | 104.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
4c09e1bb0a806f79874ced808582140a7578e7f8b044a8b493bb45026be8d554
|
|
BLAKE2b-256 checksum How to use checksums |
e819ecc2bb2bf698c5115ce1d85ab42806adafd2a95ab735f7b9f22f7f20e734
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.10.12
|
Release files / scrapegraph_py-1.3.0-py3-none-any.whl
| Download URL | scrapegraph_py-1.3.0-py3-none-any.whl |
|---|---|
| Size | 11.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
924a1c34797da6ad7b133c29763630b0d959976a03fb57d08b69f23e8985694d
|
|
BLAKE2b-256 checksum How to use checksums |
a2f7f4483c9bd24f374ceda9aafbc66d65e10777415f8f79b18de7ddd8b403e3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.10.12
|
