Pre-release
This release is a pre-release and may not be stable for production use.
🌐 ScrapeGraph Python SDK
Official Python SDK for the ScrapeGraph AI API - Smart web scraping powered by AI.
🚀 Features
- ✨ Smart web scraping with AI
- 🔄 Both sync and async clients
- 📊 Structured output with Pydantic schemas
- 🔍 Detailed logging with emojis
- ⚡ Automatic retries and error handling
- 🔐 Secure API authentication
📦 Installation
Using pip
pip install scrapegraph-py
Using uv
We recommend using uv to install the dependencies and pre-commit hooks.
# Install uv if you haven't already
pip install uv
# Install dependencies
uv sync
# Install pre-commit hooks
uv run pre-commit install
🔧 Quick Start
[!NOTE] If you prefer, you can use the environment variables to configure the API key and load them using
load_dotenv()
from scrapegraph_py import SyncClient
from scrapegraph_py.logger import get_logger
# Enable debug logging
logger = get_logger(level="DEBUG")
# Initialize client
sgai_client = SyncClient(api_key="your-api-key-here")
# Make a request
response = sgai_client.smartscraper(
website_url="https://example.com",
user_prompt="Extract the main heading and description"
)
print(response["result"])
🎯 Examples
Async Usage
import asyncio
from scrapegraph_py import AsyncClient
async def main():
async with AsyncClient(api_key="your-api-key-here") as sgai_client:
response = await sgai_client.smartscraper(
website_url="https://example.com",
user_prompt="Summarize the main content"
)
print(response["result"])
asyncio.run(main())
With Output Schema
from pydantic import BaseModel, Field
from scrapegraph_py import SyncClient
class WebsiteData(BaseModel):
title: str = Field(description="The page title")
description: str = Field(description="The meta description")
sgai_client = SyncClient(api_key="your-api-key-here")
response = sgai_client.smartscraper(
website_url="https://example.com",
user_prompt="Extract the title and description",
output_schema=WebsiteData
)
print(response["result"])
📚 Documentation
For detailed documentation, visit docs.scrapegraphai.com
🛠️ Development
Setup
- Clone the repository:
git clone https://github.com/ScrapeGraphAI/scrapegraph-sdk.git
cd scrapegraph-sdk/scrapegraph-py
- Install dependencies:
uv sync
- Install pre-commit hooks:
uv run pre-commit install
Running Tests
# Run all tests
uv run pytest
# Run specific test file
poetry run pytest tests/test_client.py
📝 License
This project is licensed under the MIT License - see the LICENSE file for details.
🤝 Contributing
Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.
- Fork the repository
- Create your feature branch (
git checkout -b feature/AmazingFeature) - Commit your changes (
git commit -m 'Add some AmazingFeature') - Push to the branch (
git push origin feature/AmazingFeature) - Open a Pull Request
🔗 Links
💬 Support
- 📧 Email: support@scrapegraphai.com
- 💻 GitHub Issues: Create an issue
- 🌟 Feature Requests: Request a feature
Made with ❤️ by ScrapeGraph AI
1 maintainer
Metadata
1 maintainer
Release files for scrapegraph-py 1.5.0b1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| scrapegraph_py-1.5.0b1.tar.gz | 106.0 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| scrapegraph_py-1.5.0b1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 118.3 kB
Release files / scrapegraph_py-1.5.0b1.tar.gz
| Download URL | scrapegraph_py-1.5.0b1.tar.gz |
|---|---|
| Size | 106.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
9e21a95f8455eb9813933b5272506e9e558453117f6ec76dea622cf7d4ebd40c
|
|
BLAKE2b-256 checksum How to use checksums |
6d6fb131a78a4164a31b84676f7bb19f251c7b3e873b9e695fd453821bbe54ba
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.10.12
|
Release files / scrapegraph_py-1.5.0b1-py3-none-any.whl
| Download URL | scrapegraph_py-1.5.0b1-py3-none-any.whl |
|---|---|
| Size | 12.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
4e55999b1cb1e895076ecf23a9e5dcfe481c5243359c740e6b3982cdef5aa9a3
|
|
BLAKE2b-256 checksum How to use checksums |
b2eace5f3dbac754534b7819dedd9d7dac05f8fb127a2c9373df12a17509731a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.10.12
|
