Scraping AI Python SDK (scraping-ai)

A modern, type-safe Python SDK for interacting with Scraping AI service API endpoints. It supports both synchronous and asynchronous operations using httpx and pydantic.

This SDK focuses strictly on core service execution endpoints (pipeline orchestration, web extraction, crawling, finder, keywords, etc.), excluding account management, billing, payments, and administrative routes.

Installation

pip install scraping-ai

Quick Start

1. One-Line Data Extraction

from scraping_ai import ScrapingAIClient

client = ScrapingAIClient(api_key="your_api_key_here")

# Extract web data directly in one step
data = client.extract(
    url="https://example.com",
    schema={"title": "string", "price": "number"}
)
print(data)

2. Standalone Service Modules (Keywords, Crawler, Finder, Extractor, etc.)

import asyncio
from scraping_ai import AsyncScrapingAIClient

async def main():
    async with AsyncScrapingAIClient(api_key="your_api_key_here") as client:
        # Start keyword generation
        run_info = await client.keywords.run(context="dummy_context", num_keywords=10)
        print(f"Accepted run for State ID {run_info.state_id}")

        # Check status
        status_info = await client.keywords.status(run_info.state_id)
        print(f"Current status: {status_info.status}")

        # Retrieve result when finished
        res = await client.keywords.result(run_info.state_id)
        print(f"Keywords generated: {res.keywords}")

asyncio.run(main())

3. Pipeline Orchestration

from scraping_ai import ScrapingAIClient

client = ScrapingAIClient(api_key="your_api_key_here")

# Create a multi-step pipeline job
state = client.pipeline.create(
    base_url="https://example.com",
    user_instruction="dummy_user_instruction",
    schema_instruction="dummy_schema_instruction"
)
flow = client.pipeline.run_flow(state.id)
print(f"Pipeline started: {flow.first_step}")

Service Modules Overview

The SDK exposes all service operations under clean client modules:

• client.extract(url, ...): One-step extraction helper.
• client.generate_keywords(context, ...): One-step keyword generation helper.
• client.pipeline: Pipeline creation, orchestration, auto-flow, and status polling.
• client.keywords: Standalone keyword generation module (run, status, result).
• client.finder: URL discovery module (run, status, result).
• client.crawler: Page crawling module (run, status, result).
• client.extractor: Data extraction module (run, status, result).
• client.ranker: Relevance ranking module (run, status, result).
• client.schema: JSON Schema generation module (run, status, result).
• client.search_url: Search URL generator module (run, status, result).
• client.search_results: Search results discovery module (run, status, result).
• client.exports: S3 exports management (create, list, get).
• client.data: Query extracted records (get_by_state, export_by_state).