Scraping AI Python SDK (scraping-ai)

A modern, type-safe Python SDK for interacting with the Scraping AI core service API endpoints. It supports both synchronous and asynchronous operations using httpx and pydantic.

This SDK focuses strictly on core service execution endpoints (v1 pipeline orchestration, data extraction, and v2 standalone modules), excluding account management, billing, payments, and administrative routes.

Installation

pip install scraping-ai

Core Service Usage

1. v1 Unified Pipeline Flow

from scraping_ai import ScrapingAIClient

# Initialize client using API Key
client = ScrapingAIClient(api_key="your_api_key_here")

# Create an InputState
state = client.input_states.create(
    base_url="https://example.com",
    user_instruction="dummy_user_instruction",
    schema_instruction="dummy_schema_instruction"
)
print(f"Created input state ID: {state.id}")

# Trigger the auto-flow pipeline
flow = client.input_states.run_flow(state.id)
print(f"Flow started. First step: {flow.first_step}")

# Wait for extraction step to finish
state = client.input_states.wait_for_task(state.id, "extractor", poll_interval=5.0)
print(f"Extraction status: {state.extraction_status}")

# Fetch extracted data rows
data_res = client.data.get_by_state(state.id)
for row in data_res.results:
    print(row.data)

2. v2 Standalone Async Modules

import asyncio
from scraping_ai import AsyncScrapingAIClient

async def main():
    async with AsyncScrapingAIClient(api_key="your_api_key_here") as client:
        # Start a v2 keyword generation run
        run_info = await client.v2_keywords.run(context="dummy_context", num_keywords=10)
        print(f"Accepted run for State ID {run_info.state_id}")

        # Poll status
        status_info = await client.v2_keywords.status(run_info.state_id)
        print(f"Current status: {status_info.status}")

        # Retrieve result when finished
        res = await client.v2_keywords.result(run_info.state_id)
        print(f"Keywords generated: {res.keywords}")

asyncio.run(main())

Core Service Operations Summary

The SDK exposes service operations under core namespaces:

v1 Pipeline & Data Operations

1. client.input_states: Create, list, retrieve, update, patch, trigger run_flow, and wait_for_task helper.
2. client.tasks: List available background tasks (list_available) and trigger tasks (run).
3. client.task_entries: Create an InputState and run a task in one call (create), or query task status (get_status).
4. client.data: Retrieve extracted rows (get_by_state) and download CSV exports (export_by_state).

v2 Standalone Service Modules

1. client.v2_keywords: run, status, result for standalone keyword generation.
2. client.v2_finder: run, status, result for URL discovery.
3. client.v2_crawler: run, status, result for page crawling.
4. client.v2_extractor: run, status, result for LLM data extraction.
5. client.v2_exports: create, list, get for presigned S3 exports.
6. client.v2_ranker: run, status, result for URL relevance ranking.
7. client.v2_schema: run, status, result for JSON Schema generation.
8. client.v2_search_url: run, status, result for search URL generation.
9. client.v2_search_results: run, status, result for search results finder.