Scraping AI Python SDK (scraping-ai)

A modern, type-safe Python SDK for interacting with Scraping AI service API endpoints. It supports both synchronous and asynchronous operations using httpx and pydantic.

This SDK focuses strictly on core service execution endpoints (pipeline orchestration, web extraction, crawling, finder, keywords, etc.), excluding account management, billing, payments, and administrative routes.

Installation

pip install scraping-ai

Quick Start (High-Level 1-Line Helpers)

1. Web Extraction in 1 Line

from scraping_ai import ScrapingAIClient

client = ScrapingAIClient(api_key="your_api_key_here")

# Extract web data directly in one step (polls until finished)
data = client.extract(
    url="https://example.com",
    schema={"title": "string", "price": "number"}
)
print(data.results)

2. Page Crawling in 1 Line

# Crawl a web page and get extracted content & markdown
crawl_result = client.crawl(url="https://example.com", use_browser=True)
print(crawl_result.results)

3. Discover URLs in 1 Line

# Discover URLs on a target site up to a specific depth
found_urls = client.find_urls(base_url="https://example.com", max_depth=2)
print(found_urls.results)

4. Keyword Generation in 1 Line

# Generate relevant search keywords for a topic
keywords_res = client.generate_keywords(context="dummy_context", num_keywords=15)
print(keywords_res.keywords)

5. Schema Generation in 1 Line

# Generate JSON Schema for an extraction prompt
schema_res = client.generate_schema(schema_instruction="Extract title, author, date")
print(schema_res.schema_)

Asynchronous Usage

import asyncio
from scraping_ai import AsyncScrapingAIClient

async def main():
    async with AsyncScrapingAIClient(api_key="your_api_key_here") as client:
        # Async one-liner extraction
        data = await client.extract(url="https://example.com")
        print(data.results)

asyncio.run(main())

Service Modules Overview

The SDK exposes low-level module controllers under clean, intuitive client attributes:

• 1-Line Helpers: client.extract(...), client.crawl(...), client.find_urls(...), client.generate_keywords(...), client.generate_schema(...).
• client.extractor: Data extraction module (run, status, result).
• client.crawler: Page crawling module (run, status, result).
• client.finder: URL discovery module (run, status, result).
• client.keywords: Keyword generation module (run, status, result).
• client.schema: JSON Schema generation module (run, status, result).
• client.ranker: Relevance ranking module (run, status, result).
• client.search_url: Search URL generator module (run, status, result).
• client.search_results: Search results discovery module (run, status, result).
• client.exports: S3 exports management (create, list, get).
• client.pipeline: Multi-step pipeline orchestration (create, list, get, patch, run_flow, wait_for_task).
• client.data: Query extracted records (get_by_state, export_by_state).