scrapix 0.1.2

Scrapix

Scrapix Python SDK

🕷️ A powerful Python SDK for web scraping, crawling, and data extraction

Installation • Quick Start • API Reference • Examples • Documentation

---

Overview

Scrapix is a Python SDK that provides a simple, powerful interface for web scraping and data extraction. Built on top of the Scrapix API, it offers:

• 🌐 Web Scraping - Extract content from any webpage in multiple formats
• 🔗 URL Collection - Discover and collect URLs from websites and sitemaps
• 🕸️ Web Crawling - Crawl websites to extract URLs with advanced filtering
• 🧠 AI-Powered Extraction - Extract structured data using AI with custom schemas
• ⚡ Built-in Resilience - Automatic retries, timeouts, and error handling
• 🔒 Premium Features - Captcha solving, premium proxies, and JavaScript rendering

Requirements

• Python 3.9+

Installation

Install the package via pip:

pip install scrapix

Or with Poetry:

poetry add scrapix

Quick Start

1. Configure the Client

import os
import scrapix

# Configure the SDK with your API key
configuration = scrapix.Configuration(
    host="https://api-scrapix.promptcloud.com",  # Your Scrapix API host
    api_key={"APIKeyHeader": os.environ.get("SCRAPIX_API_KEY")}
)

2. Scrape a Webpage

with scrapix.ApiClient(configuration) as api_client:
    api = scrapix.APIServicesApi(api_client)
    
    # Scrape a webpage
    result = api.scrape(scrapix.ScrapeInput(
        url="https://example.com",
        output_format=scrapix.OutputFormat.MARKDOWN
    ))
    
    print(result.data)

API Reference

The SDK provides access to the following API endpoints:

Method	Endpoint	Description
scrape()	POST /v1/scrape	Scrape content from a single URL
crawl()	POST /v1/crawl	Crawl a website and collect URLs
collect()	POST /v1/collect	Collect URLs from a page
extract()	POST /v1/extract	Extract structured data using AI
echo()	POST /v1/echo	Test API connectivity

Authentication

All API requests require an API key. Set it using the APIKeyHeader:

configuration = scrapix.Configuration(
    host="https://scrapix.promptcloud.com",
    api_key={"APIKeyHeader": "your-api-key-here"}
)

Examples

Scraping Content

Scrape a webpage and get the content in different formats:

import scrapix

configuration = scrapix.Configuration(
    host="https://scrapix.promptcloud.com",
    api_key={"APIKeyHeader": "your-api-key"}
)

with scrapix.ApiClient(configuration) as api_client:
    api = scrapix.APIServicesApi(api_client)
    
    # Scrape with JavaScript rendering enabled
    result = api.scrape(scrapix.ScrapeInput(
        url="https://example.com",
        render=True,  # Enable JavaScript rendering
        output_format=scrapix.OutputFormat.MARKDOWN,
        timeout=60,
        max_retries=3
    ))
    
    print(f"Content: {result.data}")

Crawling a Website

Discover URLs on a website:

import scrapix

with scrapix.ApiClient(configuration) as api_client:
    api = scrapix.APIServicesApi(api_client)
    
    # Crawl a website for URLs
    result = api.crawl(scrapix.CrawlInput(
        url="https://example.com",
        urls_limit=50,  # Maximum URLs to collect
        include_sitemap_urls=True,  # Include URLs from sitemap
        include_paths="/blog/*",  # Only include blog pages
        exclude_paths="/admin/*"  # Exclude admin pages
    ))
    
    for url in result.urls:
        print(url)

Collecting URLs

Collect all URLs from a specific page:

import scrapix

with scrapix.ApiClient(configuration) as api_client:
    api = scrapix.APIServicesApi(api_client)
    
    # Collect URLs from a page
    result = api.collect(scrapix.CollectInput(
        url="https://example.com/sitemap",
        urls_limit=1000,
        include_sitemap_urls=True
    ))
    
    print(f"Found {len(result.urls)} URLs")

AI-Powered Data Extraction

Extract structured data using AI:

import scrapix

with scrapix.ApiClient(configuration) as api_client:
    api = scrapix.APIServicesApi(api_client)
    
    # Extract structured data with a query
    result = api.extract(scrapix.ExtractInput(
        url="https://example.com/products",
        query="Extract all product names, prices, and descriptions",
        output_format=scrapix.StructuredOutputFormat.JSON
    ))
    
    print(result.data)

Using Premium Features

Enable advanced features like proxies and captcha solving:

import scrapix

with scrapix.ApiClient(configuration) as api_client:
    api = scrapix.APIServicesApi(api_client)
    
    result = api.scrape(scrapix.ScrapeInput(
        url="https://protected-site.com",
        render=True,  # JavaScript rendering
        premium_proxies=True,  # Use premium residential proxies
        use_captcha_solver=True,  # Automatically solve CAPTCHAs
        use_cache=False  # Always fetch fresh content
    ))

Error Handling

Handle API errors gracefully:

import scrapix
from scrapix.rest import ApiException

with scrapix.ApiClient(configuration) as api_client:
    api = scrapix.APIServicesApi(api_client)
    
    try:
        result = api.scrape(scrapix.ScrapeInput(url="https://example.com"))
        print(result.data)
    except ApiException as e:
        print(f"API Error: {e.status} - {e.reason}")
        print(f"Response body: {e.body}")
    except Exception as e:
        print(f"Unexpected error: {e}")

Models

Input Models

Model	Description
ScrapeInput	Input for scraping a single URL
CrawlInput	Input for crawling a website
CollectInput	Input for collecting URLs
ExtractInput	Input for AI-powered extraction

Result Models

Model	Description
ScrapeResult	Result from scrape operation
CrawlResult	Result from crawl operation
CollectResult	Result from collect operation
ExtractResult	Result from extract operation

Other Models

Model	Description
OutputFormat	Output format options (MARKDOWN, HTML, etc.)
StructuredOutputFormat	Structured output formats
StructuredOutputSchema	Schema for structured outputs
SummarizeSchema	Schema for summarization

Common Parameters

All input models share these common parameters:

Parameter	Type	Default	Description
url	str	required	The URL to scrape/crawl
timeout	int	40	Request timeout in seconds
max_retries	int	5	Maximum retry attempts
render	bool	False	Enable JavaScript rendering
premium_proxies	bool	False	Use premium residential proxies
use_captcha_solver	bool	False	Enable automatic CAPTCHA solving
use_cache	bool	True	Use cached responses when available

Documentation

For detailed API documentation, see the docs directory:

• APIServicesApi - Main API endpoints
• DefaultApi - Default API endpoints

Development

Running Tests

# Install dev dependencies
pip install -e ".[dev]"

# Run tests
pytest

Type Checking

mypy scrapix

Support

• 📧 Email: [e-mail]
• 🌐 Website: https://scrapix.promptcloud.com

License

This SDK is provided by PromptCloud. See the LICENSE file for details.