ssscraper 0.1.0

Async web scraping library for SiteSudharo — downloads resources to local/S3/R2 with full metadata

SSscraper

Async web scraping library for SiteSudharo.
Built on Crawl4ai — downloads page resources to local disk, AWS S3, or Cloudflare R2 and returns structured metadata for every file.

Features

• Full-page scraping via Crawl4ai (headless browser)
• Downloads: images, CSS, fonts, JS, documents, video, audio
• Storage backends: Local · S3 · R2 (drop-in swappable)
• Per-resource metadata: URL, stored path, content-type, size, MD5 + SHA-256
• Async & concurrent — configurable parallelism
• Retry with backoff, per-file size cap

Quick start

import asyncio
from ssscraper import SScraper, LocalStorage, ScraperConfig

async def main():
    scraper = SScraper(
        storage=LocalStorage("./downloads"),
        config=ScraperConfig(download_images=True, download_css=True, download_fonts=True),
    )
    result = await scraper.scrape("https://example.com")
    print(f"Downloaded {len(result.succeeded)} resources")
    for r in result.resources:
        print(r.model_dump_json())

asyncio.run(main())

Storage backends

Local

from ssscraper import LocalStorage
storage = LocalStorage(base_dir="./downloads")

AWS S3

from ssscraper import S3Storage
storage = S3Storage(
    bucket="my-bucket",
    prefix="sitesudharo",
    region="us-east-1",
    aws_access_key_id="...",
    aws_secret_access_key="...",
)

Cloudflare R2

from ssscraper import R2Storage
storage = R2Storage(
    bucket="my-bucket",
    account_id="<cf-account-id>",
    access_key_id="...",
    secret_access_key="...",
    prefix="sitesudharo",
    public_domain="assets.yourdomain.com",  # optional
)

ScrapeResult shape

result.page            # PageMetadata — url, title, description, scraped_at
result.html            # raw HTML
result.markdown        # Crawl4ai markdown
result.resources       # List[ResourceMetadata]
result.succeeded       # filter: status == SUCCESS
result.failed          # filter: status == FAILED
result.images          # filter: type == image
result.stylesheets     # filter: type == css
result.fonts           # filter: type == font

ResourceMetadata fields

Field	Type	Description
original_url	str	Source URL
storage_key	str	Relative key inside the storage backend
stored_path	str	Absolute local path or full cloud URL
resource_type	ResourceType	image / css / javascript / font / …
content_type	str	HTTP Content-Type
size_bytes	int	File size
checksum_md5	str	MD5 hex digest
checksum_sha256	str	SHA-256 hex digest
status	ResourceStatus	success / failed / skipped
error	str	Error message if failed
downloaded_at	datetime	UTC timestamp

Install

pip install -e .          # from source
pip install ssscraper     # once published

Python ≥ 3.11 required.

Config options

See ssscraper/config.py — all fields are optional with sensible defaults.