websource

Standalone website discovery, fetching, parsing, rendering, content extraction, structured extraction, refresh checking, and adapter contracts.

websource is built around strict Pydantic models and protocol interfaces. The foundation layer owns value objects, enums, request/result models, adapter descriptors, protocol interfaces, and error contracts. Runtime HTTP, browser, provider, and website-specific adapters sit outside websource.foundation.

Structured extraction is modeled separately from readable content extraction, so article markdown, recipes, product records, property listings, vehicle ads, jobs, events, and custom schemas can share one transport contract without forcing every page into an article-shaped model.

Install

pip install websource

Optional browser rendering uses Playwright:

pip install "websource[browser]"
python -m playwright install chromium

For local development:

uv sync --all-extras --group dev
uv run --group dev pytest

CLI

websource adapters list --json
websource inspect https://example.com --json
websource discover https://example.com --mode crawl --max-depth 1 --jsonl
websource fetch https://example.com --json
websource fixture replay tests/fixtures/sites/example.com/article.html --url https://example.com/articles/one --json
websource provider run tavily tavily-http https://api.tavily.example/extract extract --input-url https://example.com/article --param schema=article --json

The CLI emits JSON envelopes for validation and runtime errors so agents and automation can handle failures deterministically. inspect performs a shallow seed URL probe: robots facts and verified source hints stay in fields such as robots.sitemap_urls and sitemap_urls, while unverified conventional fallbacks are labeled as candidate_sitemap_urls. When no sitemap hint has been verified, sitemap_urls is null, not []. Use inspect --include-http-headers to append request and response headers for the HTTP exchanges made during inspection.

Python API

from websource.adapters import DefaultWebsiteClient
from websource.foundation import FetchRequest

client = DefaultWebsiteClient()
result = await client.fetch(FetchRequest(url="https://example.com"))
page = await client.parse_page(result)
content = await client.extract_content(page)

Custom adapters can subclass SiteSpecificAdapter and advertise only the capabilities they override. Managed providers can use HttpManagedProviderAdapter while provider-specific packages stabilize. Built-in vertical schemas are available through websource.schemas for recipes, products, property listings, vehicle listings, job postings, and events.

Release Checks

uv run --group dev ruff check src tests
uv run --group dev mypy src tests
uv run --group dev pytest
uv run --group dev bandit -r src
uv build
scripts/verify-package-install.sh

Documentation

• Docs index
• Foundation contracts
• Website client architecture
• Adapter design
• CLI and tool surface
• Vertical schemas
• JSON schemas
• Release policy
• Implementation plan

License

MIT