CatWeb

LLM-powered classification, extraction, and summarization for web content.

Part of the CatLLM ecosystem. Thin wrapper around cat-stack that adds URL fetching and web-specific context injection.

Installation

pip install cat-web                  # pulls in cat-stack automatically
pip install "cat-web[pdf]"           # with PDF support
pip install "cat-web[agent]"         # Claude-subscription backend (model_source="claude-agent")
pip install "cat-web[codex-agent]"   # ChatGPT-subscription backend (model_source="codex-agent")

The subscription backends authenticate through your Claude or ChatGPT plan instead of a metered API key — pass model_source="claude-agent" (or "codex-agent") and leave api_key unset. model_source="claude-code" (the Claude Code CLI, if installed) needs no extra at all.

Quick Start

import catweb as cat

# Classify web pages by topic
results = cat.classify(
    categories=["News", "Opinion", "Tutorial", "Reference"],
    input_data=[
        "https://example.com/article1",
        "https://example.com/article2",
    ],
    api_key="your-api-key",
)

# Extract categories from web content
extracted = cat.extract(
    input_data=["https://example.com/page1", "https://example.com/page2"],
    description="Blog posts about technology",
    api_key="your-api-key",
)

# Summarize web pages
summaries = cat.summarize(
    input_data=["https://example.com/article1"],
    description="News articles",
    api_key="your-api-key",
)

How It Works

CatWeb accepts URLs as input, fetches the web content, strips HTML to plain text, and passes the text through cat-stack's classification/extraction/summarization pipeline. Original URLs are preserved in the output DataFrame's survey_input column.

You can also pass pre-fetched text directly — CatWeb auto-detects whether input is URLs or plain text.

API Reference

classify(categories, input_data, api_key, ...)

Classify web content into predefined categories.

extract(input_data, api_key, ...)

Discover categories from web content.

explore(input_data, api_key, ...)

Raw category extraction (with duplicates) for saturation analysis.

Same parameters as extract(), plus all cat-stack explore() parameters.

collapse_themes(input_data, api_key, ...)

Consolidate the raw label inventory from explore() into a smaller, deduplicated taxonomy (re-exported from the shared cat-stack engine): Jaro-Winkler dedup and embedding pre-merge, quality-controlled LLM merge passes, and an optional count-guided reduction to the top_n most common categories. This is the same consolidation extract() runs internally since cat-stack 2.5.0. See the cat-stack README for the full parameter table.

summarize(input_data, ...)

Summarize web content.

Web Utilities

from catweb import is_url, fetch_url_text, fetch_urls

# Check if a string is a URL
is_url("https://example.com")  # True
is_url("just text")            # False

# Fetch a single URL
text, error = fetch_url_text("https://example.com")

# Fetch multiple URLs
results = fetch_urls(["https://a.com", "https://b.com"])
# Returns: [(url, text, error), ...]

Multi-Model Ensemble

All cat-stack ensemble features work through **kwargs:

results = cat.classify(
    categories=["Positive", "Negative", "Neutral"],
    input_data=urls,
    models=[
        ("gpt-4o", "openai", "sk-..."),
        ("claude-sonnet-4-5-20250929", "anthropic", "sk-ant-..."),
    ],
    consensus_threshold="majority",
)

License

GPL-3.0-or-later