html-reader-llm

HTML simplification and intelligent extraction for LLM.

Features

• HTML Simplification — 7-step pipeline to clean HTML for LLM consumption
• Render Detection — Compare HTTP vs Chrome to decide if browser rendering is needed
• LLM Analysis — Page classification (list/detail) + CSS selector rule generation
• CSS Selector Validation — Verify selectors against actual HTML
• MCP Server — Expose tools to LLM agents via stdio protocol

Installation

pip install html-reader-llm
# or
uv tool install html-reader-llm

CLI Usage

# Simplify HTML
html-reader-llm simplify < page.html
html-reader-llm simplify --file page.html --stats

# Detect if browser rendering is needed
html-reader-llm detect https://example.com
html-reader-llm detect https://example.com --proxy http://127.0.0.1:7890

# Full analysis with LLM
html-reader-llm analyze https://example.com
html-reader-llm analyze https://example.com --dry-run
html-reader-llm analyze https://example.com --base-url https://api.openai.com/v1 --api-key sk-xxx

# Fetch HTML
html-reader-llm fetch https://example.com
html-reader-llm fetch https://example.com --chrome
html-reader-llm fetch https://example.com --json --detect-only

# Validate CSS selectors
html-reader-llm validate https://example.com --rules-file selectors.json
html-reader-llm validate --html-file page.html --rules-file selectors.json --json

MCP Mode

Expose tools to LLM agents via stdio protocol:

# Start MCP server
uvx html-reader-llm --mcp

MCP Client Configuration

Claude Desktop (claude_desktop_config.json):

{
  "mcpServers": {
    "html-reader-llm": {
      "command": "uvx",
      "args": ["html-reader-llm", "--mcp"]
    }
  }
}

Cursor (.cursor/mcp.json):

{
  "mcpServers": {
    "html-reader-llm": {
      "command": "uvx",
      "args": ["html-reader-llm", "--mcp"]
    }
  }
}

Available MCP Tools

All tools accept HTML strings directly — no network calls. The caller is responsible for fetching HTML.

Agent Workflow

Agent fetches HTML (browser tool, curl, etc.)
  │
  ├─ simplify_html(html)
  │   → Cleaned HTML for analysis
  │
  ├─ prepare_analysis(html)
  │   → System prompt + user prompt
  │   → Agent analyzes page, generates extraction rules
  │
  ├─ validate_selectors(html, rules)
  │   → Verify rules work against actual HTML
  │
  └─ extract_data(html, rules)
      → Get extracted content (detail values or list items)

Proxy Support

All network-dependent commands support --proxy:

html-reader-llm fetch https://example.com --proxy http://127.0.0.1:7890
html-reader-llm detect https://example.com --proxy http://127.0.0.1:7890
html-reader-llm analyze https://example.com --proxy http://127.0.0.1:7890
html-reader-llm validate https://example.com --rules-file rules.json --proxy http://127.0.0.1:7890

License

MIT