dompruner

한국어 | English

Python port of dompruner-mcp — DOM AST pruning for LangChain, LlamaIndex, and direct use.

When an LLM agent fetches a web page, it receives tens of thousands of raw HTML tokens it doesn't need — navigation, ads, scripts, footers. dompruner strips all of that via DOM AST parsing and passes the original content, unchanged, directly to the model. No intermediate summarization model, no API key, no vector database.

The result: 97.3% fewer tokens on average across documentation, API references, and technical pages.

> docs.python.org/3/library/asyncio-task.html
> Raw HTML   44,315 tokens
> dompruner   1,275 tokens  (97.1% reduction)
> Fetch: 133ms · Parse: 73.4ms

Quick Start

pip install dompruner

Direct use

import asyncio
from dompruner import run_pipeline

result = asyncio.run(run_pipeline("https://docs.python.org/3/library/asyncio-task.html",
                                   query="create_task"))
print(result.markdown)
print(f"Tokens: {result.original_tokens:,} → {result.refined_tokens:,} ({result.reduction_ratio:.1%} reduction)")
print(f"Fetch: {result.fetch_ms:.0f}ms · Parse: {result.parse_ms:.1f}ms")

LangChain — Document Loader

Listed in the LangChain Python integrations overview as a third-party web loader. The overview links directly to this repository — there is no separate hosted docs page.

from dompruner.langchain import DomPrunerLoader

# Synchronous
docs = DomPrunerLoader("https://fastapi.tiangolo.com/tutorial/body/",
                       query="request body").load()
doc = docs[0]
print(doc.page_content)
print(doc.metadata)
# {'source': '...', 'render_type': 'SSR', 'original_tokens': 31659,
#  'refined_tokens': 1694, 'reduction_ratio': 0.946, ...}

# Async
async def main():
    async for doc in DomPrunerLoader(url, query).alazy_load():
        print(doc.page_content)

LangChain — Tool (for agents)

from dompruner.langchain import DomPrunerFetchTool
from langchain_anthropic import ChatAnthropic

tool = DomPrunerFetchTool()
llm = ChatAnthropic(model="claude-haiku-4-5").bind_tools([tool])

Or drop it into any create_react_agent / LangGraph agent:

from langgraph.prebuilt import create_react_agent
from dompruner.langchain import DomPrunerFetchTool

agent = create_react_agent(llm, tools=[DomPrunerFetchTool()])
agent.invoke({"messages": [{"role": "user", "content":
    "Summarize the asyncio create_task docs: https://docs.python.org/3/library/asyncio-task.html"}]})

Ensuring Your Agent Always Uses dompruner

Add the rule to your agent's system prompt or instruction file:

When retrieving a URL, always use dompruner_fetch instead of any built-in web fetch.
- URL known → DomPrunerFetchTool(url=url, query=query)
- URL unknown → search for the URL first, then call DomPrunerFetchTool

How It Works

URL
 └─▶ fetch_page()       — tiered fetch: direct → UA rotation → Playwright fallback
      │
      ├─▶ [SSG]  extract_ssg_markdown()
      │           walks __NEXT_DATA__ RSC tuple tree → clean Markdown  (≥ 97% reduction)
      │           skips DOM parse entirely
      │
      └─▶ [SSR/CSR]  BeautifulSoup DOM tree
                └─▶ FQN Router (L1)        keeps p / h1–h5 / li / pre / code
                     │                     prunes nav / footer / aside / form
                     └─▶ Heading Cluster (L2)  dev-doc structure detection
                          └─▶ CETD Engine (L3)  text-density scoring fallback
                               └─▶ BM25+ Section Filter  query-aware ranking
                                    └─▶ Compact Markdown  ──▶  LLM context

Render Type Detection

Tiered Fetch

Install Playwright only if needed:

pip install "dompruner[playwright]"
playwright install chromium

BM25+ Section Filter

When query is provided, extracted sections are ranked by BM25+ score. Three adjustments:

• Heading boost (2.5×) — sections under a relevant heading rank higher
• Depth decay (0.4) — deeply nested sections score lower than top-level content
• Ancestor preservation — parent headings of selected sections are always included for context

Result: only the most relevant sections enter the LLM context, within a 1,200-token budget (configurable).

Zero-score fallback: if the query terms appear nowhere in the document (BM25 max score = 0), dompruner returns the full clean content instead of the filtered subset. No arbitrary cutoff, no small model involved.

Benchmark

All numbers are live measurements against real documentation sites. Raw HTML token counts use the len(html) // 4 estimator (same as the MCP version). Reproducible script at bench.py.

† BM25 zero-score: query terms absent from document → full clean content returned automatically.

Notes on the SSG case (React useState): raw HTML is 110,963 tokens because Next.js embeds the full RSC payload in __NEXT_DATA__. dompruner detects the __NEXT_DATA__ script tag, walks the RSC tuple tree directly, and produces 2,599 tokens — skipping the BeautifulSoup parse step entirely.

Research Backing

Web page context is too large for LLM agents FocusAgent (Oct 2025) confirms web pages routinely exceed tens of thousands of tokens, saturating context limits and increasing cost. Their LLM-based retriever achieves 50%+ observation size reduction. dompruner achieves 97%+ via deterministic DOM AST — no intermediate model, no hallucination risk in the preprocessing step. → FocusAgent: Simple Yet Effective Ways of Trimming the Large Context of Web Agents (2025)

Relevant information in long contexts is systematically missed LLM accuracy degrades 30%+ when relevant content appears in the middle of a long context (U-shaped curve). Reducing from ~60K to ~1.6K tokens structurally eliminates this problem. → Lost in the Middle: How Language Models Use Long Contexts — Liu et al., Stanford (2023)

BM25 is the strongest scalable retrieval default A 2026 controlled scaling study shows BM25 overtaking agentic search at 10M corpus tokens by ~20 points while remaining Pareto-optimal without LLM-based construction. → BM25 Wins at Scale: A Scaling Study of RAG Paradigms (2026)

API Reference

run_pipeline(url, query="") → PipelineResult

@dataclass
class PipelineResult:
    url: str
    render_type: str          # "SSG" | "SSR" | "CSR"
    markdown: str
    original_tokens: int      # len(raw_html) // 4
    refined_tokens: int       # len(markdown) // 4
    reduction_ratio: float    # 1 - refined / original
    fetch_ms: float
    parse_ms: float
    bm25_confidence: float | None   # None = no query or zero-score fallback

DomPrunerLoader(url, query="")

LangChain BaseLoader. Produces one Document per URL with metadata matching PipelineResult fields. Supports both load() (sync) and alazy_load() (async).

DomPrunerFetchTool()

LangChain BaseTool. Name: dompruner_fetch. Args: url (str), query (str, optional). Returns the pruned Markdown string. Supports _run (sync) and _arun (async).

Architecture

dompruner/
  __init__.py          — run_pipeline, PipelineResult (public API)
  pipeline.py          — Orchestrator: fetch → detect → extract → BM25 → serialize
  fetcher.py           — Tiered HTTP fetch (httpx → UA rotation → Playwright)
  extractor.py         — L1→L2→L3 extraction cascade
                         L1: FQN Router (CONTENT_TAGS selector + NOISE_TAGS prune)
                         L2: Heading Cluster (dev-doc detection, link density < 0.3)
                         L3: CETD Engine (text-density scoring, score = textLen/tagCount × (1-linkRatio) × depthPenalty)
  ssg.py               — __NEXT_DATA__ / Nuxt RSC tuple tree walker
  bm25.py              — BM25+ section filter (heading boost 2.5× + ancestor preservation)
  serializer.py        — FQNNode[] → Compact Markdown + token estimator
  langchain/
    loader.py          — DomPrunerLoader (BaseLoader)
    tool.py            — DomPrunerFetchTool (BaseTool)

Faithful Python port: the extraction logic (FQN Router, Heading Cluster, CETD, SSG RSC walker, BM25+ section filter) is a direct port of the dompruner-mcp TypeScript implementation. All scoring constants, tag sets, and fallback thresholds match the original.

Development

git clone https://github.com/dong7812/dompruner-py.git
cd dompruner-py
pip install -e ".[dev]"
pytest

Run the benchmark locally:

python bench.py

Related

• dompruner-mcp — Node.js MCP server. Adds dompruner_fetch as an MCP tool to Claude Code, Claude Desktop, Cursor, Windsurf, and any MCP-compatible client. Zero install — npx -y dompruner-mcp.
• LangChain integrations overview — dompruner is listed as a third-party web loader. LangChain's current policy links out to maintainer repos rather than hosting integration docs directly.

License

MIT