dompruner

한국어 | English

Python port of dompruner-mcp — DOM AST pruning for LangChain, LlamaIndex, and direct use.

When an LLM agent fetches a web page, it receives tens of thousands of raw HTML tokens it doesn't need — navigation, ads, scripts, footers. dompruner strips all of that via DOM AST parsing and passes the original content, unchanged, directly to the model. No intermediate summarization model, no API key, no vector database.

The result: 97.3% fewer tokens on average across documentation, API references, and technical pages.

> docs.python.org/3/library/asyncio-task.html
> Raw HTML   44,315 tokens
> dompruner   1,275 tokens  (97.1% reduction)
> Fetch: 133ms · Parse: 73.4ms

---

Quick Start

pip install dompruner

Direct use

import asyncio
from dompruner import run_pipeline

result = asyncio.run(run_pipeline("https://docs.python.org/3/library/asyncio-task.html",
                                   query="create_task"))
print(result.markdown)
print(f"Tokens: {result.original_tokens:,} → {result.refined_tokens:,} ({result.reduction_ratio:.1%} reduction)")
print(f"Fetch: {result.fetch_ms:.0f}ms · Parse: {result.parse_ms:.1f}ms")

LangChain — Document Loader

from dompruner.langchain import DomPrunerLoader

# Synchronous
docs = DomPrunerLoader("https://fastapi.tiangolo.com/tutorial/body/",
                       query="request body").load()
doc = docs[0]
print(doc.page_content)
print(doc.metadata)
# {'source': '...', 'render_type': 'SSR', 'original_tokens': 31659,
#  'refined_tokens': 1694, 'reduction_ratio': 0.946, ...}

# Async
async def main():
    async for doc in DomPrunerLoader(url, query).alazy_load():
        print(doc.page_content)

LangChain — Tool (for agents)

from dompruner.langchain import DomPrunerFetchTool
from langchain_anthropic import ChatAnthropic

tool = DomPrunerFetchTool()
llm = ChatAnthropic(model="claude-haiku-4-5").bind_tools([tool])

Or drop it into any create_react_agent / LangGraph agent:

from langgraph.prebuilt import create_react_agent
from dompruner.langchain import DomPrunerFetchTool

agent = create_react_agent(llm, tools=[DomPrunerFetchTool()])
agent.invoke({"messages": [{"role": "user", "content":
    "Summarize the asyncio create_task docs: https://docs.python.org/3/library/asyncio-task.html"}]})

---

Ensuring Your Agent Always Uses dompruner

Add the rule to your agent's system prompt or instruction file:

When retrieving a URL, always use dompruner_fetch instead of any built-in web fetch.
- URL known → DomPrunerFetchTool(url=url, query=query)
- URL unknown → search for the URL first, then call DomPrunerFetchTool

Framework	Where to set
LangChain / LangGraph	SystemMessage or agent instructions
AutoGen	system_message in ConversableAgent
CrewAI	Agent backstory or task description
LlamaIndex	ReActAgent system prompt

---

How It Works

URL
 └─▶ fetch_page()       — tiered fetch: direct → UA rotation → Playwright fallback
      │
      ├─▶ [SSG]  extract_ssg_markdown()
      │           walks __NEXT_DATA__ RSC tuple tree → clean Markdown  (≥ 97% reduction)
      │           skips DOM parse entirely
      │
      └─▶ [SSR/CSR]  BeautifulSoup DOM tree
                └─▶ FQN Router (L1)        keeps p / h1–h5 / li / pre / code
                     │                     prunes nav / footer / aside / form
                     └─▶ Heading Cluster (L2)  dev-doc structure detection
                          └─▶ CETD Engine (L3)  text-density scoring fallback
                               └─▶ BM25+ Section Filter  query-aware ranking
                                    └─▶ Compact Markdown  ──▶  LLM context

Render Type Detection

Type	Signal	Strategy
SSG	__NEXT_DATA__, window.__NUXT__, window.page	RSC tuple tree walk — DOM parse skipped
SSR	Body text density ≥ 2%	Full DOM AST pipeline (L1→L2→L3)
CSR	Body text density < 2%	DOM AST pipeline (partial content)

Tiered Fetch

Level	Trigger	Method
L1	Default	httpx direct fetch
L2	403 / 429 response	User-Agent rotation (3 browser UA strings)
L3	CSR detected or L2 fails	playwright headless browser (optional dep)

Install Playwright only if needed:

pip install "dompruner[playwright]"
playwright install chromium

BM25+ Section Filter

When query is provided, extracted sections are ranked by BM25+ score. Three adjustments:

• Heading boost (2.5×) — sections under a relevant heading rank higher
• Depth decay (0.4) — deeply nested sections score lower than top-level content
• Ancestor preservation — parent headings of selected sections are always included for context

Result: only the most relevant sections enter the LLM context, within a 1,200-token budget (configurable).

Zero-score fallback: if the query terms appear nowhere in the document (BM25 max score = 0), dompruner returns the full clean content instead of the filtered subset. No arbitrary cutoff, no small model involved.

---

Benchmark

All numbers are live measurements against real documentation sites. Raw HTML token counts use the len(html) // 4 estimator (same as the MCP version). Reproducible script at bench.py.

Site	Raw HTML	dompruner	Reduction	Fetch	Parse	Mode
Python asyncio	44,315	1,275	97.1%	133ms	73.4ms	BM25
Rust Book ch04	14,003	2,713	80.6%	92ms	13.2ms	BM25
React useState (SSG)	110,963	2,599	97.7%	186ms	68.8ms	BM25
FastAPI Body	31,659	1,694	94.6%	85ms	33.9ms	full†
MDN Fetch API	38,086	691	98.2%	257ms	30.4ms	full†
Next.js Routing	156,578	1,828	98.8%	732ms	66.5ms	full†
TypeScript Handbook	47,333	305	99.4%	324ms	15.8ms	full†
Vue Reactivity	38,375	2,045	94.7%	656ms	50.2ms	BM25
Average	60,164	1,643	97.3%	308ms	44.0ms

† BM25 zero-score: query terms absent from document → full clean content returned automatically.

Notes on the SSG case (React useState): raw HTML is 110,963 tokens because Next.js embeds the full RSC payload in __NEXT_DATA__. dompruner detects the __NEXT_DATA__ script tag, walks the RSC tuple tree directly, and produces 2,599 tokens — skipping the BeautifulSoup parse step entirely.

---

Research Backing

Web page context is too large for LLM agents FocusAgent (Oct 2025) confirms web pages routinely exceed tens of thousands of tokens, saturating context limits and increasing cost. Their LLM-based retriever achieves 50%+ observation size reduction. dompruner achieves 97%+ via deterministic DOM AST — no intermediate model, no hallucination risk in the preprocessing step. → FocusAgent: Simple Yet Effective Ways of Trimming the Large Context of Web Agents (2025)

Relevant information in long contexts is systematically missed LLM accuracy degrades 30%+ when relevant content appears in the middle of a long context (U-shaped curve). Reducing from ~60K to ~1.6K tokens structurally eliminates this problem. → Lost in the Middle: How Language Models Use Long Contexts — Liu et al., Stanford (2023)

BM25 is the strongest scalable retrieval default A 2026 controlled scaling study shows BM25 overtaking agentic search at 10M corpus tokens by ~20 points while remaining Pareto-optimal without LLM-based construction. → BM25 Wins at Scale: A Scaling Study of RAG Paradigms (2026)

---

API Reference

run_pipeline(url, query="") → PipelineResult

@dataclass
class PipelineResult:
    url: str
    render_type: str          # "SSG" | "SSR" | "CSR"
    markdown: str
    original_tokens: int      # len(raw_html) // 4
    refined_tokens: int       # len(markdown) // 4
    reduction_ratio: float    # 1 - refined / original
    fetch_ms: float
    parse_ms: float
    bm25_confidence: float | None   # None = no query or zero-score fallback

DomPrunerLoader(url, query="")

LangChain BaseLoader. Produces one Document per URL with metadata matching PipelineResult fields. Supports both load() (sync) and alazy_load() (async).

DomPrunerFetchTool()

LangChain BaseTool. Name: dompruner_fetch. Args: url (str), query (str, optional). Returns the pruned Markdown string. Supports _run (sync) and _arun (async).

---

Architecture

dompruner/
  __init__.py          — run_pipeline, PipelineResult (public API)
  pipeline.py          — Orchestrator: fetch → detect → extract → BM25 → serialize
  fetcher.py           — Tiered HTTP fetch (httpx → UA rotation → Playwright)
  extractor.py         — L1→L2→L3 extraction cascade
                         L1: FQN Router (CONTENT_TAGS selector + NOISE_TAGS prune)
                         L2: Heading Cluster (dev-doc detection, link density < 0.3)
                         L3: CETD Engine (text-density scoring, score = textLen/tagCount × (1-linkRatio) × depthPenalty)
  ssg.py               — __NEXT_DATA__ / Nuxt RSC tuple tree walker
  bm25.py              — BM25+ section filter (heading boost 2.5× + ancestor preservation)
  serializer.py        — FQNNode[] → Compact Markdown + token estimator
  langchain/
    loader.py          — DomPrunerLoader (BaseLoader)
    tool.py            — DomPrunerFetchTool (BaseTool)

Faithful Python port: the extraction logic (FQN Router, Heading Cluster, CETD, SSG RSC walker, BM25+ section filter) is a direct port of the dompruner-mcp TypeScript implementation. All scoring constants, tag sets, and fallback thresholds match the original.

---

Development

git clone https://github.com/dong7812/dompruner-py.git
cd dompruner-py
pip install -e ".[dev]"
pytest

Run the benchmark locally:

python bench.py

---

Related

• dompruner-mcp — Node.js MCP server. Adds dompruner_fetch as an MCP tool to Claude Code, Claude Desktop, Cursor, Windsurf, and any MCP-compatible client. Zero install — npx -y dompruner-mcp.

---

License

MIT