winnow-md 0.1.0

Remove boilerplate from scraped markdown before it reaches an LLM. Subtractive-only, auditable, zero dependencies.

Winnow

Remove boilerplate from scraped markdown before it reaches an LLM.

Scraping and extract APIs like Tavily hand you "LLM-ready" markdown that still carries navigation, footers, cookie banners, promos, and link farms — 10–90% of the tokens depending on the page. The same goes for your own scraper or loader pipeline: if it produces markdown (or HTML converted to markdown), Winnow slots in right after it — any markdown in, leaner markdown out, with a receipt for every removed block.

• Subtractive-only. Winnow deletes blocks; it never rewrites a word. Zero hallucination risk by construction.
• Recall-first. Dropping real content is data loss; keeping boilerplate just costs tokens. When uncertain, Winnow keeps.
• Auditable. Every removed block comes back with a reason code and score.
• Template memory. Feed Winnow multiple pages from one site and it learns the site's template — blocks repeated across pages are boilerplate, near-certainly.
• Zero runtime dependencies. pip install winnow-md adds nothing else.

Quickstart

import winnow

# One page, heuristics only
res = winnow.clean(markdown_text)
print(res.markdown)          # cleaned markdown
print(res.stats)             # tokens before/after, reduction %
for r in res.removed:        # the receipt
    print(r.reasons, r.text[:60])

# A crawl — template memory kicks in across pages of the same domain
w = winnow.Winnow(aggressiveness=0.5)
results = w.clean_many(pages)          # list of markdown strings (or (md, url) tuples)

# Streaming with a persistent per-domain template store
w = winnow.Winnow(store="winnow.db")
res = w.clean(md, url="https://example.com/post/1")

winnow clean page.md                     # cleaned markdown to stdout
winnow clean ./crawl/ --report out.html  # batch + side-by-side audit report

Jina Reader output (Title: / URL Source: preamble) is auto-detected and the source URL is used for template memory.

Benchmark

Five generations of independently-labeled, adversarially-arbitrated exam batches (each fetched fresh, dual-labeled blind, disputes refereed) — ~12,000 hand-adjudicated blocks across 21 domains:

Exam batch	Content recall	Junk recall	Token cut
batch 5 (newest, still converging)	0.956	0.61	−42%
batch 4	0.979	0.59	−49%
batch 3	0.992	0.56	−35%
batch 2	0.997	0.55	−30%
batch 1	1.000	0.66	−41%

Add --url-mode strip for roughly 15 additional points of token cut with zero text loss. use_model=True (extra: pip install winnow-md[model]) adds a learned block-sequence scorer that raises junk recall further and is structurally incapable of deleting a block on its own.

The benchmark harness, labeling pipeline, and mutation self-test live in bench/ — see ARCHITECTURE.md.