fast-web-md-extract

Extract web page content as clean Markdown. Tries a fast HTTP request first, falls back to a Camoufox browser for anti-bot-protected sites.

Install

pip install fast-web-md-extract

Quick start

CLI

# Extract a single URL
fwm https://docs.python.org/3/tutorial/classes.html

# Extract multiple URLs
fwm https://example.com https://docs.python.org

# Save to a specific directory
fwm https://example.com -o my_output

# Extract from a file of URLs
fwm -f urls.txt

# Enable verbose logging
fwm https://example.com -v

# Use fast mode (skip fallback extraction)
fwm https://example.com --fast

Python (Async)

import asyncio
from fast_web_md_extract import Extractor

async def main():
    async with Extractor() as ex:
        result = await ex.extract("https://docs.python.org/3/tutorial/classes.html")
        print(result.markdown[:200])

asyncio.run(main())

Python (Sync)

from fast_web_md_extract import Extractor

with Extractor() as ex:
    result = ex.extract_sync("https://docs.python.org/3/tutorial/classes.html")
    print(result.markdown[:200])

Save to file

# Markdown (default)
result.save("output/page.md")

# Plain text (strips Markdown syntax)
result.save("output/page.txt", fmt="txt")

Batch extract

async with Extractor() as ex:
    results = await ex.extract_many([
        "https://example.com",
        "https://docs.python.org",
    ])
    for r in results:
        if r.markdown:
            r.save(f"output/{r.url.split('/')[2]}.md")

Configuration

Extractor(
    headless=True,       # Run browser headless (default True)
    timeout=15,          # Cloudflare challenge timeout in seconds
    min_content=1000,    # Min chars to accept from fast path
    concurrency=3,       # Max concurrent browser pages
    http_concurrency=50, # Max concurrent HTTP requests
    fast=False,          # Use trafilatura fast mode
)

How it works

1. Fast path (httpx) — plain HTTP GET, no browser needed. Works for static sites like Wikipedia, Python Docs, blogs. Uses connection pooling and high concurrency (50 concurrent requests).
2. Slow path (Camoufox) — launches a stealth browser to bypass Cloudflare and other anti-bot protection. Automatic fallback when the fast path is blocked or returns too little content.
3. Content extraction (trafilatura + html-to-markdown) — trafilatura auto-detects main content first. Falls back to html-to-markdown for pages where trafilatura strips links or returns too little content.

Benchmark

Measured on Windows, Python 3.14, September 2026:

Static sites: ~1-4s | Protected sites: ~20-60s

CLI benchmark

$ fwm https://docs.python.org/3/tutorial/classes.html https://th.wikipedia.org/wiki/ประเทศไทย
Extracting: 100%|██████████| 2/2 [00:04<00:00, 2.05s/url]
  0.7s  OK: 36,732 chars -> output/docs.python.org.md
  4.1s  OK: 293,464 chars -> output/th.wikipedia.org.md

2/2 extracted successfully.

License

MIT